Parameter estimation method, device and electronic equipment

Abstract

A parameter estimation method, device and electronic device, the method comprising: obtaining N Flink subtasks, processing data to be processed based on the parallelism in each Flink subtask, obtaining the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask and the resource allocation rate of the Flink subtask, obtaining N resource allocation rates, and screening out the target resource allocation rate and the actual parallelism of the Flink subtask corresponding to the target resource allocation rate from the N resource allocation rates. Through the above method, the quantitative value of the parallelism of each Flink subtask is estimated in advance, ensuring that there is a sufficient number of parallelisms to process the data to be processed, and screening out the target resource allocation rate from the N resource allocation rates, ensuring that the screened target resource allocation rate is the optimal parameter, further improving the performance of stream computing, and achieving the purpose of optimizing stream computing.

Description

Technical Field

[0001] The present application relates to the field of distributed computing technology, and in particular to a parameter estimation method, device and electronic device. Background Art

[0002] With the development of big data processing technology, in the traditional data processing process, data is always collected first and then put into the database. When people need it, they query the data through the database to get answers or query related data. However, in some real-time search application environments, when there are some specific problems, the traditional data processing process will search for the relevant data of the problem in the database. When the amount of data in the database is too large or the number of users querying the problem is too large, the query results cannot be returned in real time. Therefore, the effectiveness of querying data using the traditional data processing process is poor.

[0003] In order to process the data generated in the network in real time, a new data computing structure, stream computing, has been introduced. Stream computing can analyze large-scale flow data in real time during the ever-changing movement process, capture useful information, and the response time of stream computing is in seconds, realizing real-time data processing.

[0004] In order to optimize stream computing, a Flink-based stream computing performance optimization system and method is currently used. After collecting indicators in the production environment, this method analyzes the collected data to obtain analysis results, and then uses the analysis results to adjust the running parameters of the task.

[0005] First of all, Flink (full name Apache Flink) is a distributed processing engine. There are multiple task managers in the Flink cluster. Data reading, calculation and other operations are all run in the form of distributed tasks in the task manager. When data calculation is performed in the task manager, after the data calculation is completed, the task manager will return the obtained resource allocation rate, parallelism and data calculation results to the caller.

[0006] The principle of distributed computing by task manager is shown in Figure 1 :Each task manager includes several slots. Slots are the smallest units for performing calculations. Operators to be calculated (including data reading, mapping, grouping, storage, etc.) are divided into slots according to the parallelism of the operators, and the slots perform calculations in sequence. After executing several operators on a slot, they are assigned to the next slot for further execution according to the needs of data grouping until they are finally output or saved. Several threads can be executed simultaneously in each slot to ensure the efficiency of Flink's distributed operation.

[0007] Currently, the above stream computing performance optimization method provides a method for real-time collection of Flink task runtime performance indicators in a production environment, and dynamically adjusts the parallelism of the task manager, the number of slots on each (Taskmanager) task manager, and the memory size allocated to each task manager according to the collected performance indicators.

[0008] After collecting indicators in the production environment, this method analyzes the data collected by the indicators to obtain analysis results, and then uses the analysis results to adjust the operating parameters of the task, causing the configuration of the optimal operating parameters of the task to lag behind the needs of actual data processing, resulting in a large difference between the operating parameters corresponding to the analysis results and the optimal operating parameters of the task at the current moment. In addition, in the process of collecting indicators, when the amount of data to be collected is too much, the analysis results cannot be obtained in time, and the parameters of the task operation cannot be adjusted, resulting in data backlog, making it impossible to feedback data in real time, and the optimization effect is poor. Summary of the invention

[0009] The present application provides a parameter estimation method, device and electronic device, which sets N Flink subtasks, processes the data to be processed with the parallelism in each Flink subtask, obtains the actual quantity value of the data to be processed with the parallelism in each Flink subtask and the resource allocation rate corresponding to each Flink subtask, and then determines the target resource allocation rate and the actual parallelism of the Flink subtask that processes the data to be processed corresponding to the target resource allocation rate from the N resource allocation rates, thereby ensuring that the obtained target resource allocation rate and the actual parallelism of the Flink subtask that processes the data to be processed are optimal parameters, thereby optimizing the convection computing.

[0010] In a first aspect, the present application provides a parameter estimation method, the method comprising:

[0011] Obtain N Flink subtasks, where N is a positive integer, the Flink subtask has multiple degrees of parallelism, and the number of degrees of parallelism in each Flink subtask is inconsistent;

[0012] Process the data to be processed based on the parallelism in each Flink subtask, obtain the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask and the resource allocation rate of the Flink subtask, and obtain N resource allocation rates, wherein the resource allocation rate is the percentage of device resources occupied by all parallelisms in processing the data to be processed in each Flink subtask;

[0013] A target resource allocation rate and an actual parallelism of the Flink subtask corresponding to the target resource allocation rate are screened out from the N resource allocation rates.

[0014] In a possible design, a target resource allocation rate and an actual parallelism of a Flink subtask corresponding to the target resource allocation rate are screened out from the N resource allocation rates, including:

[0015] Get the resource allocation rates of N Flink subtasks and check whether each resource allocation rate is within the preset range;

[0016] If yes, the minimum resource allocation rate within the preset range is used as the target resource allocation rate, and the actual parallelism of the Flink subtask corresponding to the target resource allocation rate is obtained;

[0017] If not, extract the target endpoint value in the preset range, calculate the difference between the resource allocation rate and the target endpoint value, take the resource allocation rate corresponding to the minimum difference as the target resource allocation rate, and obtain the actual parallelism of the Flink subtask corresponding to the target resource allocation rate.

[0018] In one possible design, obtaining the actual parallelism of each Flink subtask that processes the data to be processed includes:

[0019] Read the current number value of the parallelism of processing the data to be processed in each Flink subtask, and obtain the back pressure indicator corresponding to the parallelism of the current number value;

[0020] According to the relationship between the back pressure index and the preset threshold, the current number of values ​​of all parallelisms for processing the data to be processed is adjusted until the back pressure index is lower than the preset threshold, and the current number of values ​​of parallelism that meets the first preset rule is used as the actual number value.

[0021] In a possible design, taking the current quantity value of the parallelism that complies with the first preset rule as the actual quantity value includes:

[0022] Read a first quantity value interval corresponding to each Flink subtask, and update the first quantity value interval based on the current quantity values ​​of all parallelisms of processing the data to be processed;

[0023] When the minimum value of the first quantity value interval is not 1 and the difference between the maximum value and the minimum value in the first quantity value interval does not exceed 1, the actual parallelism of the Flink subtask is calculated.

[0024] In a possible design, taking the current quantity value of the parallelism that complies with the first preset rule as the actual quantity value includes:

[0025] Obtaining back pressure indicators of all parallelisms for processing the data to be processed in different first quantity value intervals;

[0026] The back pressure indicators are arranged according to a preset arrangement order, a first quantity value interval corresponding to a minimum back pressure indicator is selected, and the actual parallelism of all Flink subtasks that process the data to be processed is calculated based on the first quantity value interval.

[0027] In a possible design, adjusting the current number of parallelism values ​​according to the relationship between the back pressure indicator and the preset threshold value includes:

[0028] Check whether the back pressure index of all parallelisms of the data to be processed exceeds the preset threshold;

[0029] If so, obtaining the current number values ​​corresponding to all the parallelisms of processing the data to be processed, and adjusting the current number values ​​of all the parallelisms of processing the data to be processed based on the second preset rule;

[0030] If not, when it is determined that the quantity value of all parallelisms for processing the data to be processed is greater than 1 and the difference between the maximum quantity value and the minimum quantity value in the first quantity value interval is greater than 1, the current quantity value of all parallelisms for processing the data to be processed is adjusted based on a third preset rule.

[0031] In a possible design, adjusting the current number of all parallelisms of processing the data to be processed based on the second preset rule includes:

[0032] Parse the maximum and minimum values ​​in the first quantity value interval corresponding to the Flink subtask;

[0033] Overwriting the minimum value with the current value of the parallelism of processing the data to be processed;

[0034] The maximum quantity value is added to the current quantity value to obtain a first added quantity value, and a quantity value corresponding to an average of the first added quantity values ​​is taken as the current quantity value.

[0035] In one possible design, adding the maximum quantity value to the current quantity value to obtain a first added quantity value includes:

[0036] If the maximum quantity value or the current quantity value is an odd number, adding 1 to the maximum quantity value or the current quantity value to obtain a first added quantity value;

[0037] If the maximum quantity value and the current quantity value are even numbers, the maximum quantity value and the current quantity value are added to obtain a first added quantity value.

[0038] In a possible design, adjusting the current number of all parallelisms of processing the data to be processed based on the third preset rule includes:

[0039] Parse the maximum and minimum values ​​in the first quantity value interval corresponding to the Flink subtask;

[0040] Overwriting the maximum value with the current value of the degree of parallelism of processing the data to be processed;

[0041] The current quantity value is added to the minimum quantity value to obtain a second added quantity value, and a quantity value corresponding to the average of the second added quantity values ​​is taken as the current quantity value.

[0042] In one possible design, adding the current quantity value to the minimum quantity value to obtain a second added quantity value includes:

[0043] If the minimum quantity value or the current quantity value is an odd number, adding 1 to the minimum quantity value or the current quantity value to obtain a second added quantity value;

[0044] If the minimum quantity value and the current quantity value are even numbers, the minimum quantity value and the current quantity value are added to obtain a second added quantity value.

[0045] In a second aspect, the present application provides a thread binding device, the device comprising:

[0046] Get module, used to get N Flink subtasks;

[0047] A processing module is used to process the data to be processed based on the parallelism in each Flink subtask, obtain the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask and the resource allocation rate of the Flink subtask, and obtain N resource allocation rates;

[0048] The screening module is used to screen out a target resource allocation rate and an actual parallelism of the Flink subtask corresponding to the target resource allocation rate from the N resource allocation rates.

[0049] In a possible design, the screening module is specifically used to obtain the resource allocation rates of N Flink subtasks, detect whether each resource allocation rate is within a preset range, and if so, use the minimum resource allocation rate within the preset range as the target resource allocation rate, and obtain the actual parallelism of the Flink subtask corresponding to the target resource allocation rate; if not, extract the target endpoint value in the preset range, calculate the difference between the resource allocation rate and the target endpoint value, use the resource allocation rate corresponding to the minimum difference as the target resource allocation rate, and obtain the actual parallelism of the Flink subtask corresponding to the target resource allocation rate.

[0050] In a possible design, the processing module is specifically used to read the current number value of the parallelism of processing the data to be processed in each Flink subtask, and obtain the back pressure index corresponding to the parallelism of the current number value, and adjust the current number value of all parallelisms of processing the data to be processed according to the relationship between the back pressure index and the preset threshold value, until the back pressure index is lower than the preset threshold value, and the current number value of the parallelism that meets the first preset rule is used as the actual number value;

[0051] In a possible design, the processing module is also used to read a first quantity value interval corresponding to each Flink subtask, update the first quantity value interval based on the current quantity value of all parallelisms of processing the data to be processed, and when the minimum quantity value of the first quantity value interval is not 1 and the difference between the maximum quantity value and the minimum quantity value in the first quantity value interval does not exceed 1, calculate the actual parallelism of the Flink subtask.

[0052] In a possible design, the processing module is also used to obtain back pressure indicators of all parallelisms for processing the data to be processed in different first quantity value intervals, arrange the back pressure indicators according to a preset arrangement order, select the first quantity value interval corresponding to the minimum back pressure indicator, and calculate the actual parallelism of all Flink subtasks that process the data to be processed based on the first quantity value interval.

[0053] In one possible design, the processing module is also used to detect whether the back pressure index of all parallelisms of processing the data to be processed exceeds a preset threshold. If so, the current quantity values ​​corresponding to all parallelisms of processing the data to be processed are obtained, and the current quantity values ​​of all parallelisms of processing the data to be processed are adjusted based on the second preset rule. If not, when it is determined that the quantity values ​​of all parallelisms of processing the data to be processed are greater than 1 and the difference between the maximum quantity value and the minimum quantity value in the first quantity value interval is greater than 1, the current quantity values ​​of all parallelisms of processing the data to be processed are adjusted based on the third preset rule.

[0054] In a possible design, the processing module is also used to parse out the maximum quantity value and the minimum quantity value in the first quantity value interval corresponding to the Flink subtask, overwrite the minimum quantity value with the current quantity value of the parallelism of processing the data to be processed, add the maximum quantity value to the current quantity value to obtain a first added quantity value, and take the quantity value corresponding to the average of the first added quantity value as the current quantity value.

[0055] In one possible design, the processing module is also used to, if the maximum number value or the current number value is an odd number, add 1 to the maximum number value or the current number value to obtain a first added number value; if the maximum number value and the current number value are even numbers, add the maximum number value and the current number value to obtain a first added number value.

[0056] In a possible design, the processing module is also used to parse out the maximum quantity value and the minimum quantity value in the first quantity value interval corresponding to the Flink subtask, overwrite the maximum quantity value with the current quantity value of the parallelism of processing the data to be processed, add the current quantity value to the minimum quantity value to obtain a second added quantity value, and take the quantity value corresponding to the average of the second added quantity value as the current quantity value.

[0057] In one possible design, the processing module is also used to, if the minimum quantity value or the current quantity value is an odd number, add 1 to the minimum quantity value or the current quantity value to obtain a second added quantity value; if the minimum quantity value and the current quantity value are even numbers, add the minimum quantity value and the current quantity value to obtain a second added quantity value.

[0058] In a third aspect, the present application provides an electronic device, including:

[0059] Memory, used to store computer programs;

[0060] The processor is used to implement the above-mentioned parameter estimation method steps when executing the computer program stored in the memory.

[0061] In a fourth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the parameter estimation method described above are implemented.

[0062] For each aspect from the first to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of the principle of distributed computing by the task manager provided in this application;

[0064] Figure 2 A flow chart of the steps of a parameter estimation method provided in this application;

[0065] Figure 3 A schematic diagram of the structure of a parameter estimation device provided in this application;

[0066] Figure 4 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0068] In the past, in order to optimize stream computing, a stream computing performance optimization system and method based on Flink was used. The method is to analyze the data collected by the indicators after collecting indicators in the production environment, obtain the analysis results, and then use the analysis results to adjust the running parameters of the task. After the indicator collection process, the data collected by the indicators is analyzed to obtain the analysis results, and then the analysis results are used to adjust the running parameters of the task, which causes the configuration of the optimal running parameters of the task to lag behind the needs of actual data processing, resulting in a large difference between the running parameters corresponding to the analysis results and the optimal running parameters of the task at the current moment. In addition, in the process of collecting indicators, when the amount of data to be collected is too much, it will cause the inability to obtain the analysis results in time, and then the parameters of the task operation cannot be adjusted, resulting in data backlog, and thus it is impossible to feedback data in real time, and the optimization effect is poor.

[0069] In order to solve the above-described problems, the embodiments of the present application provide a parameter estimation method for selecting the optimal parameters from N sets of data, thereby optimizing convection calculation. The method and device described in the embodiments of the present application are based on the same technical concept. Since the principles of the problems solved by the method and the device are similar, the embodiments of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0070] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0071] Reference Figure 2, the present application provides a parameter estimation method, which can screen out the optimal resource allocation rate from the resource allocation rates corresponding to N groups of Flink subtasks, as well as the actual parallelism of the Flink subtasks in the Flink subtasks corresponding to the resource allocation rate, thereby optimizing the stream computing. The implementation process of the method is as follows:

[0072] Step S21: Obtain N Flink subtasks.

[0073] The embodiment of the present application is to filter out the optimal parameters from N groups of data to optimize the stream computing. Therefore, it is necessary to set N Flink subtasks in advance. Each Flink subtask has multiple degrees of parallelism, and the number of degrees of parallelism in each Flink subtask is inconsistent.

[0074] There are multiple degrees of parallelism in a Flink subtask. The number of degrees of parallelism in a Flink subtask is different, and the amount of resource allocation for the degree of parallelism in the Flink subtask is also different. The parallelism set in each Flink subtask can ensure that the data to be processed can be processed completely, so as to estimate the number of degrees of parallelism in the Flink subtask according to the total amount of data to be processed.

[0075] It should be further explained that the number of parallelism values ​​in Flink subtasks is equivalent to the number of slots. The CPU of the device, the resource allocation of a single slot, and the number of slots are shown in Table 1:

[0076] Serial number CPU Core Resource allocation for a single slot The number of slots 1 1 100％ 1 2 0.5 50％ 2 3 0.25 25％ 4 4 0.125 12.5％ 8

[0077] Table 1

[0078] In Table 1, the total amount of resources is unchanged. When the number of slots is 1, the CPU core is 1, and 1 slot can be allocated to 100% of the CPU resources at most; when the number of slots is 2, a single slot can be allocated to 50% of the CPU resources at most, and a single slot can only occupy 0.5 of the CPU core; when the number of slots is 4, a single slot can be allocated to 25% of the CPU resources at most, and a single slot can only occupy 0.25 of the CPU core; when the number of slots is 8, a single slot can be allocated to 12.5% ​​of the CPU resources at most, and a single slot can only occupy 0.125 of the CPU core. Therefore, when the number value of 0.125 CPU cores occupied by a single slot changes, the number of slots will also change accordingly.

[0079] By obtaining N Flink subtasks, more sample data can be obtained, ensuring the accuracy of selecting the final optimal parameters from the N sample data.

[0080] Step S22: Process the data to be processed based on the parallelism in each Flink subtask, obtain the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask and the resource allocation rate of the Flink subtask, and obtain N resource allocation rates.

[0081] After obtaining N Flink subtasks, in order to obtain the optimal number of parallelism values ​​for processing the data to be processed in each Flink subtask, it is necessary to continuously adjust the number of parallelism values ​​for processing the data to be processed in each Flink subtask in the process of processing the data to be processed in parallel, so as to obtain the optimal number of parallelism values ​​for processing the data to be processed in each Flink subtask. The relationship between Flink subtasks and parallelism is shown in Table 2:

[0082] Flink Subtasks The amount of parallelism The first quantity value interval Flink subtask 1 16 [1，16] Flink subtask 2 32 [1，32] ...... ...... ......

[0083] Table 2

[0084] In the above Table 2, the number of parallelism values ​​in the Flink subtasks and the first number value intervals corresponding to the Flink subtasks are recorded. Table 2 only cites the number of parallelism values ​​and the first number value intervals corresponding to Flink subtask 1 and Flink subtask 2 respectively. The number of parallelism values ​​and the first number value intervals corresponding to other Flink subtasks can refer to any Flink subtask in Table 1, and no further elaboration is given here.

[0085] It should be further explained that in each Flink subtask, the degree of parallelism is the maximum degree of parallelism available for processing the data to be processed. The first value range is the maximum degree of parallelism available for processing the data to be processed in each Flink subtask and the minimum degree of parallelism available. For example, in Flink subtask 1, a minimum of 1 degree of parallelism and a maximum of 16 degrees of parallelism can be used to process the data to be processed.

[0086] After the above description of Flink subtasks, in order to determine the optimal number of parallelism values ​​for processing data to be processed in each Flink subtask, it is necessary to adjust the number of parallelism values ​​for processing data to be processed multiple times. The specific process of adjusting the number of parallelism values ​​for processing data to be processed in a Flink subtask is as follows:

[0087] After determining the first quantity value interval corresponding to the current Flink subtask, the parallelism in the Flink subtask processes the data to be processed. The amount of the data to be processed remains unchanged in the embodiment of the present application. After the parallelism in the Flink subtask processes the data to be processed, the current quantity value of the parallelism for processing the data to be processed is counted.

[0088] After obtaining the current number of parallelism values ​​for processing the data to be processed, in order to ensure the highest efficiency in processing the data to be processed, it is necessary to determine the actual parallelism of the Flink subtask that processes the data to be processed from the Flink subtask. After ensuring that the parallelism of the Flink subtask completes the processing of the data to be processed, it is also necessary to determine whether the parallelism of processing the data to be processed is in a load state. The specific process of determining whether the parallelism of processing the data to be processed is in a load state is as follows:

[0089] After counting the current number of parallelism values ​​for processing the data to be processed, it is necessary to read the back pressure index corresponding to the current number of parallelism values ​​for processing the data to be processed. The back pressure index represents the percentage of the number of parallelism values ​​for queuing when processing the data to be processed to the number of all parallelism values ​​for processing the data to be processed. After the back pressure index corresponding to the current number of parallelism values ​​for processing the data to be processed exceeds the preset threshold, it means that the current number of parallelism values ​​is in a load state when processing the data to be processed. After the back pressure index corresponding to the current number of parallelism values ​​for processing the data to be processed is lower than the preset threshold, it means that the current number of parallelism values ​​is in a normal state when processing the data to be processed.

[0090] The back pressure indicator can be used to determine whether there is back pressure in the parallelism of the data to be processed, so that the actual parallelism of the data to be processed can be adjusted to optimize the parallelism of the Flink subtasks.

[0091] In the above description, when the parallelism of processing the data to be processed is in a load state, it will cause the parallelism in the Flink subtask to take a long time to process the data to be processed. In order to make the parallelism of processing the data to be processed normal, it is necessary to readjust the number of parallelisms in the Flink subtask for processing the data to be processed. The specific situation of readjusting the number of parallelisms in the Flink subtask for processing the data to be processed is as follows:

[0092] Case 1: when the degree of parallelism for processing the data to be processed is in a normal state, the current quantity value of the degree of parallelism for processing the data to be processed is adjusted based on the third preset rule.

[0093] When the parallelism of processing the data to be processed is in a normal state, the current quantity value of the parallelism of processing the data to be processed that meets the first preset rule is used as the actual quantity value. The first preset rule is to ensure that the resources allocated for the parallelism of processing the data to be processed are not wasted. The first preset rule is that the minimum quantity value in the first quantity value interval corresponding to the Flink subtask is not 1 and the difference between the maximum quantity value and the minimum quantity value is not greater than 1.

[0094] It should be further explained that when the minimum value in the first quantity value interval corresponding to the Flink subtask is 1, it means that the quantity value of the parallelism for processing the data to be processed has not been adjusted. When the difference between the maximum value and the minimum value in the first quantity value interval corresponding to the Flink subtask is greater than 1, it means that the parallelism of the current quantity value allocated by the device is not in the most efficient state when processing the data to be processed. Therefore, the quantity value of the parallelism for processing the data to be processed needs to be reallocated.

[0095] In the above description, the first quantity value interval corresponding to the Flink subtask has been learned. When the parallelism of processing the data to be processed is in a normal state, in order to reallocate the quantity value of the parallelism of processing the data to be processed, it is necessary to obtain the current quantity value corresponding to the normal state of the parallelism of processing the data to be processed. In order to update the first quantity interval corresponding to the Flink subtask, it is also necessary to obtain the first quantity value interval corresponding to the Flink subtask, and extract the maximum quantity value and the minimum quantity value from the first quantity value interval. Since the current parallelism of processing the data to be processed is already in a normal state, it is necessary to reduce the quantity value of the parallelism of processing the data to be processed.

[0096] In order to reduce the number of parallelism values ​​for processing the data to be processed, it is necessary to overwrite the maximum number value described above with the current number value corresponding to the normal state of parallelism for processing the data to be processed. When the maximum number value is added to the current number value, if the sum of the maximum number value and the current number value is an even number, the first added number value is obtained, and the number value corresponding to the average of the first added number value is used as the current number value. If the sum of the maximum number value and the current number value is an odd number, the maximum number value or the current number value is increased by 1 to obtain the first added number value, and the number value corresponding to the average of the first added number value is used as the current number value. At this time, the first number value interval and the current number value corresponding to the Flink subtask have been updated.

[0097] For example, the first quantity value interval of Flink subtask 1 is [2, 16], the preset threshold is 50%, and when the parallelism of processing the data to be processed is normal, the quantity value of the parallelism of processing the data to be processed is adjusted as shown in Table 3:

[0098]

[0099] Table 3

[0100] In Table 3, since Table 3 only describes the process of adjusting the number of parallelism values ​​when the parallelism of processing the data to be processed is in a normal state, the minimum number value in the first number value interval of Flink subtask 1 will not change. In the actual process, the minimum number value will definitely change. Therefore, the first number value interval of Flink subtask 1 is set to [2, 16], and it is determined that the actual number value of the data to be processed by Flink subtask 1 is 3.

[0101] Case 2: When the degree of parallelism for processing the data to be processed is in a load state, the current number value of the degree of parallelism for processing the data to be processed is adjusted based on the second preset rule.

[0102] The above description is about the situation when the parallelism of processing the data to be processed is in a normal state. In order to determine the actual parallelism of the Flink subtask that processes the data to be processed, it is necessary to extract the maximum number value and the minimum number value from the first quantity value interval. Since the current parallelism of processing the data to be processed is in a load state, it is necessary to increase the number of parallelisms for processing the data to be processed.

[0103] When increasing the number of parallelisms for processing the data to be processed, it is necessary to obtain the current number value of the parallelism in the load state during the processing of the data to be processed, overwrite the current number value with the minimum number value in the first number value interval described above, and add the maximum number value to the current number value.

[0104] After adding the maximum quantity value to the current quantity value, in order to make the recalculated quantity value an integer, it is necessary to detect whether the quantity value after adding the maximum quantity value to the current quantity value is an even number. If the quantity value after adding the maximum quantity value to the current quantity value is an even number, the first added quantity value is obtained, and the quantity value corresponding to the first added quantity value after taking the average is used as the current quantity value. If the quantity value after adding the maximum quantity value to the current quantity value is an odd number, the maximum quantity value or the current quantity value described above is added by 1 to obtain the first added quantity value, and then the quantity value corresponding to the first added quantity value after taking the average is used as the current quantity value. At this time, the first quantity value interval and the current quantity value corresponding to the Flink subtask have been updated.

[0105] For example, the first quantity value interval of Flink subtask 1 is [1, 16], the preset threshold is 50%, and the process of adjusting the quantity value of processing parallelism is shown in Table 4:

[0106]

[0107] Table 4

[0108] In Table 4, when the number of parallelism values ​​for processing the data to be processed is 9, 13, and 15, the parallelism for processing the data to be processed is in a load state; when the number of parallelism values ​​is 16, the parallelism for processing the data to be processed is in a normal state, and the first number value interval [15, 16] of the Flink subtask meets the first preset rule, then it is determined that the actual parallelism of the Flink subtask for processing the data to be processed in Flink subtask 1 is 16.

[0109] In the above-described situations 1 and 2, in the actual application of the embodiments of the present application, there is a situation where situation 1 and situation 2 alternate irregularly. If the current number of parallelism values ​​for processing the data to be processed is adjusted in the manner of situation 1, the number of parallelism values ​​for processing the data to be processed that is reallocated by the device according to the second preset rule is obtained, and the back pressure index corresponding to the parallelism for processing the data to be processed after the current number of values ​​is adjusted is obtained again. If the back pressure index is higher than the preset threshold, the number of parallelism values ​​for processing the data to be processed is continued to be adjusted according to the second preset rule. If the back pressure index is lower than the preset threshold, when it is determined that the parallelism for processing the data to be processed does not comply with the first preset rule, the number of parallelism values ​​for processing the data to be processed is readjusted according to the third preset rule. The specific process of adjusting the number of parallelism values ​​for processing the data to be processed refers to the above-mentioned situations 1 and 2, and will not be elaborated on here.

[0110] It should be further explained that, based on Case 1 and Case 2, in the embodiment of the present application, the parallelism of the processing data to be processed corresponding to the minimum back pressure index can be screened out from the parallelism of the processing data to be processed that is lower than the preset threshold, and the quantitative value of the parallelism of the processing data to be processed can be used as the actual quantitative value. Since this is only an optional method, it will not be elaborated here.

[0111] In an embodiment of the present application, the quantity value of the parallelism for processing the data to be processed in each Flink subtask will be adjusted multiple times until the first quantity value interval corresponding to the Flink subtask meets the condition that the minimum quantity value is not 1 and the difference between the maximum quantity value and the minimum quantity value is not greater than 1. The calculated current quantity value will be used as the actual parallelism of the Flink subtask for processing the data to be processed in the Flink subtask. After obtaining the actual parallelism of the Flink subtask for processing the data to be processed, the resource allocation rate corresponding to all the parallelisms of the actual quantity value is read out. The resource allocation rate represents the percentage of the total resources of the device occupied by the resources allocated to the parallelism in the Flink subtask by the device.

[0112] It should be further explained that all the above-mentioned actual number values ​​of parallelism also correspond to a memory occupancy rate, and the memory occupancy rate is the percentage of the memory occupied by all the parallelisms of the actual number values ​​to process the data to be processed in the total memory of the device.

[0113] Through the method described above, the range of the first quantity value interval is continuously narrowed through the second preset rule and the third preset rule, and the actual parallelism of the Flink subtask that processes the data to be processed is determined by the first preset rule. When the parallelism of processing the data to be processed is in a load state, the quantity value of the parallelism in the load state covers the minimum quantity value of the first quantity interval. When the parallelism of processing the data to be processed is in a normal state, the quantity value of the parallelism in the normal state covers the maximum quantity value of the first quantity interval, thereby realizing the update of the first quantity value interval, ensuring that the current quantity value calculated based on the updated proposed quantity value interval is the optimal quantity value, and further optimizing the performance of stream computing.

[0114] Step S23: Filter out the target resource allocation rate and the actual parallelism of the Flink subtask corresponding to the target resource allocation rate from the N resource allocation rates.

[0115] After determining the actual parallelism of a Flink subtask that processes the data to be processed in a Flink subtask and the resource allocation rate corresponding to the subtask thread group, since the process of processing the data to be processed by the parallelism in each Flink subtask is consistent, only the quantity values ​​of all the parallelisms for processing the data to be processed are different, the embodiment of the present application only describes the specific process of processing the data to be processed by the parallelism of a Flink subtask, and the specific process of processing the data to be processed by the parallelism in other Flink subtasks refers to the specific process of processing the data to be processed by the parallelism in a Flink subtask in the embodiment of the present application, and no further elaboration is made here.

[0116] After describing the process of processing the data to be processed in parallel in other Flink subtasks, the method described in the above steps is applied to the remaining N-1 Flink subtasks in turn to obtain the resource allocation rate corresponding to each Flink subtask, the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask, and the memory usage.

[0117] To avoid device freezes caused by excessive memory usage, you need to delete the Flink subtasks whose memory usage exceeds the preset memory usage, and use the remaining Flink subtasks as the first Flink subtasks.

[0118] In order to determine the target resource allocation rate and the actual parallelism of the first Flink subtask that processes the data to be processed corresponding to the target resource allocation rate, it is necessary to detect whether the resource allocation rate of each first Flink subtask is within a preset range. When the resource allocation rate of the first Flink subtask is within the preset range, calculate the difference between the resource allocation rate within the preset range and the resource allocation rate corresponding to the target endpoint value of the preset range.

[0119] It should be noted that, in the embodiment of the present application, the resource allocation rate corresponding to the target endpoint value may be the minimum resource allocation rate within a preset range, and the case where the target resource allocation rate is other resource allocation rates is not described here.

[0120] After calculating the difference between the resource allocation rate within the preset range and the minimum resource allocation rate corresponding to the preset range, the resource allocation rate corresponding to the minimum difference is used as the target resource allocation rate, and the actual parallelism of the first Flink subtask processing the data to be processed corresponding to the target resource allocation rate is obtained.

[0121] The above description is about the situation that the resource allocation rate is within the preset range. When the resource allocation rate is not within the preset range, the difference between each resource allocation rate and the minimum resource allocation rate in the preset range is still calculated, and the resource allocation rate corresponding to the minimum difference is used as the target resource allocation rate, and the actual parallelism of the first Flink subtask processing the data to be processed corresponding to the target resource allocation rate is obtained.

[0122] Through the method described above, N Flink subtasks are set, and it is ensured that the parallelism in each Flink subtask can fully process the data to be processed, thereby realizing the estimation of the number of parallelism values ​​required to complete the processing of the data to be processed. In the process of the parallelism in each Flink subtask processing the data to be processed, the number of parallelism values ​​for processing the data to be processed is continuously adjusted through the second preset rule and the third preset rule, so as to determine the resource allocation rate corresponding to each Flink subtask and the actual parallelism of the Flink subtask processing the data to be processed, thereby ensuring that the resource allocation rate and the actual number value are the optimal parameters corresponding to the Flink subtask, and then filtering out the target resource allocation rate from the obtained N resource allocation rates, and obtaining the actual parallelism of the Flink subtask processing the data to be processed corresponding to the target resource allocation rate, thereby realizing multi-layer screening of the resource allocation rate, thereby ensuring that the target resource allocation rate and the actual number value are finally obtained. The optimal parameters corresponding to the device, the performance of stream computing is improved, and the purpose of optimizing stream computing is achieved.

[0123] Based on the same inventive concept, a parameter estimation device is also provided in the embodiment of the present application. The thread-bound device is used to implement the function of a parameter estimation method. Figure 3 , the device comprises:

[0124] An acquisition module 301 is used to obtain N Flink subtasks;

[0125] A processing module 302 is used to process the data to be processed based on the parallelism in each Flink subtask, obtain the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask and the resource allocation rate of the Flink subtask, and obtain N resource allocation rates;

[0126] The screening module 303 is used to screen out the target resource allocation rate and the actual parallelism of the Flink subtask corresponding to the target resource allocation rate from the N resource allocation rates.

[0127] In one possible design, the screening module 303 is specifically used to obtain the resource allocation rates of N Flink subtasks, detect whether each resource allocation rate is within a preset range, and if so, use the minimum resource allocation rate within the preset range as the target resource allocation rate, and obtain the actual parallelism of the Flink subtask corresponding to the target resource allocation rate; if not, extract the target endpoint value in the preset range, calculate the difference between the resource allocation rate and the target endpoint value, use the resource allocation rate corresponding to the minimum difference as the target resource allocation rate, and obtain the actual parallelism of the Flink subtask corresponding to the target resource allocation rate.

[0128] In a possible design, the processing module 302 is specifically used to read the current number value of the parallelism of processing the data to be processed in each Flink subtask, and obtain the back pressure index corresponding to the parallelism of the current number value, and adjust the current number value of all parallelisms of processing the data to be processed according to the relationship between the back pressure index and the preset threshold value, until the back pressure index is lower than the preset threshold value, and the current number value of the parallelism that meets the first preset rule is used as the actual number value;

[0129] In a possible design, the processing module 302 is also used to read a first quantity value interval corresponding to each Flink subtask, update the first quantity value interval based on the current quantity value of all parallelisms of processing the data to be processed, and when the minimum quantity value of the first quantity value interval is not 1 and the difference between the maximum quantity value and the minimum quantity value in the first quantity value interval does not exceed 1, calculate the actual parallelism of the Flink subtask.

[0130] In one possible design, the processing module 302 is also used to obtain back pressure indicators of all parallelisms for processing the data to be processed in different first quantity value intervals, arrange the back pressure indicators according to a preset arrangement order, select the first quantity value interval corresponding to the minimum back pressure indicator, and calculate the actual parallelism of all Flink subtasks that process the data to be processed based on the first quantity value interval.

[0131] In one possible design, the processing module 302 is also used to detect whether the back pressure index of all parallelisms of processing the data to be processed exceeds a preset threshold. If so, the current quantity values ​​corresponding to all parallelisms of processing the data to be processed are obtained, and the current quantity values ​​of all parallelisms of processing the data to be processed are adjusted based on the second preset rule. If not, when it is determined that the quantity values ​​of all parallelisms of processing the data to be processed are greater than 1 and the difference between the maximum quantity value and the minimum quantity value in the first quantity value interval is greater than 1, the current quantity values ​​of all parallelisms of processing the data to be processed are adjusted based on the third preset rule.

[0132] In a possible design, the processing module 302 is also used to parse out the maximum quantity value and the minimum quantity value in the first quantity value interval corresponding to the Flink subtask, overwrite the minimum quantity value with the current quantity value of the parallelism of processing the data to be processed, add the maximum quantity value to the current quantity value to obtain a first added quantity value, and take the quantity value corresponding to the average of the first added quantity value as the current quantity value.

[0133] In one possible design, the processing module 302 is also used to, if the maximum number value or the current number value is an odd number, add 1 to the maximum number value or the current number value to obtain a first added number value; if the maximum number value and the current number value are even numbers, add the maximum number value and the current number value to obtain a first added number value.

[0134] In a possible design, the processing module 302 is also used to parse out the maximum quantity value and the minimum quantity value in the first quantity value interval corresponding to the Flink subtask, overwrite the maximum quantity value with the current quantity value of the parallelism of processing the data to be processed, add the current quantity value to the minimum quantity value to obtain a second added quantity value, and take the quantity value corresponding to the average of the second added quantity value as the current quantity value.

[0135] In one possible design, the processing module 302 is also used to, if the minimum quantity value or the current quantity value is an odd number, add 1 to the minimum quantity value or the current quantity value to obtain a second added quantity value; if the minimum quantity value and the current quantity value are even numbers, add the minimum quantity value and the current quantity value to obtain a second added quantity value.

[0136] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the function of the aforementioned parameter estimation device, referring to Figure 4 , the electronic device comprises:

[0137] At least one processor 401, and a memory 402 connected to the at least one processor 401. The specific connection medium between the processor 401 and the memory 402 is not limited in the embodiment of the present application. Figure 4 In the example, the processor 401 and the memory 402 are connected via the bus 400. The bus 400 is Figure 4 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 400 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 401 can also be called a controller, and there is no limitation on the name.

[0138] In the embodiment of the present application, the memory 402 stores instructions that can be executed by at least one processor 401. The at least one processor 401 can execute a parameter estimation method discussed above by executing the instructions stored in the memory 402. The processor 401 can implement Figure 3 The functions of each module in the device shown.

[0139] Among them, the processor 401 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 402 and calling the data stored in the memory 402, the various functions of the device and process data, the device can be monitored as a whole.

[0140] In one possible design, the processor 401 may include one or more processing units, and the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0141] Processor 401 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of a parameter estimation method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.

[0142] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 402 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 402 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0143] By programming the processor 401, the code corresponding to the parameter estimation method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1The step of parameter estimation in the embodiment shown is as follows: How to design and program the processor 401 is a technique known to those skilled in the art and will not be described in detail here.

[0144] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes a parameter estimation method discussed above.

[0145] In some possible implementations, various aspects of a parameter estimation method provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of a parameter estimation method according to various exemplary implementations of the present application described above in this specification.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A parameter estimation method, characterized in that: The method comprises: Obtain N Flink subtasks, where N is a positive integer, the Flink subtask has multiple degrees of parallelism, and the number of degrees of parallelism in each Flink subtask is inconsistent; Process the data to be processed based on the parallelism in each Flink subtask, obtain the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask and the resource allocation rate of the Flink subtask, and obtain N resource allocation rates, wherein the resource allocation rate is the percentage of resources allocated by the device to all the parallelisms that process the data to be processed in each Flink subtask; Filter out a target resource allocation rate and an actual parallelism of the Flink subtask corresponding to the target resource allocation rate from the N resource allocation rates; The obtaining of the actual parallelism of each Flink subtask for processing the data to be processed includes: Read the current number value of the parallelism of processing the data to be processed in each Flink subtask, and obtain the back pressure index corresponding to the parallelism of the current number value, wherein the back pressure index is the percentage of the number value of the parallelism for queuing when processing the data to be processed in each Flink subtask to the number value of all parallelisms for processing the data to be processed; According to the relationship between the back pressure indicator and the preset threshold, the current number of values ​​of all parallelisms for processing the data to be processed is adjusted until the back pressure indicator is lower than the preset threshold. The current number of values ​​of parallelism that meets the first preset rule is used as the actual number value, and the first preset rule is that the minimum number value in the first number value interval corresponding to the Flink subtask is not 1 and the difference between the maximum number value and the minimum number value is not greater than 1.

2. The method according to claim 1, characterized in that Filtering out a target resource allocation rate and an actual parallelism of the Flink subtask corresponding to the target resource allocation rate from the N resource allocation rates includes: Get the resource allocation rates of N Flink subtasks and check whether each resource allocation rate is within the preset range; If yes, the minimum resource allocation rate within the preset range is used as the target resource allocation rate, and the actual parallelism of the Flink subtask corresponding to the target resource allocation rate is obtained; If not, extract the target endpoint value in the preset range, calculate the difference between the resource allocation rate and the target endpoint value, take the resource allocation rate corresponding to the minimum difference as the target resource allocation rate, and obtain the actual parallelism of the Flink subtask corresponding to the target resource allocation rate.

3. The method according to claim 1, characterized in that The current number value of the parallelism that meets the first preset rule is used as the actual number value, including: Read a first quantity value interval corresponding to each Flink subtask, and update a maximum quantity value or a minimum quantity value in the first quantity value interval based on the current quantity values ​​of all parallelisms of processing the data to be processed; When the minimum quantity value of the updated first quantity value interval is not 1 and the difference between the maximum quantity value and the minimum quantity value in the updated first quantity value interval does not exceed 1, the actual parallelism of the Flink subtask is calculated.

4. The method according to claim 1, characterized in that The current number value of the parallelism that meets the first preset rule is used as the actual number value, including: Obtaining back pressure indicators of all parallelisms for processing data to be processed in different first quantity value intervals; The back pressure indicators are arranged according to a preset arrangement order, a first quantity value interval corresponding to a minimum back pressure indicator is selected, and the actual parallelism of all Flink subtasks that process the data to be processed is calculated based on the first quantity value interval.

5. The method according to claim 1, characterized in that Adjusting the current number of parallelism values ​​according to the relationship between the back pressure indicator and the preset threshold value includes: Check whether the back pressure index of all parallelisms of the data to be processed exceeds the preset threshold; If so, obtaining the current number values ​​corresponding to all the parallelisms of processing the data to be processed, and adjusting the current number values ​​of all the parallelisms of processing the data to be processed based on the second preset rule; If not, when it is determined that the number of all parallelisms for processing the data to be processed is greater than 1 and the difference between the maximum number value and the minimum number value in the first number value interval is greater than 1, the current number of all parallelisms for processing the data to be processed is adjusted based on a third preset rule; The step of adjusting the current number of all parallelisms of processing the data to be processed based on the second preset rule includes: Parse the maximum and minimum values ​​in the first quantity value interval corresponding to the Flink subtask; overwrite the minimum value with the current value of the parallelism of processing the data to be processed; add the maximum value to the current value to obtain a first added value, and take the value corresponding to the average of the first added value as the current value; The step of adjusting the current number of all parallelisms of processing the data to be processed based on the third preset rule includes: Parse the maximum and minimum values ​​in the first quantity value interval corresponding to the Flink subtask; overwrite the maximum quantity value with the current quantity value of the parallelism of processing the data to be processed; add the current quantity value to the minimum quantity value to obtain a second added quantity value, and take the quantity value corresponding to the average of the second added quantity value as the current quantity value.

6. The method according to claim 5, characterized in that Adding the maximum quantity value to the current quantity value to obtain a first added quantity value includes: If the maximum number value or the current number value is an odd number, adding 1 to the maximum number value or the current number value to obtain a first added number value; If the maximum quantity value and the current quantity value are even numbers, the maximum quantity value and the current quantity value are added to obtain a first added quantity value.

7. The method according to claim 5, characterized in that Adding the current quantity value to the minimum quantity value to obtain a second added quantity value comprises: If the minimum quantity value or the current quantity value is an odd number, adding 1 to the minimum quantity value or the current quantity value to obtain a second added quantity value; If the minimum quantity value and the current quantity value are even numbers, the minimum quantity value and the current quantity value are added to obtain a second added quantity value.

8. A parameter estimation device, characterized in that: The device comprises: An acquisition module is used to obtain N Flink subtasks, where N is a positive integer, the Flink subtask has multiple degrees of parallelism, and the number of parallelism values ​​in each Flink subtask is inconsistent; A processing module is used to process the data to be processed based on the parallelism in each Flink subtask, obtain the actual parallelism of the Flink subtask that processes the data to be processed in each Flink subtask and the resource allocation rate of the Flink subtask, and obtain N resource allocation rates; A screening module is used to screen out a target resource allocation rate and an actual parallelism of the Flink subtask corresponding to the target resource allocation rate from N resource allocation rates; Wherein, the processing module is specifically used for: Read the current number value of the parallelism of processing the data to be processed in each Flink subtask, and obtain the back pressure index corresponding to the parallelism of the current number value, wherein the back pressure index is the percentage of the number value of the parallelism for queuing when processing the data to be processed in each Flink subtask to the number value of all parallelisms for processing the data to be processed; According to the relationship between the back pressure indicator and the preset threshold, the current number of values ​​of all parallelisms for processing the data to be processed is adjusted until the back pressure indicator is lower than the preset threshold. The current number of values ​​of parallelism that meets the first preset rule is used as the actual number value, and the first preset rule is that the minimum number value in the first number value interval corresponding to the Flink subtask is not 1 and the difference between the maximum number value and the minimum number value is not greater than 1.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images