Resource configuration methods, apparatuses, media, and computing devices

Abstract

Embodiments of the present disclosure provide a resource configuration method. The method comprises: obtaining a plurality of message groups to be read by a task to be executed and historical network parameters of a distributed system executing the task to be executed, the message groups comprising a plurality of message units; determining a first quantity of unit resources required for processing each of the message units according to the historical network parameters; and allocating corresponding message units to each working node of the distributed system according to the first quantity of unit resources required for processing each of the message units, wherein the sum of the first quantity of unit resources required for processing each of the message units allocated to the working node is less than or equal to the quantity of unit resources of the working node. The present disclosure shortens the execution time of a stream task and improves the execution efficiency of the stream task without manually adjusting the allocation of message units. Furthermore, embodiments of the present disclosure provide a resource configuration apparatus, a medium and a computing device.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of distributed systems, and more specifically, embodiments of the present disclosure relate to a resource configuration method, device, medium and computing device. BACKGROUND

[0002] This section is intended to provide a background or context to the embodiments of the disclosure recited in the claims. The description herein does not constitute an admission that any of the information provided herein is prior art.

[0003] A stream task refers to a plurality of tasks with association, and the stream task can be executed by a distributed system.

[0004] After the stream task is created, the stream task reads a plurality of message groups, each message group can be divided into a plurality of message units, and the message group includes a plurality of messages of the same type. Each message unit is allocated to each worker node of the distributed system in a uniform manner for processing.

[0005] The data amount of each message group is different, and the data amount of the message units of different message groups is also different. Since the message units cannot be divided, the message units are as evenly distributed as possible to the worker nodes. If there are a plurality of message groups, the worker nodes will be allocated to the message units of different message groups. The data amount of each message group is different, and thus the data amount processed by the message units of different message groups is also different, resulting in a large data amount of the message units allocated to the worker nodes. If this situation occurs, the message units of the worker nodes with a large processing amount need to be manually allocated to the worker nodes with a small processing amount; or a new worker node is opened, and the message units of the worker nodes with a large processing amount are manually allocated to the newly opened worker node, so that the execution time of the stream task is too long, resulting in low execution efficiency of the stream task. SUMMARY

[0006] The present disclosure provides a resource configuration method, device, medium and computing device to solve the problem of low execution efficiency of the stream task.

[0007] In a first aspect of the embodiments of the present disclosure, a resource configuration method is provided, including: obtaining a plurality of message groups to be read by a to-be-executed task and historical network parameters of a distributed system for executing the to-be-executed task, the message groups including a plurality of message units, and the message units including one or more messages; determining a first number of unit resources required for processing each message unit according to the historical network parameters; and allocating corresponding message units to each worker node of the distributed system according to the first number of unit resources required for processing each message unit, wherein the sum of the first number of unit resources required for processing each message unit allocated to the worker node is less than or equal to the number of unit resources of the worker node.

[0008] In an embodiment of the present disclosure, the method further includes: sorting each message unit in descending order according to a first quantity of unit resources required for processing each message unit; and assigning each sorted message unit to a corresponding working node, wherein a remaining quantity of unit resources of the working node after the assignment is less than a minimum first quantity.

[0009] In another embodiment of the present disclosure, the method further includes: determining a second quantity of unit resources required for processing each message group according to the historical network parameters; and determining the first quantity of unit resources required for processing each message unit in the message group according to the second quantity and a quantity of message units in the message group, wherein the message group includes a plurality of message units with equal data quantities.

[0010] In another embodiment of the present disclosure, the method further includes: obtaining a serialization parameter of the message group; inputting the historical network parameters and the serialization parameter of the message group into a prediction model; and determining the second quantity of unit resources required for processing the message group according to a result output by the prediction model.

[0011] In another embodiment of the present disclosure, the prediction model is trained according to network parameters of the distributed system in a target time period, a quantity of unit resources required for the distributed system to perform a task in the target time period, and a serialization parameter of a message group in the target time period, the target time period is a time period corresponding to adjacent first and second sampling time points, the distributed system does not have a delay at the first sampling time point, and the distributed system has a delay at the second sampling time point.

[0012] In another embodiment of the present disclosure, before the assigning of each message unit to each working node of the distributed system according to the first quantity of unit resources required for processing each message unit, the method further includes: if a maximum first quantity is greater than a quantity of unit resources set by each working node, sending prompt information to each working node, the prompt information being used to indicate an increase in the quantity of unit resources of each working node.

[0013] In another embodiment of the present disclosure, before the obtaining of the plurality of message groups to be read for performing a task, the method further includes: obtaining basic information of the task to be performed; and performing parameter configuration on each working node according to the basic information.

[0014] In another embodiment of the present disclosure, after the first quantity of unit resources required for processing each message unit is determined according to the historical network parameters, the method further comprises: obtaining a task index of the task to be executed; and if the task index matches a preset diagnosis rule, outputting diagnosis suggestion information corresponding to the preset diagnosis rule.

[0015] In another embodiment of the present disclosure, after the task index of the task to be executed is obtained, the method further comprises: if the task index does not match a preset diagnosis rule, inputting the task index into a diagnosis model; and if a probability value of parameter modification output by the diagnosis model is greater than a preset probability value, outputting prompt information of parameter modification.

[0016] In a second aspect of the embodiments of the present disclosure, a resource configuration apparatus is further provided, comprising: an obtaining module configured to obtain a plurality of message groups to be executed and historical network parameters of a distributed system for executing the task to be executed, wherein each message group comprises a plurality of message units, and each message unit comprises one or more messages; a determining module configured to determine a first quantity of unit resources required for processing each message unit according to the historical network parameters, wherein each message group comprises a plurality of message units, and each message unit comprises one or more messages; and an allocating module configured to allocate corresponding message units to each working node in the distributed system according to the first quantity of unit resources required for processing each message unit, wherein a sum of the first quantity of unit resources required for processing each message unit allocated to the working node is less than or equal to a quantity of unit resources of the working node.

[0017] In an embodiment of the present disclosure, the apparatus further comprises: an ordering module configured to order each message unit from large to small according to the first quantity of unit resources required for processing each message unit; and the allocating module is further configured to allocate each ordered message unit to each working node, wherein a quantity of unit resources remaining in the working node after the allocation of the message units is less than the smallest first quantity.

[0018] In another embodiment of the present disclosure, the determining module is further configured to determine a second quantity of unit resources required for processing each message group according to the historical network parameters; and the determining module is further configured to determine the first quantity of unit resources required for processing each message unit in the message group according to the second quantity and a quantity of message units in the message group, wherein each message group comprises a plurality of message units with equal data quantities.

[0019] In another embodiment of the present disclosure, the obtaining module is further configured to obtain serialization parameters of the message group; the input module is configured to input the historical network parameters and the serialization parameters of the message group into a prediction model; and the determining module is further configured to determine the second number of unit resources required for processing the message group according to a result output by the prediction model.

[0020] In another embodiment of the present disclosure, the prediction model is trained according to network parameters of the distributed system in a target time period, a number of unit resources required by the distributed system for performing a task in the target time period, and serialization parameters of a message group in the target time period, the target time period being a time period corresponding to adjacent first and second sampling time points, the distributed system not having a delay at the first sampling time point, and the distributed system having a delay at the second sampling time point.

[0021] In another embodiment of the present disclosure, the sending module is configured to send prompt information to each of the worker nodes if the maximum first number is greater than the number of unit resources set by the worker node, the prompt information being used to indicate increasing the number of unit resources of the worker node.

[0022] In another embodiment of the present disclosure, the obtaining module is further configured to obtain basic information of the task to be performed; and the configuration module is configured to perform parameter configuration on each of the worker nodes according to the basic information.

[0023] In another embodiment of the present disclosure, the obtaining module is further configured to obtain a task index of the task to be performed; and the output module is configured to output diagnosis suggestion information corresponding to a preset diagnosis rule if the task index matches the preset diagnosis rule.

[0024] In another embodiment of the present disclosure, the input module is configured to input the task index into a diagnosis model if the task index does not match the preset diagnosis rule; and the output module is further configured to output prompt information of parameter modification if a probability value of parameter modification output by the diagnosis model is greater than a preset probability value.

[0025] In a third aspect of the embodiments of the present disclosure, a medium is further provided, including computer execution instructions, which are configured to be executed by a processor to implement the resource configuration method.

[0026] In a fourth aspect of the embodiments of the present disclosure, a computing device is further provided, including a memory and a processor; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory, so that the processor executes the resource configuration method.

[0027] According to the embodiments of the present disclosure, the number of unit resources required for processing the message units of the message group is determined by the historical network parameters of the distributed system, and each working node is allocated with the corresponding message units according to the corresponding number of each message unit, so that the total number of unit resources required for processing each message unit allocated to the working node is less than the number of unit resources provided by the working node, avoiding the situation that the resources required for processing the message units allocated to the working node exceed the resources of the working node, and thus the process of manually adjusting the allocation of message units is eliminated, the execution time of the stream task is shortened, and the execution efficiency of the stream task is improved. BRIEF DESCRIPTION OF DRAWINGS

[0028] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which a number of embodiments of the present disclosure are illustrated by way of example and not limitation. In the drawings:

[0029] FIG. 1 A system architecture schematic diagram of an application scenario of a resource configuration method according to an embodiment of the present disclosure is schematically shown;

[0030] FIG. 2 A flowchart according to an embodiment of the present disclosure is schematically shown;

[0031] FIG. 3 A flowchart according to another embodiment of the present disclosure is schematically shown;

[0032] FIG. 4 A flowchart according to still another embodiment of the present disclosure is schematically shown;

[0033] FIG. 5 A flowchart according to yet another embodiment of the present disclosure is schematically shown;

[0034] FIG. 6 A flowchart according to still another embodiment of the present disclosure is schematically shown;

[0035] FIG. 7 A schematic diagram of a program product according to an embodiment of the present disclosure is schematically shown;

[0036] FIG. 8 A structure schematic diagram of a resource configuration apparatus according to an embodiment of the present disclosure is schematically shown;

[0037] FIG. 9 A structure schematic diagram of a computing device according to an embodiment of the present disclosure is schematically shown.

[0038] In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts. DETAILED DESCRIPTION

[0039] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only so that those skilled in the art can better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided so that the present disclosure is more thorough and complete, and the scope of the present disclosure is fully conveyed to those skilled in the art.

[0040] Those skilled in the art will appreciate that the embodiments of the present disclosure can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure can be embodied in the form of entire hardware, entire software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0041] According to embodiments of the present disclosure, a resource configuration method, device, medium and computing device are proposed.

[0042] In addition, the number of any elements in the drawings is used for illustration only and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0043] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only so that those skilled in the art can better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided so that the present disclosure is more thorough and complete, and the scope of the present disclosure is fully conveyed to those skilled in the art. SUMMARY

[0045] The resource configuration device reads a plurality of message groups when performing a stream task, and then divides the message groups into message units, and finally allocates the message units to the worker nodes of the distributed system.

[0046] The present inventors have found that the message groups are divided into a plurality of message units, and the message units are as evenly divided as possible among the worker nodes. However, the data volume of each message group is different, resulting in different data volume of the message units under different message groups, thereby causing the data volume of the message units allocated to some worker nodes to be large, and the message units of the worker nodes need to be manually adjusted, thereby causing the execution time of the stream task to be too long.

[0047] The present inventors have therefore thought that, after reading the message groups, the number of unit resources required for processing the message units of the message groups is determined, and the corresponding message units are allocated to each worker node through the number corresponding to each message unit, so that the sum of the unit resources required for processing each message unit allocated to the worker nodes is less than the number of unit resources set by the worker nodes, avoiding the resource required for processing the message units allocated to the worker nodes exceeding the resource of the worker nodes, and thereby eliminating the process of manually adjusting the allocation of the message units, shortening the execution time of the stream task, and improving the execution efficiency of the stream task.

[0048] OVERVIEW OF APPLICATION SCENARIOS

[0049] First refer to FIG. 1 , FIG. 1 This is a schematic diagram illustrating an application scenario of the resource allocation method according to an embodiment of this disclosure. The distributed system includes worker nodes 110, 120, and 130. It is assumed that each worker node has a unit resource of 1, and the resources of each worker node are considered as a bucket. The resource allocation device 200 reads message groups A, B, and C. Message group A is divided into two message units a, message group B into four message units b, and message group C into four message units c. The resource allocation device 200 calculates that the number of unit resources required to process each message unit a is 0.5, the number of unit resources required to process each message unit b is 0.25, and the number of unit resources required to process each message unit c is 0.2. The principle for allocating message units to each worker node is that the sum of the number of unit resources of each message unit allocated to a worker node does not exceed 1 (1 being the number of unit resources of a worker node). FIG. 1 As shown, the resource allocation device 200 allocates message units as follows: worker node 110 allocates 2 message units a, worker node 120 allocates 4 message units b, and worker node 130 allocates 4 message units c. It should be noted that the message unit allocation method is not limited to... FIG. 1 The allocation method shown can be further divided into several other allocation methods, provided that the sum of the unit resources of each message unit allocated to the worker node does not exceed 1. These methods will not be listed in detail here.

[0050] EXEMPLARY METHOD

[0051] The following is combined FIG. 1 Application scenarios, refer to FIGS. 2-6 This document describes a resource allocation method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0052] For example, the resource configuration device involved in the embodiments of this disclosure can be any device with data processing capabilities, such as a computer.

[0053] refer to FIG. 2 , FIG. 2 An exemplary flowchart of an embodiment of a resource configuration method provided according to this disclosure is shown. The resource configuration method includes:

[0054] Step S201, obtain a plurality of message groups to be read by the to-be-executed task and historical network parameters of the distributed system executing the to-be-executed task, the message group comprising a plurality of message units, and each message unit comprising one or more messages.

[0055] In the embodiment, the execution subject is a resource configuration device, which can be any device with data processing capability. For example, the resource configuration device can be a device provided with a Filnk, which is a real-time computing processing engine and can be used to execute a stream task. For ease of description, the device is used to represent the resource configuration device in the following description.

[0056] A user can create a stream task on a stream task management platform, and the stream task management platform submits the stream task to the device, and the submitted stream task is the to-be-executed task of the device. After obtaining the to-be-executed task, the device reads a plurality of message groups of the to-be-executed task. The message group is a Topic, which can be a collection of a plurality of messages belonging to the same class, that is, the message group comprises a plurality of messages belonging to the same class. Each message group can be divided into a plurality of message units, and the message unit is a partition, which refers to the smallest storage unit and comprises part of the data of the message group. In an actual scenario, the data in the partition cannot be divided, and the partition is the smallest granularity of the message group, that is, each partition can only be read by one worker node, and cannot be read by multiple worker nodes. The message unit comprises one or more messages.

[0057] The device is provided with a plug-in, and the historical network parameters of the distributed system can be read through the plug-in. The historical network parameters comprise information such as the flow rate and the stream rate of the distributed system processing the message group, that is, the historical network parameters are parameters such as the historical flow rate and the historical stream rate of the message group. For example, the message group is a Topic composed of a plurality of messages of type A, and the device reads historical network parameters such as the historical flow rate and the historical stream rate of the distributed system processing the message group of type A.

[0058] Step S202, determining a first number of unit resources required for processing each message unit according to the historical network parameters.

[0059] The device can determine the number of unit resources required for processing the message unit of each message group based on the historical network parameters, and the number is defined as the first number. The unit resource refers to one resource, and the size of one resource can be set. Alternatively, the resource possessed by a worker node can be set as one unit resource, and each worker node in the distributed system has the same resource.

[0060] In an example, the apparatus determines an average network parameter required for processing the message group based on historical network parameters of the message group, such as an average traffic, an average flow rate, etc. The apparatus can convert the average network parameter to obtain an average number of unit resources required for processing the message group. Since the average number has a deviation, the average number can be appropriately increased, and the increased average number is divided by the number of message units into which the message group is divided to obtain the first number.

[0061] In step S203, the first number of unit resources required for processing each message unit is used to allocate corresponding message units to each working node of the distributed system, and the sum of the first number of unit resources required for processing each message unit allocated to the working node is less than or equal to the number of unit resources of the working node.

[0062] After determining the first number of each message unit, the apparatus allocates corresponding message units to each working node of the distributed system based on the first number of each message unit. It should be noted that the sum of the first number of unit resources required for processing each message unit allocated to the working node needs to be less than or equal to the number of unit resources of the working node. The following is an example.

[0063] The message group D is divided into 2 message units d, and the first number of the message unit d is 0.75; the message group E is divided into 3 message units e, and the first number of the message unit e is 0.5; and the message group F is divided into 4 message units f, and the first number of the message unit f is 0.25. The number of unit resources of each working node is 1.

[0064] The sum of the first number of each message unit M = 0.75*2 + 0.5*3 + 0.25*4 = 4, that is, at least 4 working nodes are required to process all the message units. The optimal allocation mode is that 2 working nodes are allocated with 1 message unit d and 1 message unit f, 1 working node is allocated with 2 message units e, and 1 working node is allocated with 1 message unit e and 2 message units f, that is, the sum of the first number of each message unit allocated to each working node is 1.

[0065] Of course, more working nodes can be used to allocate multiple message units, for example, 5 working nodes are used to allocate message units. Specifically, 2 working nodes are allocated with 1 message unit d, 1 working node is allocated with 2 message units e, 1 working node is allocated with 1 message unit e and 2 message units f, and the last working node is allocated with 2 message units f, that is, the sum of the first number of message units of each working node is less than 1.

[0066] In the embodiment, the number of unit resources required for processing the message units of the message group is determined by the historical network parameters of the distributed system, the corresponding message units are allocated to the respective working nodes according to the corresponding numbers of the respective message units, so that the total number of unit resources required for processing the respective message units allocated to the working nodes is less than the number of unit resources set by the working nodes, the resource required for processing the message units allocated to the working nodes is prevented from exceeding the resource of the working nodes, and the process of manually adjusting the allocation of the message units is eliminated, the execution time of the stream task is shortened, and the execution efficiency of the stream task is improved.

[0067] With reference to FIG. 3 , FIG. 3 The flowchart of another embodiment of the resource configuration method provided by the present disclosure is based on FIG. 2 As shown in the embodiment, step S203 comprises:

[0068] In step S301, the respective message units are sorted in descending order according to the first number of unit resources required for processing the respective message units.

[0069] In the embodiment, the device can allocate the message units to the respective working nodes in a way of local optimal solution.

[0070] Specifically, the device sorts the respective message units in descending order according to the first number corresponding to the respective message units.

[0071] In step S302, the sorted respective message units are allocated to the respective working nodes, wherein the number of unit resources remaining in the working nodes after completing the allocation of the message units is less than the smallest first number.

[0072] The device allocates the sorted respective message units to the respective working nodes. It should be noted that the local optimal solution in the embodiment can be realized by a greedy algorithm, that is, the number of unit resources remaining in the working nodes after completing the allocation of the message units is less than the smallest first number. The following is an example.

[0073] Suppose there are 2 message units g, the first quantity of message units g is 0.75; 3 message units h, the first quantity of message units h is 0.5; 4 message units j, the first quantity of message units j is 0.25; 3 message units k, the first quantity of message units k is 0.2. The quantity of unit resources of each worker node is 1. Then the message units are arranged in descending order as follows: message unit g, message unit g, message unit h, message unit h, message unit h, message unit j, message unit j, message unit j, message unit j, message unit j, message unit k, message unit k, message unit k. The device first allocates the first message unit g to worker node 1, the quantity of unit resources remaining in worker node 1 is 0.25, and then the device allocates the second message unit g to worker node 2, the quantity of unit resources remaining in worker node 2 is 0.25, and worker nodes 1 and 2 cannot be allocated 0.5 message unit h, so the device allocates the first two message units h to worker node 3. The device then allocates the third message unit h to worker node 4. At this time, the arranged message units also include 4 message units j and 3 message units k, and the device allocates one message unit j to worker node 1 and the second message unit j to worker node 2, so that worker nodes 1, 2, 3 and 4 have completed allocation. The device then allocates the third, fourth message units j, the first message unit k and the second message unit k to worker node 5, and there is still one message unit k left. Since the sum of the first quantities of the allocated message units of worker node 5 is 0.25*2+2*0.2=0.9, the quantity of unit resources remaining in worker node 5 is 0.1, and the last message unit k cannot be allocated to worker node 5, so the device allocates the message unit k to worker node 6. It should be noted that the quantity of unit resources set for the worker node can be referred to as 1 concurrent computing power.

[0074] As shown in the above example, the quantity of unit resources remaining in worker nodes 1-4 is 0, and the quantity of unit resources remaining in worker node 5 is 0.1, that is, the quantity of unit resources remaining in worker nodes 1-5 is less than the minimum first quantity 0.2. The greedy intention is to allocate each message unit to the worker node until the quantity of unit resources remaining in the worker node is less than the first quantity of any unallocated message unit, while ensuring that the sum of the first quantities of the allocated message units of the worker node does not exceed the quantity of unit resources of the worker node.

[0075] In the embodiment, the device sorts the message units based on the first quantity of each message unit, and then allocates the sorted message units to the working nodes, so that the remaining quantity of unit resources of the working nodes after the allocation of the message units is less than the smallest first quantity, and the working nodes are allocated with the maximum message units in the case that the sum of the first quantities of the message units allocated to the working nodes is less than the quantity of unit resources of the working nodes.

[0076] With reference to FIG. 4 , FIG. 4 FIG. 1 is a flowchart of a method for resource configuration according to an embodiment of the present disclosure. FIG. 2 or FIG. 3 In the embodiment, the device can determine the second quantity of unit resources required for processing each message group based on the historical network parameters.

[0077] In step S401, the second quantity of unit resources required for processing each message group is determined according to the historical network parameters.

[0078] In the embodiment, the device can determine the second quantity of unit resources required for processing each message group based on the historical network parameters.

[0079] In an example, the second quantity can be calculated in the manner of the determination of the average quantity, which will not be described herein.

[0080] In another example, the device can determine the second quantity of unit resources required for processing each message group by using a prediction model, which can be a regression model. The device obtains the historical network parameters corresponding to the message group, and obtains the serialization parameters of the message group. The serialization parameters refer to the format of the message group converted into data that can be stored or transmitted. Different formats result in different quantities of unit resources required for processing the message group. The device inputs the historical network parameters and the serialization parameters into the prediction model, and the prediction model outputs the second quantity of unit resources required for processing the message group. In addition, the device can also input the historical network parameters, the serialization parameters, and the time consumption required for processing the messages in the message group into the prediction model. For example, the format of the message group is json format, and the message group does not include strings, so the time consumption is 0; if the format of the message group is avro, the message group includes strings, and the strings need to be converted into Mysql format, and the average time consumption of each string is 10 ms, that is, the input time consumption is 10 ms.

[0081] The prediction model is trained based on a plurality of training samples. The training samples include network parameters of the distributed system in a target time period, a number of unit resources required by the distributed system to perform a task in the target time period, and serialization parameters of a message group in the target time period. The target time period is a time period corresponding to adjacent first and second sampling time points, and the distributed system has no time delay at the first sampling time point and has a time delay at the second sampling time point. In simple terms, the device obtains, as the training samples, information such as traffic, flow rate, serialization parameters of a message group, and a number of unit resources required to process the message group in a time period in which the worker node suddenly has a time delay from having no time delay.

[0082] In step S402, a first number of unit resources required to process a message unit of the message group is determined according to the second number and a number of message units in the message group, where the message group includes a plurality of message units with equal data amounts.

[0083] The message group is evenly divided into message units. Thus, after obtaining the second number, the device divides the second number by a number of message units included in the message group to obtain the first number of unit resources required to process the message unit.

[0084] In this embodiment, the device determines the second number of unit resources required to process the message group based on historical network parameters, so as to accurately determine the first number of unit resources required to process the message unit according to the second number.

[0085] In an embodiment, the device determines a maximum first number after determining the first number of each message unit. If the maximum first number is greater than the number of unit resources set by the worker node, it is determined that the worker node cannot process the maximum first number of message units, that is, resource expansion of the worker node is required. In this regard, the device sends prompt information to each worker node, where the prompt information is used to instruct to increase the number of unit resources of the worker node, so that the number of unit resources of the worker node after expansion is greater than or equal to the maximum first number, thereby avoiding the problem that the worker node cannot process the message unit.

[0086] Reference is made to FIG. 5 , FIG. 5 For another embodiment of the resource configuration method of the present disclosure, based on any of the embodiments shown above, before step S201, the method further includes: FIGS. 2-4

[0087] In step S501, basic information of a task to be executed is obtained.

[0088] In this embodiment, after obtaining the task to be executed, the device parses the task to be executed to obtain the basic information.

[0089] ​In an example, the device can adopt a syntax tree parsing method to detect metadata information used by the task to be executed, so as to obtain basic information of the task to be executed, such as Kafka (Kafka is a high-throughput distributed publish / subscribe message system) address, Topic (message group), bloodline data, and the like.

[0090] In step S502, the basic information is used to configure parameters of each working node.

[0091] The device configures parameters of the working node based on the basic information. The parameter configuration includes: whether the cluster and the queue are enabled for checkpoint, checkpoint interval, checkpoint storage mode, memory ratio of in-memory and out-of-memory, and the like.

[0092] In an example, the label of the task to be executed includes offline testing or online testing. If the label is online testing, an online cluster is allocated. If the label is offline testing, an offline cluster is allocated. The device can also determine the queue according to the group to which the user belongs and the stability level of the task. For example, if the task to be executed includes an aggregated task, the checkpoint storage mode is set to rockdb checkpoint storage. If the task to be executed includes a non-aggregated task, hdfs is used for storage. If the priority of the non-aggregated task is low, hdfs (Hadoop Distributed File System) storage is closed. For different storage, the device also configures different memory ratios. For example, rockdb uses more out-of-memory. These optimization parameters are stored in a configuration manner.

[0093] After the device configures parameters of the distributed system, the configured parameters can be output, and the user can adjust the configured parameters.

[0094] In the embodiment, the device obtains basic information of the task to be executed, and configures parameters of each working node based on the basic information. Therefore, manual parameter configuration of the working node is not needed, and the execution efficiency of the flow task is improved.

[0095] Reference FIG. 6 , FIG. 6 The resource configuration method provided by the present disclosure also includes a flowchart of an embodiment. Based on any of the embodiments shown in the drawings, FIGS. 2-5 Any of the embodiments shown in the drawings, step S203 further includes:

[0096] In step S601, a task index of a task to be executed is obtained.

[0097] In the embodiment, the device also has a diagnosis and optimization service. Specifically, the device is provided with a rule engine, and the rule engine is configured with rules for task diagnosis and optimization, i.e., diagnosis rules. The rules include two parts, i.e., a rule range and a corresponding strategy. For example, if the pressure ratio is high (the pressure ratio is out of range), the parallelism needs to be increased (the corresponding strategy); if the GC (Garbage Collection) time is long, the memory configuration needs to be increased; and if the failover frequency is high, the logs of the ERROR level (the ERROR level indicates an error that cannot be repaired by itself) in the logs need to be retrieved and irrelevant logs need to be removed.

[0098] The device obtains a task indicator of a task to be executed, for example, a pressure ratio less than a preset pressure ratio.

[0099] In step S602, if the task indicator matches the preset diagnosis rule, diagnosis suggestion information corresponding to the preset diagnosis rule is output.

[0100] The rule engine in the device performs rule traversal based on the task indicator. If it is determined that the task indicator matches the preset diagnosis rule, diagnosis suggestion information corresponding to the preset diagnosis rule is output. For example, the diagnosis rule is that the memory configuration of a worker node is adapted to the memory required for execution of a stream task. If it is detected that the memory required for execution of the stream task is lower than 2048 MB, and the memory configuration of the worker node is 4096 MB, diagnosis suggestion information of “reducing the memory configuration to 2048 MB” is output.

[0101] Further, the device is also provided with a model engine. The model engine is trained by log data and indicator data of a historically executed stream task. The log data includes data of multiple levels (for example, the ERROR level), and the indicator data includes data delay, input / output checkpoint time, memory usage, GC time, etc. The device inputs the task indicator to the model engine, and the model engine outputs a result. For example, the result output by the model engine is that the probability that the memory required for execution of the task needs to be reduced is 95%.

[0102] The device performs weighted processing or sorting processing based on the results obtained by the rule engine and the model engine to obtain a final result, and then generates and outputs diagnosis suggestion information based on the final result.

[0103] In an example, the result of the rule engine is set as the main, that is, the weight of the rule engine is much greater than the weight of the model engine. If the rule engine determines that the task index can match the preset diagnosis rule, regardless of the result output by the model engine, the device will finally output the diagnosis suggestion information corresponding to the preset diagnosis rule. If the task index does not match the preset diagnosis rule, the task index is input into the diagnosis model (the diagnosis model is the model engine). If the probability value of the parameter modification output by the diagnosis model is greater than the preset probability value, the prompt information of the parameter modification is output. For example, the parameter is the memory, the model engine determines that the probability of reducing the memory is 80%, and the preset probability value is 70%. Therefore, the suggestion information of reducing the memory configuration is output.

[0104] In the embodiment, the device acquires the task index of the task to be executed. If the task index matches the preset diagnosis rule, the diagnosis suggestion information corresponding to the preset diagnosis rule is output, so that the optimization of the parameters of the distributed system is performed.

[0105] EXEMPLARY MEDIUM

[0106] After the method of the example embodiment of the present disclosure is introduced, next, with reference to FIG. 7 The storage medium of the example embodiment of the present disclosure is described.

[0107] Referring to FIG. 7 As shown in the figure, the storage medium 70 stores the program product for implementing the above method according to the embodiment of the present disclosure, which can adopt a portable compact disc read-only memory (CD-ROM) and includes computer execution instructions for making the computing device execute the resource configuration method provided by the present disclosure. However, the program product of the present disclosure is not limited to this.

[0108] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0109] The readable medium can include a data signal transported in a baseband or propagated as a carrier wave in a propagated signal, where the data signal or propagated signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. A readable medium can also be any medium that is not a storage medium or any medium that is not a transmission medium.

[0110] Computer-executable instructions can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-executable instructions can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote cloud device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, as in the case of the Internet.

[0111] EXEMPLARY DEVICE

[0112] After introducing the medium of the exemplary embodiments of the present disclosure, next, reference is made to FIG. 8 The resource configuration apparatus of the exemplary embodiments of the present disclosure is described, which is used to implement the method in any of the method embodiments, and the implementation principles and technical effects are similar, and will not be repeated here.

[0113] Reference is made to FIG. 8 , FIG. 8 The structure schematic diagram of the resource configuration apparatus provided by the embodiments of the present disclosure is schematically shown. As shown in FIG. 8 The resource configuration apparatus includes:

[0114] The acquisition module 810 is configured to acquire a plurality of message groups to be read by a to-be-executed task and historical network parameters of a distributed system for executing the to-be-executed task. The message group includes a plurality of message units, and the message unit includes one or more messages.

[0115] The determination module 820 is configured to determine a first quantity of unit resources required for processing each message unit according to the historical network parameters. The message group includes a plurality of message units, and the message unit includes one or more messages.

[0116] The allocation module 830 is configured to allocate corresponding message units to each working node in the distributed system according to the first quantity of unit resources required for processing each message unit. The sum of the first quantities of unit resources required for processing each message unit allocated to the working node is less than or equal to the quantity of unit resources of the working node.

[0117] In an embodiment of the present disclosure, the resource configuration apparatus comprises:

[0118] The sorting module 840 is configured to sort the message units in descending order according to the first quantity of unit resources required for processing each message unit.

[0119] The allocation module 830 is further configured to allocate the sorted message units to the working nodes, wherein the working nodes remaining after the allocation of the message units have a quantity of unit resources less than the minimum first quantity.

[0120] In another embodiment of the present disclosure, the resource configuration apparatus comprises:

[0121] The determination module 820 is further configured to determine the second quantity of unit resources required for processing each message group according to the historical network parameters.

[0122] The determination module 820 is further configured to determine the first quantity of unit resources required for processing the message units of the message group according to the second quantity and the quantity of message units in the message group, wherein the message group comprises a plurality of message units with equal data quantities.

[0123] In another embodiment of the present disclosure, the resource configuration apparatus comprises:

[0124] The acquisition module 810 is further configured to acquire the serialization parameters of the message group.

[0125] The input module 850 is configured to input the historical network parameters and the serialization parameters of the message group into the prediction model.

[0126] The determination module 820 is further configured to determine the second quantity of unit resources required for processing the message group according to the output result of the prediction model.

[0127] In another embodiment of the present disclosure, the prediction model is trained according to the network parameters of the distributed system in a target time period, the quantity of unit resources required for the distributed system to perform a task in the target time period, and the serialization parameters of the message group in the target time period, the target time period is a time period corresponding to adjacent first and second sampling time points, the distributed system does not have a delay at the first sampling time point, and the distributed system has a delay at the second sampling time point.

[0128] In another embodiment of the present disclosure, the resource configuration apparatus comprises:

[0129] The sending module 860 is configured to send prompt information to each working node if the maximum first quantity is greater than the quantity of unit resources set by the working node, the prompt information being used to indicate increasing the quantity of unit resources of the working node.

[0130] In another embodiment of this disclosure, the resource allocation device includes:

[0131] The acquisition module 810 is also used to acquire basic information about the task to be executed;

[0132] Configuration module 870 is used to configure parameters for each working node based on basic information.

[0133] In another embodiment of this disclosure, the resource allocation device includes:

[0134] The acquisition module 810 is also used to acquire task metrics for tasks to be executed.

[0135] The output module 880 is used to output diagnostic suggestion information corresponding to the preset diagnostic rules if the task indicators match the preset diagnostic rules.

[0136] In another embodiment of this disclosure, the resource allocation device includes:

[0137] Input module 850 is used to input the task indicators into the diagnostic model if the task indicators do not match the preset diagnostic rules.

[0138] The output module 880 is also used to output a parameter modification prompt message if the probability value of parameter modification output by the diagnostic model is greater than the preset probability value.

[0139] EXEMPLARY COMPUTING DEVICE

[0140] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... FIG. 9 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0141] FIG. 9 The computing device 90 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein. FIG. 9 As shown, the computing device 90 is presented in the form of a general-purpose computing device. The components of the computing device 90 may include, but are not limited to: at least one processing unit 901, at least one storage unit 902, and a bus 903 connecting different system components (including the processing unit 901 and the storage unit 902). The at least one storage unit 902 stores computer-executable instructions; the at least one processing unit 901 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0142] The 903 bus includes a data bus, a control bus, and an address bus.

[0143] The storage unit 902 can include a readable medium that can be non- volatile storage (e.g., flash memory, solid state memory, disk drives, etc.). The storage unit 902 can include a volatile memory, such as a random access memory (RAM) 9021 and / or a cache memory 9022. The storage unit 902 can also include a non-volatile memory, such as a read-only memory (ROM) 9023.

[0144] The storage unit 902 can also include the programs / utility 9025 having a set (at least one) of program modules 9024, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment, or a combination of some or all of these examples.

[0145] The computing device 90 can also communicate with one or more external devices 904 such as a keyboard or pointing device, through Input / Output (I / O) interface(s) 905. Further, the computing device 90 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet, through network adapter 906. As FIG. 9 illustrated, the network adapter 906 is in communication with the other modules of the computing device 90 through the bus 903. As will be appreciated, although not explicitly illustrated, individual components of the computing device 90 can include internal memory which can store documents, data and applications on a repository, as well as external, portable, or fixed, memory storage devices can be used for documents, data and applications.

[0146] It should be noted that although several units / modules or sub-units / modules of the resource configuration apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. Indeed, according to embodiments of the present disclosure, features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, features and functions of one unit / module described above can be further divided into a plurality of units / modules for embodiment.

[0147] Further, although operations of the methods of the present disclosure are described in a particular order in the drawings, this is not required or implied, and the desired results can be achieved without performing all of the operations shown, or performing the operations in the particular order. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or split into multiple steps. While the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it is understood that the present disclosure is not limited to the specific embodiments disclosed and that various modifications and equivalents can be included within the spirit and scope of the appended claims.

Claims

1. A resource configuration method, characterized by, The method comprises the following steps: obtaining a plurality of message groups to be read by a task to be executed and historical network parameters of a distributed system for executing the task to be executed, wherein each message group comprises one or more message units; determining a first quantity of unit resources required for processing each message unit according to the historical network parameters; allocating each message unit to each working node of the distributed system according to the first quantity of unit resources required for processing each message unit, wherein the sum of the first quantity of unit resources required for processing each message unit allocated to the working node is less than or equal to the quantity of unit resources of the working node; the step of determining the first quantity of unit resources required for processing each message unit according to the historical network parameters comprises: obtaining serialization parameters of the message group; inputting the historical network parameters and the serialization parameters of the message group into a prediction model; determining a second quantity of unit resources required for processing the message group according to the output result of the prediction model; determining the first quantity of unit resources required for processing each message unit of the message group according to the second quantity and the quantity of message units in the message group, wherein the message group comprises a plurality of message units with equal data quantity.

2. The resource configuration method of claim 1, wherein, the step of allocating each message unit to each working node of the distributed system according to the first quantity of unit resources required for processing each message unit comprises: sorting each message unit in descending order according to the first quantity of unit resources required for processing each message unit; allocating each sorted message unit to each working node, wherein the quantity of unit resources remaining in the working node after completing the allocation of the message unit is less than the minimum first quantity.

3. The resource configuration method of claim 1, wherein, The prediction model is trained according to network parameters of the distributed system in a target time period, a quantity of unit resources required for the distributed system to execute a task in the target time period, and serialization parameters of a message group in the target time period, wherein the target time period is a time period corresponding to adjacent first and second sampling time points, the distributed system does not have a delay at the first sampling time point, and the distributed system has a delay at the second sampling time point.

4. The resource configuration method of any of claims 1-3, wherein, before the step of allocating each message unit to each working node of the distributed system according to the first quantity of unit resources required for processing each message unit, the method further comprises: if the maximum first quantity is greater than the quantity of unit resources set by each working node, sending prompt information to each working node, wherein the prompt information is used to indicate increasing the quantity of unit resources of the working node.

5. The resource configuration method of any of claims 1-3, wherein, before the step of obtaining a plurality of message groups to be read by a task to be executed, the method further comprises: obtaining basic information of the task to be executed; performing parameter configuration on each working node according to the basic information.

6. The resource configuration method of any of claims 1-3, wherein, The first quantity of unit resources required for processing each of the message units is used to allocate corresponding message units to each of the worker nodes of the distributed system, and the method further includes: obtaining a task index of the task to be executed; if the task index matches a preset diagnosis rule, outputting diagnosis suggestion information corresponding to the preset diagnosis rule.

7. The method of claim 6, wherein, The method further includes, after the task index of the task to be executed is obtained: if the task index does not match a preset diagnosis rule, inputting the task index into a diagnosis model; if a probability value of parameter modification output by the diagnosis model is greater than a preset probability value, outputting prompt information of parameter modification.

8. A resource configuration apparatus, characterized by comprising: The method includes: a obtaining module, configured to obtain a plurality of message groups read by a task to be executed and historical network parameters of a distributed system for executing the task to be executed, wherein the message groups include a plurality of message units, and each of the message units includes one or more messages; a determining module, configured to determine, according to the historical network parameters, a first quantity of unit resources required for processing each of the message units, wherein each of the message units includes one or more messages; an allocating module, configured to allocate, according to the first quantity of unit resources required for processing each of the message units, corresponding message units to each of worker nodes in the distributed system, wherein a sum of the first quantities of unit resources required for processing each of the message units allocated to the worker nodes is less than or equal to a quantity of unit resources of the worker nodes; the obtaining module is further configured to obtain a serialization parameter of the message groups; an inputting module, configured to input the historical network parameters and the serialization parameter of the message groups into a prediction model; the determining module is further configured to determine, according to a result output by the prediction model, a second quantity of unit resources required for processing the message groups; the determining module is further configured to determine, according to the second quantity and a quantity of message units in the message groups, the first quantity of unit resources required for processing each of the message units in the message groups, wherein the message groups include a plurality of message units with equal data quantities.

9. The apparatus for resource configuration of claim 8, wherein, The method includes: an ordering module, configured to order each of the message units from large to small according to the first quantity of unit resources required for processing each of the message units; the allocating module is further configured to allocate each of the ordered message units to each of the worker nodes, wherein a quantity of unit resources remaining in the worker nodes after the allocation of the message units is completed is less than the smallest first quantity.

10. The apparatus for resource configuration of claim 8, wherein, The prediction model is trained according to network parameters of the distributed system in a target time period, a quantity of unit resources required for executing a task by the distributed system in the target time period, and a serialization parameter of message groups in the target time period, the target time period is a time period corresponding to adjacent first and second sampling time points, the distributed system does not have a delay at the first sampling time point, and the distributed system has a delay at the second sampling time point.

11. The apparatus for resource configuration of any of claims 8-10, wherein, The method includes: The sending module is configured to send prompt information to each of the worker nodes if the maximum first quantity is greater than the quantity of unit resources set by the worker nodes, and the prompt information is used to indicate increasing the quantity of unit resources of the worker nodes.

12. The apparatus for resource configuration of any of claims 8-10, wherein, The method comprises the following steps: The obtaining module is further configured to obtain basic information of the to-be-executed task. The configuration module is configured to perform parameter configuration on each of the worker nodes according to the basic information.

13. The apparatus for resource configuration of any of claims 8-10, wherein, The method comprises the following steps: The obtaining module is further configured to obtain a task index of the to-be-executed task. The output module is configured to output diagnostic suggestion information corresponding to a preset diagnostic rule if the task index matches the preset diagnostic rule.

14. The apparatus for resource configuration of claim 13, wherein, The input module is configured to input the task index into a diagnostic model if the task index does not match the preset diagnostic rule. The output module is further configured to output prompt information of parameter modification if a probability value of parameter modification output by the diagnostic model is greater than a preset probability value. The computer execution instruction is executed by the processor to implement the resource configuration method in any one of claims 1 to 7.

15. A medium characterized by, The method comprises the following steps: The memory stores computer execution instructions, and the processor executes the computer execution instructions stored in the memory, so that the processor executes the resource configuration method in any one of claims 1 to 7.

16. A computing device, comprising: The memory stores computer execution instructions, and the processor executes the computer execution instructions stored in the memory, so that the processor executes the resource configuration method in any one of claims 1 to 7. ​ ​

Images