Chained resource decoupling method for distributed machine learning reasoning

Abstract

The invention discloses a chained resource decoupling method for distributed machine learning reasoning, which belongs to the field of distributed machine learning and comprises the following steps: a task processor acquires current load data of a reasoning task being executed and sends the current load data to a task distributor of a distributed machine learning system; the task distributor determines a data processing decision according to all the current load data and the historical load data after acquiring the historical load data and the current load data sent from each task processor, and sends the data processing decision to each task processor; and after receiving the data processing decision, the task processor adjusts the inference task being executed according to the current load data and the data processing decision, so that the adjusted task load is matched with the load feature threshold range corresponding to the task processor. By dynamically monitoring and adjusting the task load, the problem of low system efficiency in the distributed machine learning reasoning process is solved.

Description

Technical Field

[0001] The present application belongs to the field of distributed machine learning, and specifically relates to a chain resource decoupling method, device, electronic device and computer-readable storage medium for distributed machine learning reasoning. Background Art

[0002] As the scale of machine learning models continues to expand, especially the widespread use of large language models (LLMs), the number of model parameters ranges from billions to hundreds of billions, requiring distributed processing during inference. Large language models use an autoregressive generation method based on context windows, which enables them to handle longer conversations, documents, and code blocks, significantly improving the quality of interaction and the scope of application.

[0003] Currently, distributed reasoning mainly adopts two strategies: model parallelism and data parallelism. Model parallelism divides the parameters of the model to different computing devices, and each device processes a part of the model parameters. It is specifically divided into tensor parallelism and pipeline parallelism. Tensor parallelism divides the weight of each layer into equal parts and distributes them to each computing device, while pipeline parallelism divides the model weight by layer. Data parallelism runs multiple model copies on multiple computing devices, and each request is inferred by a specific model copy.

[0004] However, as the context window increases, the iteration delay will also increase. Under the existing distributed strategy, each model replica serves a fixed number of requests in a fixed model parallel mode, resulting in a tight coupling of the allocation of fixed resources and variable resources: that is, when requests with longer contexts and requests with shorter contexts are mixed, the variable resource requirements of the batch task depend on the longest context request, resulting in insufficient resources for requests with longer contexts and low resource utilization for requests with shorter contexts. This increases the overall latency of the system and low resource utilization. Summary of the invention

[0005] The present application aims to provide a chain resource decoupling method, device, electronic device and computer-readable storage medium for distributed machine learning reasoning, at least to solve the problem of low system efficiency in the distributed machine learning reasoning process.

[0006] In a first aspect, the embodiment of the present application further discloses a chain resource decoupling method for distributed machine learning reasoning, which is applied to a task processor of a distributed machine learning system, including: Acquire current load data of the inference task being executed; the current load data is used to characterize the task load generated by the inference task; The current load data is sent to a task distributor of a distributed machine learning system, so that after obtaining the historical load data and the current load data sent from all the task processors in the distributed system, the task distributor determines the data processing decision of the distributed machine learning system according to all the current load data and the historical load data, and sends the data processing decision to each of the task processors respectively; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system; Receive a data processing decision sent from the task distributor, and adjust the reasoning task being executed according to the data processing decision, so that the task load generated by the adjusted reasoning task matches the load characteristic threshold range corresponding to the task processor.

[0007] In a second aspect, an embodiment of the present application discloses a chain resource decoupling method for distributed machine learning reasoning, which is applied to a task distributor of a distributed machine learning system, including: Obtain historical load data and current load data sent from each task processor in the distributed system; the current load data is used to characterize the task load generated by the inference task being executed by the corresponding task processor; the historical load data is used to characterize the task load generated by all historical inference tasks in the distributed machine learning system; Determining a data processing decision of the distributed machine learning system based on all of the current load data and the historical load data; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; The data processing decision is sent to each of the task processors respectively, so that after receiving the data processing decision, the task processor adjusts the reasoning task being executed by the task processor according to the data processing decision, so that the task load generated by the adjusted reasoning task of the task processor matches the load characteristic threshold range corresponding to the task processor.

[0008] In a third aspect, the embodiment of the present application further discloses a chain resource decoupling device for distributed machine learning reasoning, which is applied to a task processor of a distributed machine learning system, including: A monitoring module, used to obtain current load data of the inference task being executed; the current load data is used to characterize the task load generated by the inference task; A monitoring data sending module, used for sending the current load data to a task distributor of a distributed machine learning system, so that after obtaining the historical load data and the current load data sent from all the task processors in the distributed system, the task distributor determines the data processing decision of the distributed machine learning system according to all the current load data and the historical load data, and sends the data processing decision to each of the task processors respectively; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system; A decision execution module is used to receive a data processing decision sent from the task distributor, and adjust the reasoning task being executed according to the data processing decision, so that the task load generated by the adjusted reasoning task matches the load characteristic threshold range corresponding to the task processor.

[0009] In a fourth aspect, the embodiment of the present application further discloses a chain resource decoupling device for distributed machine learning reasoning, which is applied to a task distributor of a distributed machine learning system, including: A monitoring data acquisition module is used to acquire historical load data and current load data sent from each task processor in the distributed system; the current load data is used to characterize the task load generated by the reasoning task being executed by the corresponding task processor; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system; A decision module, configured to determine a data processing decision of the distributed machine learning system based on all of the current load data and the historical load data; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; A decision sending module is used to send the data processing decision to each of the task processors respectively, so that after receiving the data processing decision, the task processor adjusts the reasoning task being executed by the task processor according to the data processing decision, so that the task load generated by the adjusted reasoning task of the task processor matches the load characteristic threshold range corresponding to the task processor.

[0010] In a fifth aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0011] In a sixth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0012] In summary, in the embodiments of the present application, by combining and analyzing the load data of the real-time monitoring reasoning task with the historical data, a more optimized resource allocation decision is made comprehensively to predict future load trends, and a reasonable resource allocation strategy is formulated to adjust the current reasoning task load, to ensure that the load of each task processor operates within a reasonable range, to avoid overload or idle resources, to improve resource utilization, to achieve dynamic scheduling of resources, to improve the flexibility and adaptability of the system, and to better respond to requests for different voluntary requirements. Thus, based on the method of the embodiments of the present application, by dynamically monitoring and adjusting the task load, the system can allocate resources more efficiently, improve the overall reasoning efficiency and response speed. The chain decoupling of resources in the distributed machine learning reasoning process is achieved, solving the problem of low system efficiency in the distributed machine learning reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the attached picture: Figure 1 The embodiment of the present application provides an architecture diagram of a distributed machine learning system in an embodiment of the present application; Figure 2 It is a flowchart of the steps of a chain resource decoupling method for distributed machine learning reasoning provided by an embodiment of the present application; Figure 3 It is a flowchart of the steps of another method for chain resource decoupling of distributed machine learning reasoning provided by an embodiment of the present application; Figure 4 This is the data transfer process based on the method of the embodiment of the present application; Figure 5 It is the overall steps of the method of the embodiment of the present application; Figure 6 It is a block diagram of a chain resource decoupling device for distributed machine learning reasoning provided by an embodiment of the present application; Figure 7 It is a block diagram of another chain resource decoupling device for distributed machine learning reasoning provided by an embodiment of the present application; Figure 8 is a block diagram of an electronic device according to an embodiment of the present application; Fig. 9 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0016] Figure 1 The distributed machine learning system in the embodiment of the present application, wherein the number of task processors is only a reference, and may be more or less in actual situations, wherein the task distributors are connected to the task processors one by one, and the task processors are connected to each other, based on Figure 1 System architecture, Figure 2 This embodiment provides a chain resource decoupling method for distributed machine learning reasoning.

[0017] Among them, step 101, step 102, and step 106 are applied to the task processor of the distributed machine learning system; step 103, step 104, and step 105 are applied to the task distributor of the distributed machine learning system.

[0018] like Figure 2 As shown, the chain resource decoupling method for distributed machine learning reasoning provided in this embodiment specifically includes the following steps: Step 101, obtaining current load data of the inference task being executed.

[0019] The current load data is used to characterize the task load generated by the reasoning task.

[0020] In some embodiments of the present application, the current load data of the reasoning task being executed will be obtained. The current load data refers to the data used to characterize the task load generated by the reasoning task. These data include but are not limited to the number of requests processed by the task processor within a specific time window, the context length of each request, the iteration arrival rate (the number of iterations arriving per unit time), and the output length of the completed request. By collecting this data, the current load of the task processor can be understood, providing a basis for subsequent load prediction and resource allocation. It should be emphasized that due to the continuity of distributed machine learning reasoning, when the reasoning task involved in this application is being executed, it is bounded by the time window of the task, that is, all reasoning tasks within the current time window will be identified as reasoning tasks being executed.

[0021] For example, in a distributed machine learning system, suppose that task processor A is processing multiple inference tasks. The runtime monitoring module counts the context length and iteration arrival rate of all requests processed by task processor A in a time window. For example, in the past 10 minutes, the task processor processed two requests, ten of which increased in length from 50 to 150, and another ten of which increased in length from 100 to 200. Then, the iteration arrival rate of context lengths of 50 to 100 is 1 per minute, the iteration arrival rate of context lengths of 100 to 150 is 2 per minute, and the iteration arrival rate of context lengths of 150 to 200 is 1 per minute. This means that they have several iterations in each time window, so each context length has a corresponding number of iterations. The runtime monitoring module also records the output length of each request, and summarizes these data into a load histogram and submits it to the task dispatcher. Through these current load data, the system can accurately understand the load of task processor A, providing a basis for subsequent load prediction and resource allocation.

[0022] Step 102, sending the current load data to the task distributor of the distributed machine learning system.

[0023] Among them, after obtaining the historical load data and the current load data sent from all task processors in the distributed system, the task distributor will determine the data processing decision of the distributed machine learning system based on all the current load data and historical load data, and send the data processing decision to each task processor respectively.

[0024] Among them, the data processing decision includes the load characteristic threshold range defined for each task processor in the distributed machine learning system; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system.

[0025] In some embodiments of the present application, the current load data is sent to the task distributor of the distributed machine learning system. After collecting the current load data, the task processor sends the data to the task distributor. After receiving the current load data and historical load data from all task processors, the task distributor comprehensively analyzes the data to determine the overall data processing decision of the system. The data processing decision includes a load characteristic threshold range defined for each task processor to ensure that each task processor operates within a reasonable load range to avoid overload or idle resources.

[0026] For example, in a distributed machine learning system, suppose that task processor A and task processor B have collected the current load data of the system respectively. Each task processor sends these current load data to the task distributor. The task distributor also has the total historical load data of the distributed machine learning system, such as the average load of task processor A in the past hour. The task distributor comprehensively analyzes these data and determines that in the next time window, the load characteristic threshold range of task processor A is 800 to 1200 characters in length, and the load characteristic threshold range of task processor B is 1201 to 1600 characters in length. The task distributor sends these data processing decisions back to each task processor so that they can adjust the current reasoning task according to the new load characteristic threshold range.

[0027] Step 103, obtaining historical load data and current load data sent from each task processor in the distributed system.

[0028] Among them, the current load data is used to characterize the task load generated by the reasoning task being executed by the corresponding task processor; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system.

[0029] In some embodiments of the present application, historical load data and current load data sent from each task processor in the distributed system are obtained. The current load data is used to characterize the task load generated by the inference task currently being executed by the task processor, while the historical load data is used to characterize the task load generated by the inference tasks executed in the past in the entire distributed machine learning system. By combining these two types of data, the system can have a more comprehensive understanding of the load of the task processor, providing a basis for subsequent load prediction and resource allocation.

[0030] For example, in a distributed machine learning system, assume that task processor A and task processor B have collected the current load data of the system respectively. Each task processor sends these current load data to the task distributor. At the same time, the task distributor also obtains the historical load data of the distributed machine learning system, that is, the cumulative iteration arrival rate of each context length in the past time window, and the average output length of each context length request. By combining the current current load data and historical load data, the task distributor can more accurately predict future load trends and formulate reasonable resource allocation strategies.

[0031] Step 104, determine the data processing decision of the distributed machine learning system based on all current load data and historical load data.

[0032] The data processing decision includes a load characteristic threshold range for each task processor in the distributed machine learning system.

[0033] In some embodiments of the present application, the data processing decision of the distributed machine learning system is determined based on all the current load data and historical load data. The data processing decision includes a load characteristic threshold range defined for each task processor to ensure that each task processor operates within a reasonable load range. By comprehensively analyzing the current current load data and historical load data, the system can predict future load trends and formulate reasonable resource allocation strategies to optimize system performance and avoid overload or idle resources.

[0034] For example, in a distributed machine learning system, the task dispatcher receives current load data and historical load data from task processor A and task processor B. Task processor A processes requests of 1,000 characters in the current time window, and historical data shows that its average load in the past hour is 900 character requests. Task processor B processes requests of 800 characters in the current time window, and historical data shows that its average load in the past hour is 750 character requests. The task dispatcher comprehensively analyzes these data and predicts that in the next time window, the load of task processor A may increase to 1,100 character requests per minute, while the load of task processor B may remain at around 800 character requests per minute. Based on these predictions, the task dispatcher determines the data processing decision, and the load feature threshold range for task processor A is 800 to 1,200 requests, and the load feature threshold range for task processor B is 1,201 to 1,600 requests. The task distributor sends these data processing decisions back to the individual task processors so that they can adjust their current inference tasks according to the new load characteristic threshold range.

[0035] Step 105 , sending the data processing decision to each task processor respectively.

[0036] Among them, after receiving the data processing decision, the task processor will adjust the reasoning task being executed by the task processor according to the data processing decision, so that the task load generated by the adjusted reasoning task of the task processor matches the load characteristic threshold range corresponding to the task processor.

[0037] In some embodiments of the present application, the data processing decision is sent to each task processor separately. After determining the overall data processing decision of the system, the task distributor sends these decisions to each task processor. After receiving the data processing decision, the task processor will compare and adjust the data processing decision based on its own current load data to ensure that the task load generated by the currently executed reasoning task runs within the load characteristic threshold range corresponding to the task processor. This process helps to optimize system performance and avoid overloading the task processor or idle resources.

[0038] For example, in a distributed machine learning system, the task dispatcher has determined the load feature threshold ranges of task processors A and B, and sent these data processing decisions to them. The decision received by task processor A is a request whose load feature threshold range is 800 to 1200 tokens, and the load feature threshold range of task processor B is a request whose character length is 1201 to 1600. After receiving the data processing decision, task processor A checks the current load data and finds that the context length of a request currently being processed has changed from 1200 characters to 1201, which exceeds the threshold range. Therefore, task processor A adjusts the current reasoning task according to the data processing decision and transfers the reasoning task to other task processors so that its load returns to within 1200 characters. Similarly, after receiving the data processing decision, task processor B finds that the number of characters currently being processed is 1201, which is within the threshold range, so no adjustment is required. In this way, the system ensures that the load of each task processor operates within a reasonable range, improving the efficiency and stability of the overall system.

[0039] Step 106, receiving the data processing decision sent from the task distributor, and adjusting the inference task being executed according to the data processing decision, so that the task load generated by the adjusted inference task matches the load characteristic threshold range corresponding to the task processor.

[0040] In some embodiments of the present application, the task processor receives the data processing decision sent from the task distributor, and adjusts the reasoning task being executed according to the current load data and the data processing decision. The purpose of the adjustment is to ensure that the task load generated by the adjusted reasoning task runs within the load characteristic threshold range corresponding to the task processor. This process helps to optimize system performance, avoid task processor overload or resource idleness, and thus improve the overall efficiency and stability of the system.

[0041] For example, in a distributed machine learning system, task processor A receives a data processing decision sent by the task distributor, which stipulates that its load feature threshold range is 800 to 1200 character length requests. Task processor A checks the current load data and finds that the context length of a request currently being processed has changed from 1200 characters to 1201 characters, which is beyond the threshold range. In order to meet the requirements of the data processing decision, task processor A needs to adjust the inference task being executed. Task processor A first identifies which requests can be transferred to other task processors. Then, it migrates the metadata and related key-value caches of these requests to other task processors to reduce its own load. After adjustment, the context lengths of all tasks on task processor A are in the range of 800 to 1200, which meets the requirements of the data processing decision. In this way, task processor A ensures that its load runs within a reasonable range, avoids overload, and improves the stability and response speed of the system.

[0042] In summary, in the embodiments of the present application, by combining and analyzing the load data of the real-time monitoring reasoning task with the historical data, a more optimized resource allocation decision is made comprehensively to predict future load trends, and a reasonable resource allocation strategy is formulated to adjust the current reasoning task load, to ensure that the load of each task processor operates within a reasonable range, to avoid overload or idle resources, to improve resource utilization, to achieve dynamic scheduling of resources, to improve the flexibility and adaptability of the system, and to better respond to requests for different voluntary requirements. Thus, based on the method of the embodiments of the present application, by dynamically monitoring and adjusting the task load, the system can allocate resources more efficiently, improve the overall reasoning efficiency and response speed. The chain decoupling of resources in the distributed machine learning reasoning process is achieved, solving the problem of low system efficiency in the distributed machine learning reasoning process.

[0043] Figure 3 Another chain resource decoupling method for distributed machine learning reasoning is provided in an application embodiment.

[0044] Among them, step 201, step 202, and step 206 are applied to the task processor of the distributed machine learning system, and step 203, step 204, and step 205 are applied to the task distributor of the distributed machine learning system.

[0045] The specific steps include: Step 201, obtaining current load data of the inference task being executed.

[0046] The current load data is used to characterize the task load generated by the reasoning task.

[0047] The method shown in this step has been explained in step 101 and will not be repeated here.

[0048] Step 202, sending the current load data to the task distributor of the distributed machine learning system.

[0049] Among them, after obtaining the historical load data and the current load data sent from all task processors in the distributed system, the task distributor will determine the data processing decision of the distributed machine learning system based on all the current load data and historical load data, and send the data processing decision to each task processor respectively.

[0050] Among them, the data processing decision includes the load characteristic threshold range defined for each task processor in the distributed machine learning system; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system.

[0051] The method shown in this step has been explained in step 102 and will not be repeated here.

[0052] Step 203, obtaining historical load data and current load data sent from each task processor in the distributed system.

[0053] Among them, the current load data is used to characterize the task load generated by the reasoning task being executed by the corresponding task processor; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system.

[0054] The method shown in this step has been explained in step 103 and will not be repeated here.

[0055] Step 204, determine the data processing decision of the distributed machine learning system based on all current load data and historical load data.

[0056] The data processing decision includes a load characteristic threshold range for each task processor in the distributed machine learning system.

[0057] The method shown in this step has been described in step 104 and will not be repeated here.

[0058] Optionally, the task load includes the text length of the data processed by the corresponding task processor, and the historical load data includes the historical length of the data processed by the distributed machine learning system; step 204 includes the following sub-steps: Sub-step 2041, determining the text length interval range corresponding to the maximum text length, and dividing the text length interval range into a plurality of continuous and equal-width text length interval sub-ranges according to a preset number of intervals.

[0059] In some embodiments of the present application, the text length interval range corresponding to the maximum text length is determined, and the text length interval range is divided into a plurality of continuous and equal-width text length interval sub-ranges according to a preset number of intervals. The text length interval range refers to the maximum text length configuration of the machine learning model. By dividing this range into a plurality of equal-width sub-ranges, text data of different lengths can be analyzed and processed more carefully, thereby optimizing the resource allocation and load management of the system.

[0060] For example, in a distributed machine learning system, assume that the text lengths processed by task processor A and task processor B range from 50 to 500. In order to perform more refined load management, the system presets the text length interval to be divided into 10 sub-ranges of equal width. First, the maximum text length is determined to be 500, so the text length interval range is 0 to 500. Next, this interval is divided into 10 sub-ranges of equal width, and the width of each sub-range is (500 - 0) / 10 = 50. Thus, the text length interval is divided into the following sub-ranges: 0-50, 51-100, 101-150, …, 401-450, 451-500. Through this division, the system can more accurately analyze the load conditions of different text lengths and provide a basis for subsequent resource allocation and optimization.

[0061] In sub-step 2042, the test interval extracted from the text length interval range and the number of corresponding text length interval sub-ranges are used as traversal parameters, and the text length and history length are used as statistical parameters to calculate the standardized iterative delay evaluation value corresponding to the test interval when the traversal parameters take different values.

[0062] The text length interval subranges in the test interval are continuous; the first text length interval subrange in the continuous text length interval subranges in the test interval is the first text length interval subrange in the remaining text length interval subranges in the text length interval range.

[0063] In some embodiments of the present application, the number of text length interval sub-ranges corresponding to the test interval extracted from the text length interval range is used as the traversal parameter, and the text length and the history length are used as statistical parameters to calculate the standardized iteration delay evaluation value corresponding to the test interval when the traversal parameter takes different values. The text length interval sub-ranges in the test interval must be continuous, and the first text length interval sub-range is the first text length interval sub-range remaining in the text length interval range. In this way, the system performance under different text length interval configurations can be evaluated and the optimal configuration scheme can be found.

[0064] For example, in a distributed machine learning system, assume that the text length interval range has been divided into 10 equally wide sub-ranges. In order to evaluate the system performance under different configurations, the system extracts test intervals from these sub-ranges to calculate the standardized iteration delay evaluation values ​​of these test intervals. In this way, the system can evaluate the performance under different text length interval configurations and find the optimal configuration solution to optimize the system's resource allocation and load management.

[0065] Sub-step 2043, taking the test interval corresponding to the minimum standardized iterative delay evaluation value among all the standardized iterative delay evaluation values ​​as the target load characteristic threshold range.

[0066] In some embodiments of the present application, the test interval corresponding to the minimum standardized iterative delay evaluation value among all the standardized iterative delay evaluation values ​​is used as the target load characteristic threshold range. The standardized iterative delay evaluation value is an indicator for evaluating system performance under different test interval configurations. By comparing the standardized iterative delay evaluation values ​​of all test intervals, the test interval corresponding to the minimum evaluation value is selected as the target load characteristic threshold range to optimize system performance and ensure efficient use of resources.

[0067] For example, in a distributed machine learning system, it is assumed that the standardized iteration delay evaluation values ​​of multiple test intervals have been calculated. For example, it is assumed that during the test process, it is found that the standardized iteration delay evaluation value when the test interval contains three text length interval sub-ranges is the minimum value, then the test interval consisting of the first three text length interval sub-ranges is selected as the target load feature threshold range.

[0068] Sub-step 2044, assigning the target load characteristic threshold range to a task processor that is not assigned a load characteristic threshold range.

[0069] In some embodiments of the present application, a target load characteristic threshold range is assigned to a task processor that has not been assigned a load characteristic threshold range. The target load characteristic threshold range refers to the load range that can optimize system performance determined by the previous step. Assigning this range to a task processor that has not been assigned a load characteristic threshold range can ensure that each task processor in the system has a clear load management target, thereby optimizing resource utilization and system performance.

[0070] For example, in a distributed machine learning system, assume that the target load feature threshold range is determined by the previous step to be a test interval consisting of the first three text length interval sub-ranges. There are multiple task processors in the system, among which task processor A has not yet been assigned a load feature threshold range. Therefore, the system assigns the target load feature threshold range to task processor A, making it responsible for managing and processing the load within this range. Through this allocation, task processor A can perform resource management and task processing according to the preset load feature threshold range, ensuring that the system runs under the optimal configuration and improving overall performance and resource utilization.

[0071] Sub-step 2045, remove the target load feature threshold range from the text length interval range to update the text length interval range, and return to sub-step 2042.

[0072] In some embodiments of the present application, the target load characteristic threshold range that has been allocated is removed from the text length interval range, and the text length interval range is updated, and then the allocation process is re-executed starting from the process of sub-step 2042. By removing the target load characteristic threshold range that has been allocated, it can be ensured that the remaining text length interval range only contains the unallocated part, so that optimization and allocation can be continued in subsequent steps until all intervals are reasonably allocated to task processors.

[0073] For example, in a distributed machine learning system, assuming that the text length interval range 0-1200 has been divided into three equally wide sub-ranges, through the previous step, the target load feature threshold range is determined as a test interval consisting of the first three text length interval sub-ranges, and has been assigned to the text length range of task processor A from 0 to 400. Then the system removes the target load feature threshold range 0-400 from the text length interval range, and the updated text length interval range only includes the last two text length interval sub-ranges (401 to 800 and 801 to 1200). Then, the system will return to sub-step 2042 and continue to extract new test intervals from the remaining text length interval ranges for standardized iterative delay evaluation. At this time, since task processor A has been assigned, subsequent assignments will be performed between task processors B and C until all intervals are reasonably assigned to task processors. In this way, the system can gradually optimize resource allocation to ensure that the load of each task processor runs within a reasonable range.

[0074] Optionally, in the above sub-steps, the normalized iteration delay evaluation value is obtained according to the following formula: , in, Represents a collection of history lengths of multiple task processors, A collection of text lengths representing multiple task processors; Representation interval The interval iteration delay of In the reasoning task, the request The output length of In the historical reasoning task, the request An estimate of the output length of ; At this time, the formula for the target load characteristic threshold range is: ; in, is the traversal parameter corresponding to the minimum standardized iteration delay evaluation value, n represents the number of task processors that are not assigned a load characteristic threshold range, and m represents the number of text length interval sub-ranges that have not been included in the load characteristic threshold range.

[0075] In the solution of the embodiment of the present application, in order to find the optimal interval configuration, Assign it to a task processor and Assign to the other n-1 task processors. If n-1>Mp, some task processors do not have corresponding text length interval sub-ranges to process, and the iteration delay of these task processors is recorded as 0. By trying all possible p values, the optimal value that minimizes the objective can be found. The correctness of the algorithm is guaranteed by mathematical induction.

[0076] Optionally, in the above sub-steps, the interval iteration delay is calculated according to the following formula: In the case of p=1: ; in, Indicates the length of the text. represents the preset length threshold for the text length, and the parameters a, b, and c represent the preset model coefficients respectively; In the case of p>1: ; in, Indicates that the data corresponding to the maximum text length belongs to the test interval Probability: , λ represents the total number of inference tasks in the same batch of inference tasks.

[0077] In the solution of the embodiment of the present application, the formalized system optimization goal is: given N task processors, the context length interval is equally divided into M text length interval sub-ranges. i Represents the i-th text length interval sub-range, Indicates the sub-range of the text length interval Configuring chained intervals for task processors is equivalent to assigning these M intervals to N task processors. To represent the interval assigned to the i-th task processor, so represents a chain interval configuration scheme. represents the iteration delay of iteration e running in the i-th task processor, where e represents an iteration and q(e) represents the request ID to which e belongs.

[0078] Then its normalized iteration delay is ,in It is a request The output length of is the latency of iteration e affected by the context length. For request q, all its iterations The sum of the two divided by a constant is the average iteration delay of the system. This module obtains the iterations that arrive within the time window from the load histogram, and the sets of arrival iterations of completed and uncompleted requests are denoted as E and U respectively. For the iterations in U, this paper uses the expected output length based on history Then, the interval allocation problem can be formalized as: , In the above formula, we have .

[0079] Therefore, to solve the interval configuration problem, we need to estimate the latency of the iterations running on a specific task processor, i.e. The interval iteration delay depends on the maximum context length in the batch. Indicates the sub-range of the text length interval The average context length in . The iteration delay in an interval that contains only one sub-range of the text length interval is affected by the fixed resource requirements of inference and the variable resource requirements related to the context length. Among them, the first case represents the constant overhead when the context length is small (less than or equal to θ), denoted as c, and the second case represents the linear increase in delay when the context length is large (greater than θ), and the delay is linearly related to the context length (coefficients are a and b). In an embodiment of the present application, each model can be analyzed offline to obtain specific parameters a, b, c and threshold θ, and the specific values ​​of the relevant data are determined by the specific distributed machine learning system.

[0080] Step 205 , sending the data processing decision to each task processor respectively.

[0081] Among them, after receiving the data processing decision, the task processor will adjust the reasoning task being executed by the task processor according to the data processing decision, so that the task load generated by the adjusted reasoning task of the task processor matches the load characteristic threshold range corresponding to the task processor.

[0082] The method shown in this step has been explained in step 105 and will not be repeated here.

[0083] Step 206, receiving the data processing decision sent from the task distributor, and adjusting the inference task being executed according to the data processing decision, so that the task load generated by the adjusted inference task matches the load characteristic threshold range corresponding to the task processor.

[0084] The method shown in this step has been described in step 106 and will not be repeated here.

[0085] Optionally, in order to adjust the inference task being executed according to the current load data and the data processing decision so that the task load generated by the adjusted inference task matches the load characteristic threshold range corresponding to the task processor, step 206 includes the following sub-steps: Sub-step 2061, when the current load data does not match the load characteristic threshold range corresponding to the task processor, determine the first target task processor among the task processors of the distributed machine learning system.

[0086] Among them, the first target task processor is the task processor corresponding to the first target load feature threshold range that matches the current load data in the distributed machine learning system.

[0087] In some embodiments of the present application, when the current load data does not match the load characteristic threshold range corresponding to the task processor, a first target task processor is determined in the task processor of the distributed machine learning system according to the current load data and the data processing decision. The first target task processor refers to the task processor corresponding to the first target load characteristic threshold range that matches the current load data. By determining the first target task processor, tasks that exceed the load characteristic threshold range of the current task processor can be transferred to a suitable task processor, thereby optimizing system performance and avoiding overload or idle resources.

[0088] For example, in a distributed machine learning system, suppose the load characteristic threshold range of task processor A is 800 to 1200 character length requests. If the current load data shows that the character length of the request being processed by task processor A will change from 1200 to 1201, it exceeds its load characteristic threshold range. Based on the current load data and data processing decisions, the system needs to determine a new task processor to receive some requests. The local task processor will check the load of other task processors and find that the load characteristic threshold range of task processor B is 1201 to 1600 character length. The load of task processor B is within its threshold range. Therefore, the system determines task processor B as the first target task processor and transfers the task request from task processor A to task processor B to ensure that the load of task processor A returns to a reasonable range and optimizes system performance.

[0089] Sub-step 2062 , sending the task content of the inference task being executed, the task metadata of the inference task, and the key-value cache data of the inference task to the first target task processor.

[0090] Among them, the first target task processor will re-establish the reasoning task according to the task content of the reasoning task, cache data according to the key value of the reasoning task, and continue to process the task metadata of the reasoning task so that the reasoning task continues to execute.

[0091] Among them, the key-value cache data is used to record the processing progress of the reasoning task for the corresponding task metadata.

[0092] In some embodiments of the present application, the task content, task metadata, and key-value cache data of the inference task being executed are sent to the first target task processor. After receiving these data, the first target task processor will re-establish the inference task according to the task content, and continue to process the task metadata using the key-value cache data to ensure that the inference task can continue to execute seamlessly. The key-value cache data is used to record the processing progress of the inference task to ensure that the progress information of the task is not lost during the migration process.

[0093] For example, in a distributed machine learning system, suppose that the current load of task processor A exceeds its load characteristic threshold range, and some tasks need to be migrated to task processor B. Task processor A packages the task content, task metadata, and key-value cache data of the inference task being executed and sends them to task processor B. After receiving this data, task processor B re-establishes the inference task based on the task content. For example, the task content may include an inference request for a large language model, the task metadata contains the context information and parameter settings of the request, and the key-value cache data records the processing progress of the task. Task processor B uses this data to continue processing the inference task to ensure that the task can be seamlessly connected and continued on the new processor. In this way, the system realizes dynamic migration and load balancing of tasks, improving overall performance and resource utilization.

[0094] Optionally, as the opposite side execution task of sub-steps 2061 and 2062, when the task processor is determined to be the first target task processor, for the task processor of the distributed machine learning system, the method of the embodiment of the present application further includes the following additional steps: Step 2071, obtain the task content of the inference task sent from other task processors in the distributed machine learning system, the task metadata of the inference task, and the key-value cache data of the inference task.

[0095] Among them, the key-value cache data is used to record the processing progress of the reasoning task for the corresponding task metadata.

[0096] In some embodiments of the present application, when a task processor is determined as the first target task processor, the task content, task metadata, and key-value cache data of the reasoning task sent from other task processors in the distributed machine learning system are obtained. The key-value cache data is used to record the processing progress of the reasoning task. After receiving this data, the first target task processor will re-establish the reasoning task according to the task content of the reasoning task, and continue to process the task metadata using the key-value cache data to ensure that the reasoning task can continue to be executed seamlessly.

[0097] For example, in a distributed machine learning system, suppose task processor B is determined as the first target task processor and needs to receive some inference tasks from task processor A. Task processor A sends the task content, task metadata, and key-value cache data of the inference task being executed to task processor B. After receiving this data, task processor B re-establishes the inference task based on the task content. For example, the task content may include an inference request for a large language model, the task metadata contains the context information and parameter settings of the request, and the key-value cache data records the processing progress of the task. Task processor B uses this data to continue processing the inference task to ensure that the task can be seamlessly connected and continued on the new processor. In this way, the system realizes dynamic migration and load balancing of tasks, improving overall performance and resource utilization.

[0098] Step 2072, re-establishing the reasoning task according to the task content of the reasoning task, caching data according to the key value of the reasoning task, and continuing to process the task metadata of the reasoning task, so that the reasoning task continues to be executed.

[0099] The first target task processor is the task processor corresponding to the first target load characteristic threshold range that matches the current load data in the distributed machine learning system. In some embodiments of the present application, after the task processor receives the task content, task metadata, and key-value cache data of the reasoning task from other task processors, the reasoning task is re-established according to the task content of the reasoning task, and the task metadata is continued to be processed using the key-value cache data. The key-value cache data records the processing progress of the reasoning task, ensuring that the progress information of the task will not be lost during the migration process, so that the reasoning task can continue to be executed seamlessly.

[0100] For example, in a distributed machine learning system, assume that task processor B receives inference task migration data from task processor A. The data sent by task processor A includes the task content of the inference task (for example, an inference request for a large language model), task metadata (for example, the context information and parameter settings of the request), and key-value cache data (recording the processing progress of the task). After receiving this data, task processor B re-establishes the inference task based on the task content. For example, the task content may be a request to generate text, and task processor B will reinitialize the inference task based on the context information and parameter settings of the request. Then, task processor B continues to process the task metadata using the key-value cache data to ensure that the task can be seamlessly connected and continued to execute on the new processor. In this way, the system realizes dynamic migration and load balancing of tasks, improving overall performance and resource utilization.

[0101] Optionally, the task distributor of the distributed machine learning system, the method of the embodiment of the present application further includes the following additional interaction steps, wherein the steps: Step 2081, obtaining the reasoning task to be executed and the task metadata of the reasoning task to be executed.

[0102] In some embodiments of the present application, the reasoning tasks to be executed and the task metadata of the reasoning tasks to be executed are obtained. The reasoning tasks to be executed refer to the reasoning requests that have not yet been processed, and the task metadata refers to the auxiliary information related to the reasoning task, such as the context information of the request, parameter settings, etc. By obtaining this information, the system can prepare for the reasoning tasks to be executed and ensure that the tasks can be smoothly assigned and executed.

[0103] For example, in a distributed machine learning system, suppose a user submits a new reasoning request that requires generating a piece of text. The task distributor receives the reasoning task to be executed and obtains its task metadata. The task metadata includes the context information of the request (for example, the existing text fragment), parameter settings (for example, the length of the generated text, the configuration of the model), etc. By obtaining this information, the task distributor can allocate appropriate resources for the reasoning task and ensure that the task can be executed smoothly.

[0104] Step 2082, determining the initial load of the to-be-executed reasoning task according to the task metadata, and determining a second target task processor among all task processors according to the initial load and the data processing decision.

[0105] Among them, the second target task processor is a task processor corresponding to the second target load characteristic threshold range that matches the initial load in the distributed machine learning system.

[0106] In some embodiments of the present application, the initial load of the reasoning task to be executed is determined based on the task metadata, and the second target task processor is determined among all the task processors based on the initial load and the data processing decision. The initial load refers to the amount of load that is expected to be generated when the reasoning task starts to execute. By analyzing the task metadata, the initial load can be estimated, and according to the load distribution strategy of the system, a suitable task processor (i.e., the second target task processor) is selected to process the task. The second target task processor refers to the task processor corresponding to the load characteristic threshold range that matches the initial load.

[0107] For example, in a distributed machine learning system, assume that the task distributor receives a new inference request whose task metadata includes that the length of the generated text is 500 characters. The task distributor estimates the initial load of the request based on the task metadata, for example, the initial load is 150 characters in length. The system's load distribution strategy stipulates that the load characteristic threshold range of each task processor is different, for example, the threshold range of task processor A is 100 to 200 characters in length, and the threshold range of task processor B is 201 to 400 characters in length. Based on the initial load of 150 computing units, the task distributor determines that the load range of task processor A matches. Therefore, the task distributor selects task processor A as the second target task processor and assigns the inference task to task processor A to ensure load balancing and resource optimization of the system.

[0108] Step 2083: Send the task content and task metadata of the to-be-executed reasoning task to the second target task processor.

[0109] The second target task processor will establish a to-be-executed reasoning task to process the task metadata, and establish key-value cache data of the to-be-executed reasoning task to record the processing progress of the to-be-executed reasoning task on the task metadata.

[0110] In some embodiments of the present application, the task content and task metadata of the to-be-executed reasoning task are sent to the second target task processor. After receiving these data, the second target task processor will establish the to-be-executed reasoning task and process it according to the task metadata. At the same time, the second target task processor will establish key-value cache data to record the processing progress of the reasoning task. The key-value cache data ensures that the task can accurately track and manage its progress during execution to avoid data loss or processing interruption.

[0111] For example, in a distributed machine learning system, the task dispatcher determines task processor B as the second target task processor, and sends the task content and task metadata of the to-be-executed reasoning task to task processor B. The task content may include a request to generate text, and the task metadata contains the context information and parameter settings of the request. After receiving this data, task processor B first establishes the to-be-executed reasoning task based on the task content. For example, if the task content requires the generation of a text with a specified meaning, task processor B will initialize the corresponding reasoning task. Next, task processor B processes the task metadata (such as context information and parameter settings), and establishes key-value cache data to record the processing progress of the task.

[0112] Optionally, as the opposite side execution task of steps 2081, 2082, and 2083, when the task processor is determined to be the second target task processor, for the task processor of the distributed machine learning system, the method of the embodiment of the present application further includes the following additional steps: Step 2091, obtaining the task content of the to-be-executed reasoning task sent from the task distributor and the task metadata of the to-be-executed reasoning task.

[0113] In some embodiments of the present application, the task processor will obtain the task content and task metadata of the reasoning task to be executed sent from the task distributor. The task metadata can protect the specific reasoning request, such as the text to be generated or the computing task to be performed, and the information related to the reasoning task, such as the context information of the request, parameter settings, etc. By obtaining this information, the task processor can prepare for the reasoning task to be executed and ensure that the task can be smoothly allocated and executed.

[0114] For example, in a distributed machine learning system, assume that the task distributor determines that task processor B is the second target task processor, and sends the task content and task metadata of the to-be-executed reasoning task to task processor B. The task content may include a request to generate text, and the task metadata contains the context information and parameter settings of the request. After receiving this data, task processor B first establishes the to-be-executed reasoning task based on the task content. For example, if the task content requires the generation of a text with a specified meaning, task processor B will initialize the corresponding reasoning task. Next, task processor B processes the task metadata (such as context information and parameter settings), and establishes key-value cache data to record the processing progress of the task. In this way, task processor B can accurately track the execution of the reasoning task and ensure that the task can be completed smoothly.

[0115] Step 2092, establish a to-be-executed reasoning task according to the task content of the to-be-executed reasoning task to process the task metadata of the to-be-executed reasoning task, and establish key-value cache data of the to-be-executed reasoning task to record the processing progress of the to-be-executed reasoning task on the task metadata.

[0116] Among them, the second target task processor is a task processor corresponding to the second target load characteristic threshold range that matches the initial load of the reasoning task to be executed in the distributed machine learning system.

[0117] In some embodiments of the present application, a to-be-executed reasoning task is established based on the task content of the to-be-executed reasoning task to process the task metadata, and to establish key-value cache data for recording the processing progress of the task. The second target task processor refers to a task processor that matches the initial load of the to-be-executed reasoning task. By establishing reasoning tasks and key-value cache data, the task processor can accurately track and manage the execution progress of the task to ensure that the task can be successfully completed.

[0118] For example, in a distributed machine learning system, assume that task processor B is determined as the second target task processor and receives the task content and task metadata of the reasoning task to be executed. The task content may include a request to generate text, and the task metadata contains the context information and parameter settings of the request. Task processor B establishes the reasoning task to be executed based on the task content. For example, if the task content requires the generation of a text with the specified meaning, task processor B will initialize the corresponding reasoning task. Next, task processor B processes the task metadata (such as context information and parameter settings) and establishes key-value cache data to record the processing progress of the task. The key-value cache data ensures that the progress of the task can be accurately tracked and managed during execution to avoid data loss or processing interruption. In this way, task processor B can efficiently execute reasoning tasks and ensure the stability and performance of the system.

[0119] Figure 4 Based on the method of the embodiment of the present application, when the distributed machine learning system is divided into a task processor cluster and a task distributor, the data flow mode is: Step S1: Current load data transmission: The operation monitoring module in the task processor cluster monitors the current load situation and transmits the current load data to the task dispatcher. This step ensures that the task dispatcher can understand the load status of each task processor in real time.

[0120] Step S2: Data processing and decision-making: After receiving the current load data, the task dispatcher predicts the future load situation through the load prediction module, and the interval configuration module performs chain interval configuration based on the prediction results to generate the system optimization target. The task dispatcher makes decision instructions based on these data.

[0121] Step S3: Task distribution and migration: The task distributor distributes the task metadata and task content submitted by the user to the appropriate task processor according to the decision instruction. If the task needs to be migrated, the task distributor will also control the migration of the task metadata.

[0122] Step S4: Task execution and data reading and writing: After receiving the task, the task processor executes the reasoning task through the distributed reasoning engine and uses the key-value cache to record the processing progress of the task. The task processor will perform data reading and writing operations during the execution process to ensure that the task can be completed smoothly.

[0123] like Figure 5 As shown in the figure, this scheme can be summarized into the following three steps: Step R1: Load monitoring and prediction: The operation monitoring module in the task processor cluster collects the current load data and sends it to the task dispatcher. The task dispatcher uses this data to make load predictions and evaluate future load conditions in order to make corresponding resource allocation decisions.

[0124] Step R2: Data processing decision configuration: The task distributor performs data processing and decision configuration based on the current load data and prediction results. Specifically, the task distributor sets the chain interval configuration strategy through the interval configuration module and generates the system optimization target. Then, the task distributor adjusts the load distribution of the task processor based on this decision information to ensure the efficient operation of the system.

[0125] Step R3: Task distribution and migration: The task distributor distributes the tasks submitted by the user to the appropriate task processor according to the decision configuration. If the load of the task processor exceeds the threshold range, the task distributor will also control the migration of tasks and transfer some tasks to other task processors to achieve load balancing and resource optimization.

[0126] In summary, in the embodiments of the present application, by combining and analyzing the load data of the real-time monitoring reasoning task with the historical data, a more optimized resource allocation decision is made comprehensively to predict future load trends, and a reasonable resource allocation strategy is formulated to adjust the current reasoning task load, to ensure that the load of each task processor operates within a reasonable range, to avoid overload or idle resources, to improve resource utilization, to achieve dynamic scheduling of resources, to improve the flexibility and adaptability of the system, and to better respond to requests for different voluntary requirements. Thus, based on the method of the embodiments of the present application, by dynamically monitoring and adjusting the task load, the system can allocate resources more efficiently, improve the overall reasoning efficiency and response speed. The chain decoupling of resources in the distributed machine learning reasoning process is achieved, solving the problem of low system efficiency in the distributed machine learning reasoning process.

[0127] refer to Figure 6 , which shows a chain resource decoupling device 30 for distributed machine learning reasoning provided by an embodiment of the present application, applied to a task processor of a distributed machine learning system, including: The monitoring module 301 is used to obtain the current load data of the inference task being executed; the current load data is used to characterize the task load generated by the inference task; The monitoring data sending module 302 is used to send the current load data to the task distributor of the distributed machine learning system, so that after obtaining the historical load data and the current load data sent from all the task processors in the distributed system, the task distributor determines the data processing decision of the distributed machine learning system according to all the current load data and the historical load data, and sends the data processing decision to each task processor respectively; the data processing decision includes the load characteristic threshold range defined for each task processor in the distributed machine learning system; the historical load data is used to characterize the task load generated by all the historical reasoning tasks in the distributed machine learning system; The decision execution module 303 is used to receive the data processing decision sent from the task distributor, and adjust the reasoning task being executed according to the data processing decision, so that the task load generated by the adjusted reasoning task matches the load characteristic threshold range corresponding to the task processor.

[0128] Optionally, the decision execution module 303 includes: A first target submodule is used to determine a first target task processor in the task processors of the distributed machine learning system when the current load data does not match the load characteristic threshold range corresponding to the task processor; the first target task processor is a task processor corresponding to the first target load characteristic threshold range matching the current load data in the distributed machine learning system; The first sending submodule is used to send the task content of the reasoning task being executed, the task metadata of the reasoning task, and the key-value cache data of the reasoning task to the first target task processor, so that the first target task processor can re-establish the reasoning task according to the task content of the reasoning task, and continue to process the task metadata of the reasoning task according to the key-value cache data of the reasoning task, so that the reasoning task can continue to be executed; the key-value cache data is used to record the processing progress of the reasoning task for the corresponding task metadata.

[0129] Optionally, when the task processor is determined to be the first target task processor, the chain resource decoupling device 30 for distributed machine learning reasoning further includes: A first acquisition module is used to acquire the task content of the inference task sent from other task processors in the distributed machine learning system, the task metadata of the inference task, and the key-value cache data of the inference task; the key-value cache data is used to record the processing progress of the inference task for the corresponding task metadata; The first execution module is used to re-establish the reasoning task according to the task content of the reasoning task, and continue to process the task metadata of the reasoning task according to the key value cache data of the reasoning task so that the reasoning task continues to be executed; the first target task processor is the task processor corresponding to the first target load feature threshold range that matches the current load data in the distributed machine learning system.

[0130] Optionally, when the task processor is determined to be the second target task processor, the chain resource decoupling device 30 for distributed machine learning reasoning further includes: A second acquisition module is used to acquire the task content of the to-be-executed reasoning task sent from the task distributor and the task metadata of the to-be-executed reasoning task; The second execution module is used to establish a to-be-executed reasoning task according to the task content of the to-be-executed reasoning task, to process the task metadata of the to-be-executed reasoning task, and to establish key-value cache data of the to-be-executed reasoning task, to record the processing progress of the task metadata of the to-be-executed reasoning task; the second target task processor is a task processor corresponding to the second target load characteristic threshold range that matches the initial load of the to-be-executed reasoning task in the distributed machine learning system.

[0131] refer to Figure 7 , which shows a chain resource decoupling device 40 for distributed machine learning reasoning provided by an embodiment of the present application, applied to a task distributor of a distributed machine learning system, including: The monitoring data acquisition module 401 is used to acquire historical load data and current load data sent from each task processor in the distributed system; the current load data is used to characterize the task load generated by the inference task being executed by the corresponding task processor; the historical load data is used to characterize the task load generated by all historical inference tasks in the distributed machine learning system; A decision module 402 is used to determine a data processing decision of the distributed machine learning system based on all current load data and historical load data; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; The decision sending module 403 is used to send the data processing decision to each task processor respectively, so that after receiving the data processing decision, the task processor adjusts the reasoning task being executed by the task processor according to the data processing decision, so that the task load generated by the adjusted reasoning task of the task processor matches the load characteristic threshold range corresponding to the task processor.

[0132] Optionally, the task load includes the text length of the data processed by the corresponding task processor, and the historical load data includes the historical length of the data processed by the distributed machine learning system; the decision module 402 includes: An interval division submodule, used to determine a text length interval range corresponding to a maximum text length, and divide the text length interval range into a plurality of continuous and equal-width text length interval subranges according to a preset number of intervals; A traversal submodule is used to take the number of text length interval subranges corresponding to the test interval extracted from the text length interval range as a traversal parameter, and the text length and the history length as statistical parameters, and calculate the standardized iteration delay evaluation value corresponding to the test interval when the traversal parameter takes different values; the text length interval subranges in the test interval are continuous; the first text length interval subrange in the continuous text length interval subranges in the test interval is the first text length interval subrange in the remaining text length interval subranges in the text length interval range; A selection submodule is used to use the test interval corresponding to the minimum standardized iterative delay evaluation value among all the standardized iterative delay evaluation values ​​as the target load characteristic threshold range; An allocating submodule, for allocating the target load characteristic threshold range to a task processor to which the load characteristic threshold range is not allocated; The iterative submodule is used to remove the target load feature threshold range from the text length interval range to update the text length interval range, and return to the test interval extracted from the text length interval range, with the corresponding number of text length interval sub-ranges as the traversal parameter, the text length and the history length as the statistical parameters, and calculate the standardized iterative delay evaluation value corresponding to the test interval when the traversal parameter takes different values.

[0133] Optionally, the chain resource decoupling device 40 for distributed machine learning reasoning further includes: A new task acquisition module is used to obtain the reasoning tasks to be executed and the task metadata of the reasoning tasks to be executed; A second selection module is used to determine the initial load of the to-be-executed reasoning task according to the task metadata, and determine a second target task processor among all the task processors according to the initial load and the data processing decision; the second target task processor is a task processor corresponding to a second target load feature threshold range that matches the initial load in the distributed machine learning system; The second sending module is used to send the task content and task metadata of the to-be-executed reasoning task to the second target task processor, so that the to-be-executed reasoning task is established in the second target task processor to process the task metadata, and establish key-value cache data of the to-be-executed reasoning task to record the processing progress of the task metadata by the to-be-executed reasoning task.

[0134] In summary, in the embodiments of the present application, by combining and analyzing the load data of the real-time monitoring reasoning task with the historical data, a more optimized resource allocation decision is made comprehensively to predict future load trends, and a reasonable resource allocation strategy is formulated to adjust the current reasoning task load, to ensure that the load of each task processor operates within a reasonable range, to avoid overload or idle resources, to improve resource utilization, to achieve dynamic scheduling of resources, to improve the flexibility and adaptability of the system, and to better respond to requests for different voluntary requirements. Thus, based on the method of the embodiments of the present application, by dynamically monitoring and adjusting the task load, the system can allocate resources more efficiently, improve the overall reasoning efficiency and response speed. The chain decoupling of resources in the distributed machine learning reasoning process is achieved, solving the problem of low system efficiency in the distributed machine learning reasoning process.

[0135] Reference Figure 8 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0136] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the chain resource decoupling method for distributed machine learning reasoning described above. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0137] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0138] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0139] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0140] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0141] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0142] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0143] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0144] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the chain resource decoupling method for distributed machine learning reasoning provided in an embodiment of the present application.

[0145] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the above instructions can be executed by the processor 520 of the electronic device 500 to complete the above-mentioned chain resource decoupling method for distributed machine learning reasoning. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0146] Fig. 9600 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server. Fig. 9 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions executable by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the chain resource decoupling method for distributed machine learning reasoning provided in an embodiment of the present application.

[0147] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0148] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0149] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A chain resource decoupling method for distributed machine learning reasoning, applied to a task processor of a distributed machine learning system, characterized in that: include: Acquire current load data of the inference task being executed; the current load data is used to characterize the task load generated by the inference task; The current load data is sent to a task distributor of a distributed machine learning system, so that after obtaining the historical load data and the current load data sent from all the task processors in the distributed system, the task distributor determines the data processing decision of the distributed machine learning system according to all the current load data and the historical load data, and sends the data processing decision to each of the task processors respectively; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system; Receive a data processing decision sent from the task distributor, and adjust the reasoning task being executed according to the data processing decision, so that the task load generated by the adjusted reasoning task matches the load characteristic threshold range corresponding to the task processor.

2. The method according to claim 1, characterized in that The step of adjusting the inference task being executed according to the data processing decision so that the task load generated by the adjusted inference task matches a load characteristic threshold range corresponding to the task processor includes: In the case where the current load data does not match the load characteristic threshold range corresponding to the task processor, a first target task processor is determined among the task processors of the distributed machine learning system; the first target task processor is a task processor corresponding to the first target load characteristic threshold range matching the current load data in the distributed machine learning system; The task content of the reasoning task being executed, the task metadata of the reasoning task, and the key-value cache data of the reasoning task are sent to the first target task processor, so that the first target task processor re-establishes the reasoning task according to the task content of the reasoning task, and continues to process the task metadata of the reasoning task according to the key-value cache data of the reasoning task, so that the reasoning task continues to be executed; the key-value cache data is used to record the processing progress of the reasoning task for the corresponding task metadata.

3. The method according to claim 1, characterized in that In the case where the task processor is determined to be the second target task processor, the chain resource decoupling method for distributed machine learning reasoning further includes: Acquire the task content of the to-be-executed reasoning task sent from the task distributor, and the task metadata of the to-be-executed reasoning task; The reasoning task to be executed is established according to the task content of the reasoning task to be executed to process the task metadata of the reasoning task to be executed, and key-value cache data of the reasoning task to be executed is established to record the processing progress of the task metadata of the reasoning task to be executed; the second target task processor is the task processor corresponding to the second target load characteristic threshold range that matches the initial load of the reasoning task to be executed in the distributed machine learning system.

4. A chain resource decoupling method for distributed machine learning reasoning, applied to a task distributor of a distributed machine learning system, characterized in that: include: Obtain historical load data and current load data sent from each task processor in the distributed system; The current load data is used to characterize the task load generated by the inference task being executed by the corresponding task processor; the historical load data is used to characterize the task load generated by all historical inference tasks in the distributed machine learning system; Determining a data processing decision of the distributed machine learning system based on all of the current load data and the historical load data; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; The data processing decision is sent to each of the task processors respectively, so that after receiving the data processing decision, the task processor adjusts the reasoning task being executed by the task processor according to the data processing decision, so that the task load generated by the adjusted reasoning task of the task processor matches the load characteristic threshold range corresponding to the task processor.

5. The method according to claim 4, characterized in that The task load includes the text length of data processed by the corresponding task processor, and the historical load data includes the historical length of data processed by the distributed machine learning system; and determining the data processing decision of the distributed machine learning system based on all of the current load data and the historical load data includes: Determine a text length interval range corresponding to the maximum text length, and divide the text length interval range into a plurality of continuous and equal-width text length interval sub-ranges according to a preset number of intervals; The number of text length interval sub-ranges corresponding to the test interval extracted from the text length interval range is used as a traversal parameter, and the text length and the history length are used as statistical parameters to calculate the standardized iteration delay evaluation value corresponding to the test interval when the traversal parameter takes different values; the text length interval sub-ranges in the test interval are continuous; the first text length interval sub-range in the continuous text length interval sub-ranges in the test interval is the first text length interval sub-range in the remaining text length interval sub-ranges in the text length interval range; The test interval corresponding to the minimum standardized iterative delay evaluation value among all the standardized iterative delay evaluation values ​​is used as the target load characteristic threshold range; assigning the target load characteristic threshold range to a task processor to which no load characteristic threshold range is assigned; The target load feature threshold range is removed from the text length interval range to update the text length interval range, and the test interval to be extracted from the text length interval range is returned to, the number of corresponding text length interval sub-ranges is used as the traversal parameter, the text length and the history length are used as statistical parameters, and the standardized iterative delay evaluation value corresponding to the test interval is calculated when the traversal parameter takes different values.

6. The method according to claim 4, characterized in that The chain resource decoupling method for distributed machine learning reasoning also includes: Obtaining a to-be-executed reasoning task and task metadata of the to-be-executed reasoning task; Determine the initial load of the to-be-executed reasoning task according to the task metadata, and determine a second target task processor among all task processors according to the initial load and the data processing decision; the second target task processor is a task processor corresponding to a second target load feature threshold range matching the initial load in the distributed machine learning system; The task content and task metadata of the to-be-executed reasoning task are sent to the second target task processor, so that the to-be-executed reasoning task is established in the second target task processor to process the task metadata, and key-value cache data of the to-be-executed reasoning task is established to record the processing progress of the to-be-executed reasoning task on the task metadata.

7. A chain resource decoupling device for distributed machine learning reasoning, applied to a task processor of a distributed machine learning system, characterized in that: include: A monitoring module, used to obtain current load data of the inference task being executed; the current load data is used to characterize the task load generated by the inference task; A monitoring data sending module, used for sending the current load data to a task distributor of a distributed machine learning system, so that after obtaining the historical load data and the current load data sent from all the task processors in the distributed system, the task distributor determines the data processing decision of the distributed machine learning system according to all the current load data and the historical load data, and sends the data processing decision to each of the task processors respectively; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system; A decision execution module is used to receive a data processing decision sent from the task distributor, and adjust the reasoning task being executed according to the data processing decision, so that the task load generated by the adjusted reasoning task matches the load characteristic threshold range corresponding to the task processor.

8. A chain resource decoupling device for distributed machine learning reasoning, applied to a task distributor of a distributed machine learning system, characterized in that: include: A monitoring data acquisition module is used to acquire historical load data and current load data sent from each task processor in the distributed system; the current load data is used to characterize the task load generated by the reasoning task being executed by the corresponding task processor; the historical load data is used to characterize the task load generated by all historical reasoning tasks in the distributed machine learning system; A decision module, configured to determine a data processing decision of the distributed machine learning system based on all of the current load data and the historical load data; the data processing decision includes a load characteristic threshold range defined for each task processor in the distributed machine learning system; A decision sending module is used to send the data processing decision to each of the task processors respectively, so that after receiving the data processing decision, the task processor adjusts the reasoning task being executed by the task processor according to the data processing decision, so that the task load generated by the adjusted reasoning task of the task processor matches the load characteristic threshold range corresponding to the task processor.

9. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the chain resource decoupling method for distributed machine learning reasoning as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the chain resource decoupling method for distributed machine learning reasoning as described in any one of claims 1 to 6.

Images