Processing method of reasoning task, clustering device, storage medium and computer product

Abstract

The invention discloses an inference task processing method, a cluster device, a storage medium and a computer product, and relates to the technical field of data processing.The inference task processing method is applied to the cluster device, the cluster device comprises a plurality of macro instances, each macro instance comprises a plurality of single instances, and the inference task processing method specifically comprises the steps that a plurality of inference tasks are received; issuing the plurality of reasoning tasks to a matched target macro instance, and screening a plurality of single instances in the target macro instance to determine a target single instance; and issuing the plurality of reasoning tasks to the target single instance, and processing the plurality of reasoning tasks through the target single instance to obtain a reasoning result, thereby achieving the technical effect of improving the processing efficiency of the large model on the reasoning tasks.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for processing inference tasks, a cluster device, a storage medium, and a computer program product. Background Technology

[0002] With the continuous development of large model technology, the reasoning ability of large models has been significantly improved. In the process of large models handling reasoning tasks, they usually predict the probability distribution of the next word based on the input and the generated text. This process involves a large amount of repetitive computation. In order to optimize computational efficiency, the industry usually adopts a key-value (KV) caching mechanism to divide the reasoning process into a prefill stage and a decoding stage. In the prefill stage, the large model processes the input data and fills the KV cache, while in the decoding stage, the large model generates the output sequence step by step based on the KV cache.

[0003] In related technologies, cluster service solutions for large model inference mainly adopt the P (Prefill) / D (Decode) fully separated system solution and the P / D non-separated system solution. The P / D fully separated system solution divides the inference cluster into a prefill cluster and a decoding cluster, and uses KV cache to transmit data between the clusters so that the prefill cluster and the decoding cluster can cooperate to complete the inference task. The P / D non-separated system solution allows a single instance to run the Prefill stage and the Decode stage alternately to complete the inference task.

[0004] However, the P / D fully separated system scheme requires the transfer of a large amount of KV cache data between clusters, resulting in high latency in the inference task processing. On the other hand, the P / D non-separated system scheme, due to the difference in computational characteristics between the Prefill and Decode stages, will generate a large number of pipeline bubbles when pipeline processing multiple inference tasks. Thus, both the P / D fully separated system scheme and the P / D non-separated system scheme result in low processing efficiency for inference tasks. Summary of the Invention

[0005] The main objective of this application is to provide a method for processing inference tasks, a cluster device, a storage medium, and a computer program product, aiming to solve the technical problem of low processing efficiency of large models for inference tasks in related technologies.

[0006] To achieve the above objectives, this application proposes a method for processing inference tasks. This method is applied to a cluster device, which includes multiple macro instances, each macro instance containing multiple single instances. The method for processing inference tasks includes:

[0007] Receive multiple inference tasks;

[0008] Multiple inference tasks are distributed to matching target macro instances, and multiple single instances in the target macro instances are filtered to determine the target single instance;

[0009] Multiple inference tasks are distributed to the target single instance, and the target single instance processes the multiple inference tasks to obtain the inference result.

[0010] In one embodiment, the step of distributing the plurality of inference tasks to the matching target macro instance includes:

[0011] Detect the request receiving switch status corresponding to each of the multiple macro instances, wherein the request receiving switch status includes an on state and a off state;

[0012] The target macro instance is determined based on the request receiving switch status of each of the aforementioned requests, and multiple inference tasks are distributed to the target macro instance, wherein the request receiving switch status of the target macro instance is in the on state.

[0013] In one embodiment, the step of filtering multiple single instances from the target macro instance to determine the target single instance includes:

[0014] Determine the running state of each of the multiple single instances in the target macro instance;

[0015] Based on the aforementioned running states, multiple single instances to be filtered are determined from among multiple single instances, wherein the running state of the single instances to be filtered is a pre-filled running state.

[0016] Determine the instance load parameters that match each of the multiple single instances to be filtered, and identify the single instance with the smallest instance load parameter as the target single instance.

[0017] In one embodiment, the step of processing multiple inference tasks through the target single instance to obtain inference results includes:

[0018] The inference task is pre-filled using the target single instance to obtain intermediate results;

[0019] Determine the inference evaluation index generated by the target single instance during the pre-filling calculation process;

[0020] If the inference evaluation index is detected to reach the preset evaluation index threshold, the running state of the target single instance is controlled to switch from the pre-filled running state to the decoding running state;

[0021] The intermediate results are decoded and processed by a single target instance in the decoding running state to obtain the inference result.

[0022] In one embodiment, prior to the step of pre-filling computational processing of the inference task using the target single instance to obtain intermediate results, the method further includes:

[0023] Determine the priority of the request task corresponding to each of the multiple inference tasks;

[0024] The multiple inference tasks are sorted according to the priority of each request task to obtain a sorting result;

[0025] Based on the sorting results, the inference task with the highest priority is determined, and the step of pre-filling the inference task with the target single instance to obtain intermediate results is executed.

[0026] In one embodiment, the step of determining the inference evaluation index generated by the target single instance during the pre-filling calculation process includes:

[0027] Determine the memory usage parameters, task completion parameters, and computational intensity parameters generated by the target single instance during the pre-filling computation process;

[0028] Determine the corresponding index weights for the video memory usage parameters, the task completion parameters, and the computational intensity parameters;

[0029] The inference evaluation index is obtained by weighting the memory usage parameter, the task completion parameter, and the computational intensity parameter according to the weights of each of the aforementioned indicators.

[0030] In one embodiment, after the step of distributing the plurality of inference tasks to the matching target macro instance, the method further includes:

[0031] Determine the running state of each of the multiple single instances in the target macro instance;

[0032] Given that all the aforementioned running states are in the decoding running state, determine the single instance with the minimum instance load parameter;

[0033] The running state of the single instance with the smallest instance load parameter is switched from the decoding running state to the pre-filling running state, and the step of distributing multiple inference tasks to the target single instance is executed.

[0034] Furthermore, to achieve the above objectives, this application also proposes a cluster device, the device comprising: multiple macro instances, multiple single instances, a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the processing method for the inference task as described above.

[0035] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the processing method for the reasoning task as described above.

[0036] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the processing method for the reasoning task as described above.

[0037] The inference task processing method provided in this application embodiment is applied to a cluster device. The cluster device includes multiple macro instances, and each macro instance includes multiple single instances. The method involves receiving multiple inference tasks; distributing the multiple inference tasks to a matching target macro instance; filtering the multiple single instances in the target macro instance to determine a target single instance; distributing the multiple inference tasks to the target single instance; and processing the multiple inference tasks through the target single instance to obtain inference results.

[0038] In this embodiment, when the cluster device is running, it first receives multiple inference tasks issued by the user. Then, based on the multiple inference tasks, the cluster device filters out a matching target macro instance from its configured macro instances and sends the multiple inference tasks to the target macro instance. The target macro instance then filters its configured single instances based on the multiple inference tasks to determine the target single instance that can process the multiple inference tasks. Finally, the target macro instance sends the multiple inference tasks to the target single instance, which processes the multiple inference tasks to obtain the inference results.

[0039] Thus, this application solves the technical problem of low processing efficiency of large models for inference tasks in related technologies. Specifically, this application aggregates multiple single instances into a macro instance and integrates multiple macro instances within a cluster device. The cluster device distributes inference tasks to single instances capable of inference through the macro instances. A single instance processes multiple inference tasks in the Prefill running state to obtain intermediate results, and then dynamically switches to the Decode running state to continue processing the intermediate results. This allows the cluster device to complete the prefilling and decoding of multiple inference tasks through a single instance. This avoids the situation where a large amount of prefilled data is transmitted through a KV cache, resulting in low processing efficiency, as seen in the P / D fully separated system scheme. It also avoids the situation where the difference in computational characteristics between the Prefill and Decode stages in the P / D non-separated system scheme leads to a large number of pipeline bubbles during pipeline processing of multiple inference tasks, resulting in low processing efficiency. Therefore, this application achieves the technical effect of improving the processing efficiency of large models for inference tasks. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram illustrating the differences in computational characteristics of a non-separated P / D system scheme involved in an embodiment of the reasoning task processing method of this application;

[0043] Figure 2 A flowchart illustrating the first embodiment of the reasoning task processing method of this application;

[0044] Figure 3 This is a schematic diagram of the cluster device involved in one embodiment of the reasoning task processing method of this application;

[0045] Figure 4 This is a schematic diagram of the module structure of the reasoning task processing system in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the reasoning task processing method in the embodiments of this application.

[0047] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0049] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0050] In this embodiment, for ease of description, the following description uses a cluster device internally configured with multiple macro instances, or a mobile terminal, data storage control terminal, PC, or other terminal connected to an electronic control unit associated with the cluster device, as the execution subject. It should be noted that each macro instance is configured with multiple single instances, and each single instance can switch between a pre-filling running state and a decoding running state. In the pre-filling running state, a single instance can perform pre-filling calculations on the inference task to obtain intermediate results; similarly, in the decoding running state, a single instance can decode the obtained intermediate results to obtain the inference result.

[0051] Based on the aforementioned cluster device, the overall concept of the reasoning task processing method of this application is proposed here.

[0052] With the continuous development of large model technology, the inference capabilities of large models have been significantly improved. In the process of handling inference tasks, large models typically predict the probability distribution of the next word based on the input and the generated text. This process involves a large amount of repetitive computation. To optimize computational efficiency, the industry commonly employs a key-value (KV) caching mechanism to divide the inference process into a prefill stage and a decoding stage. In the prefill stage, the large model processes the input data and fills the KV cache, while in the decoding stage, the large model gradually generates the output sequence based on the KV cache. In related technologies, cluster service solutions for large model inference mainly adopt two approaches: a fully separated P / D system and a non-separated P / D system. The fully separated P / D system divides the inference cluster into a prefill cluster and a decoding cluster, using KV caching for data transmission between the clusters to enable the filling and decoding clusters to collaborate in completing the inference task. The non-separated P / D system, on the other hand, allows a single instance to alternately run the prefill and decoding stages to complete the inference task. However, the P / D fully separated system solution requires the transfer of a large amount of KV cache data between clusters, resulting in high latency in the inference task processing. Furthermore, please refer to... Figure 1 , Figure 1 This is a schematic diagram illustrating the computational characteristic differences of a P / D non-separated system scheme involved in an embodiment of the reasoning task processing method of this application, as shown below. Figure 1 As shown, the P / D non-separated system scheme, due to the difference in computational characteristics between the Prefill and Decode stages, will generate a large number of pipeline bubbles when pipelined processing multiple inference tasks. Thus, both the P / D fully separated system scheme and the P / D non-separated system scheme will result in low processing efficiency of inference tasks.

[0053] To address the above issues, this application provides a method for processing inference tasks. This method is applied to a cluster device, which includes multiple macro instances, each macro instance containing multiple single instances. The method includes: receiving multiple inference tasks; distributing the multiple inference tasks to a matching target macro instance, and filtering the multiple single instances within the target macro instance to determine a target single instance; distributing the multiple inference tasks to the target single instance, and processing the multiple inference tasks through the target single instance to obtain inference results.

[0054] Thus, this application solves the technical problem of low processing efficiency of large models for inference tasks in related technologies. Specifically, this application aggregates multiple single instances into a macro instance and integrates multiple macro instances within a cluster device. The cluster device distributes inference tasks to single instances capable of inference through the macro instances. A single instance processes multiple inference tasks in the Prefill running state to obtain intermediate results, and then dynamically switches to the Decode running state to continue processing the intermediate results. This allows the cluster device to complete the prefilling and decoding of multiple inference tasks through a single instance. This avoids the situation where a large amount of prefilled data is transmitted through a KV cache, resulting in low processing efficiency, as seen in the P / D fully separated system scheme. It also avoids the situation where the difference in computational characteristics between the Prefill and Decode stages in the P / D non-separated system scheme leads to a large number of pipeline bubbles during pipeline processing of multiple inference tasks, resulting in low processing efficiency. Therefore, this application achieves the technical effect of improving the processing efficiency of large models for inference tasks.

[0055] Based on the overall concept of the reasoning task processing method of this application, the embodiments of this application provide a reasoning task processing method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of the reasoning task processing method of this application. In this embodiment, the reasoning task processing method is applied to a cluster device, the cluster device comprising multiple macro instances, each macro instance comprising multiple single instances, and the reasoning task processing method includes steps S10 to S30:

[0056] Step S10: Receive multiple inference tasks;

[0057] Step S20: Distribute multiple inference tasks to the matching target macro instance, and filter multiple single instances in the target macro instance to determine the target single instance;

[0058] Step S30: Distribute multiple inference tasks to the target single instance, and process the multiple inference tasks through the target single instance to obtain inference results.

[0059] It should be noted that, please refer to Figure 3 , Figure 3 This is a schematic diagram of the cluster device involved in one embodiment of the reasoning task processing method of this application, as shown below. Figure 3 As shown, a single instance is the basic building block of the system, responsible for specific inference processing tasks. Furthermore, a macro instance, composed of multiple single instances, is responsible for the overall scheduling and management of inference tasks. Additionally, the cluster device, composed of multiple macro instances, supports the operation of the entire system, receiving inference tasks from users, distributing these tasks, and outputting the corresponding inference results.

[0060] In this embodiment, when the cluster device is running, it first listens for inference tasks from the user. Then, the cluster device determines the target macro instance that can process multiple inference tasks from its configured macro instances and sends the received multiple inference tasks to the target macro instance. At this time, the target macro instance filters its configured single instances to determine the target single instance that can process multiple inference tasks. Finally, the target macro instance sends multiple inference tasks to the target single instance, which performs pre-filling calculation processing on the multiple inference tasks. At the same time, after the target single instance completes the pre-filling calculation processing, the target macro instance controls the target single instance to switch its running state so that the target single instance can decode the intermediate results obtained after the pre-filling calculation processing to obtain the inference results corresponding to each of the multiple inference tasks. Then, the inference results are uploaded to the cluster device through the target macro instance, and the cluster device outputs multiple inference results for the user to view.

[0061] For example, when the cluster device is running, the cluster inference server system integrated within it first listens in real time for multiple inference tasks issued by users through a listening server configured within the cluster device. Then, the cluster inference server system filters through multiple macro instances configured within itself to determine the target macro instance capable of processing the multiple inference tasks. It then distributes the multiple inference tasks to the target macro instance through an interface matching the target macro instance. The macro instance scheduler configured within the target macro instance interacts with the individual instance inference engines corresponding to the multiple single instances configured under the target macro instance via RPC to filter the multiple single instances, thereby determining the target single instance that is in a Prefill state and capable of processing multiple inference tasks. Finally, the macro instance scheduler uses RPC (Remote Procedure Call) to... A Remote Procedure Call (RPC) distributes multiple inference tasks to a target single instance. The target single instance then uses its configured single-instance inference engine to perform Prefill calculations on each inference task. Simultaneously, when the single-instance inference engine detects that the current memory usage parameters have reached a preset threshold, it controls the target single instance to switch from the Prefill state to the Decdoe running state (i.e., the decoding running state) after completing one or more inference tasks. This allows for Decdo calculations on one or more intermediate results obtained from the Prefill calculations to obtain the inference results corresponding to each inference task. The target single instance then uploads each inference result to the cluster inference server system through the target macro instance. The cluster inference server system uses a streaming output mechanism to output the multiple inference result tokens word by word for users to view.

[0062] Thus, this application solves the technical problem of low processing efficiency of large models for inference tasks in related technologies. Specifically, this application aggregates multiple single instances into a macro instance and integrates multiple macro instances within a cluster device. The cluster device distributes inference tasks to single instances capable of inference through the macro instances. A single instance processes multiple inference tasks in the Prefill running state to obtain intermediate results, and then dynamically switches to the Decode running state to continue processing the intermediate results. This allows the cluster device to complete the prefilling and decoding of multiple inference tasks through a single instance. This avoids the situation where a large amount of prefilled data is transmitted through a KV cache, resulting in low processing efficiency, as seen in the P / D fully separated system scheme. It also avoids the situation where the difference in computational characteristics between the Prefill and Decode stages in the P / D non-separated system scheme leads to a large number of pipeline bubbles during pipeline processing of multiple inference tasks, resulting in low processing efficiency. Therefore, this application achieves the technical effect of improving the processing efficiency of large models for inference tasks.

[0063] In one feasible implementation, the step of "distributing the multiple inference tasks to the matching target macro instance" in step S20 above may specifically include steps S201 to S202:

[0064] Step S201: Detect the request receiving switch status corresponding to each of the multiple macro instances, wherein the request receiving switch status includes an on state and a off state;

[0065] Step S202: Determine the target macro instance according to the request receiving switch status of each of the above, and send the multiple inference tasks to the target macro instance, wherein the request receiving switch status of the target macro instance is in the on state.

[0066] It should be noted that the request receive switch status reflects whether a macro instance can receive inference tasks. In other words, when the request receive switch is on, it means the macro instance can process inference tasks through its contained single instances; similarly, when the request receive switch is off, it means the macro instance cannot process inference tasks through its contained single instances. It is also understood that the request receive switch status can be modified by the macro instance scheduler within the macro instance based on the running status of its multiple contained single instances.

[0067] For example, after receiving multiple inference tasks, the cluster inference server system first detects multiple macro instances configured in the cluster device to determine the request receiving switch status of each macro instance. Then, the cluster inference server filters the multiple macro instances based on the request receiving switch status to determine one or more macro instances to be filtered whose request receiving switch status is on. If the number of macro instances to be filtered is one, the cluster inference server determines the macro instance to be filtered as the target macro instance that can process multiple inference tasks. If the number of macro instances to be filtered is greater than one, the cluster inference server randomly selects one macro instance to be filtered as the target macro instance and distributes the multiple inference tasks to the target macro instance.

[0068] In addition, in this embodiment and another embodiment, after determining that the number of macro instances to be filtered is greater than one, the cluster inference server can also determine the instance load parameters of each single instance contained in the multiple macro instances to be filtered, and determine the macro instance to be filtered from the multiple macro instances to be filtered based on the instance load parameters of each instance, which is the single instance with the smallest instance load parameter, as the target macro instance for processing multiple inference tasks, and distribute the multiple inference tasks to the target macro instance.

[0069] In this way, the cluster inference service system can forward inference tasks to macro instances that can process them, ensuring that inference tasks can be processed in a timely manner.

[0070] In one feasible implementation, the step of "filtering multiple single instances in the target macro instance to determine the target single instance" in step S20 above may specifically include steps S203 to S205:

[0071] Step S203: Determine the running state of each of the multiple single instances in the target macro instance;

[0072] Step S204: Determine multiple single instances to be filtered from multiple single instances based on each of the aforementioned running states, wherein the running state of the single instances to be filtered is a pre-filled running state;

[0073] Step S205: Determine the instance load parameters that match each of the multiple single instances to be filtered, and determine the single instance to be filtered with the smallest instance load parameter as the target single instance.

[0074] For example, after the cluster inference service system distributes multiple inference tasks to the target macro instance, the macro instance scheduler integrated within the target macro instance first interacts with the individual instance inference engines corresponding to each of its multiple individual instances via RPC to determine whether the running state of each individual instance is Prefill or Decdoe. Then, the macro instance scheduler filters the multiple individual instances according to their running states to identify the individual instances in the Prefill state as the single instances to be filtered. Finally, if the number of single instances to be filtered is greater than 1, the macro instance scheduler determines the instance load parameters corresponding to each of the multiple individual instances and compares the instance load parameters to identify the single instance with the smallest instance load parameter as the target single instance for processing the multiple inference tasks.

[0075] In this way, macro instances within the cluster can assign inference tasks to the single instance configured to be in Prefill mode and with the least load each time an inference task is assigned, thereby avoiding overload of a single instance, further improving system stability and inference task efficiency.

[0076] In one feasible implementation, the step S30 above, "processing multiple inference tasks through the target single instance to obtain inference results," may specifically include steps S301 to S304:

[0077] Step S301: Perform pre-filling calculations on the inference task using the target single instance to obtain intermediate results;

[0078] Step S302: Determine the inference evaluation index generated by the target single instance during the pre-filling calculation process;

[0079] Step S303: When the inference evaluation index is detected to have reached the preset evaluation index threshold, the running state of the target single instance is controlled to switch from the pre-filled running state to the decoding running state;

[0080] Step S304: The intermediate result is decoded and calculated using a single target instance in the decoding running state to obtain the inference result.

[0081] For example, after the target macro instance distributes multiple inference tasks to the target single instance, the single-instance inference engine configured within the target single instance determines the target inference task that needs to be processed among the multiple inference tasks, and performs Prefill calculation on the target inference task to obtain the intermediate result corresponding to the target inference task. Then, the single-instance inference engine detects the inference evaluation index generated by the target single instance in the process of obtaining the intermediate result. After that, the single-instance inference engine reads the storage module configured in the cluster device to obtain the preset evaluation index threshold, and compares the inference evaluation index with the evaluation index threshold to obtain the comparison result. If the single-instance inference engine determines that the inference evaluation index has reached the evaluation index threshold, it determines that the load of the target single instance is too high at this time, and controls the target single instance to switch from the Prefill running state to the Decdoe running state. Finally, the single-instance inference engine controls the intermediate result obtained in the Decdoe running state to perform Decdoe calculation to obtain the inference result corresponding to the intermediate result, and uploads the inference result to the cluster device through the target macro instance.

[0082] It should be noted that, in this embodiment and another embodiment, after obtaining the intermediate result corresponding to the target inference task, if the single-instance inference engine detects that the inference evaluation index generated by the single instance in the process of obtaining the intermediate result has not reached the evaluation index threshold, it controls the single instance to continue to filter out the inference tasks that need to be processed from other pending inference tasks in multiple inference tasks, and performs Prefill calculation on the inference tasks that need to be processed to obtain intermediate results, until the inference evaluation index generated by the target single instance in the process of obtaining the intermediate result reaches the evaluation index threshold, then controls the target single instance to switch from the Prefill running state to the Decdoe running state, and makes the single instance in the Decdoe running state perform Decdoe calculation on each intermediate result to obtain the inference result corresponding to each processed inference task.

[0083] Furthermore, in this embodiment and another embodiment, after the single-instance inference engine switches to the Decdoe running state, it can continue to detect the inference evaluation index generated by the single instance in the Decdoe running state. If the inference evaluation index is detected to be lower than the above-mentioned evaluation index threshold, the single instance is controlled to switch from the Decdoe running state to the Prefill running state, so that the single instance performs Prefill calculation processing on the unprocessed inference tasks in the Prefill running state, and then repeats the above steps to complete the processing of all inference tasks.

[0084] In this way, by integrating the Prefill and Decdoe running states within a single instance and controlling the switching between these states to process inference tasks, the cluster device can maximize the throughput of a single instance. This avoids the situation where a large amount of prefilled data is transmitted through a KV cache, resulting in low inference task processing speed, which is common in P / D fully separated systems. It also prevents a single instance from consuming excessive GPU memory during the Prefill state, which could lead to insufficient GPU memory during the Decode state and pipeline bubbles. This further improves the processing efficiency of inference tasks.

[0085] In one feasible implementation, step S302 above may specifically include steps S3021 to S3023:

[0086] Step S3021: Determine the memory usage parameters, task completion parameters, and computational intensity parameters generated by the target single instance during the pre-filling computation process;

[0087] Step S3022: Determine the corresponding index weights for the video memory usage parameters, the task completion parameters, and the computational intensity parameters;

[0088] Step S3023: Perform weighted calculations on the memory usage parameters, task completion parameters, and computational intensity parameters according to the weights of each of the aforementioned indicators to obtain the inference evaluation index.

[0089] For example, after the single-instance inference scheduling engine controls the prefill computation of the target inference task to obtain intermediate results, the single-instance inference scheduling engine further detects the memory usage parameters and computation intensity parameters generated during the prefill computation of the target inference task. Simultaneously, the single-instance inference scheduling engine reads the request queue and determines the task completion parameters based on the request queue. Then, the single-instance inference scheduling engine reads the aforementioned storage module to obtain the indicator weight A corresponding to the memory usage parameters, the indicator weight B corresponding to the task completion parameters, and the indicator weight C corresponding to the computation intensity parameters. Finally, the single-instance inference scheduling engine calculates the memory usage parameters, task completion parameters, and computation intensity parameters based on indicator weights A, B, and C to obtain the inference evaluation indicators generated by the target single instance during the prefill computation of the target inference task.

[0090] Reasoning evaluation index = index weight A * video memory usage parameter + index weight B * task completion parameter + index weight C * computation intensity parameter.

[0091] In this way, the cluster device can detect the memory usage, task completion status, and computational intensity of a single instance when processing inference tasks. It can evaluate the load of a single instance based on multiple indicators such as memory usage, task completion status, and computational intensity, and then determine whether to control the state switching of the single instance according to the load status. This avoids the situation where a single instance occupies a lot of memory in the Prefill stage, which would cause the Decode stage to be stuck due to insufficient resources and generate a lot of pipeline bubbles, thereby further improving the processing efficiency of inference tasks.

[0092] Based on the first embodiment of this application, a second embodiment of this application is proposed herein. In this second embodiment, content that is the same as or similar to the above embodiments can be referred to the above description and will not be repeated hereafter. Furthermore, prior to step S301 above, the processing method for the reasoning task of this application may further include steps A10 to A30:

[0093] Step A10: Determine the priority of the request task corresponding to each of the multiple inference tasks;

[0094] Step A20: Sort the multiple inference tasks according to the priority of each request task to obtain a sorting result;

[0095] Step A30: Based on the sorting results, determine the inference task with the highest priority of the requested task, and execute the step of pre-filling the inference task with the target single instance to obtain intermediate results.

[0096] For example, after the macro instance scheduler distributes multiple inference tasks to the target single instance, the single instance inference scheduling engine first reads the task priorities contained in each of the multiple inference tasks. Then, the single instance inference scheduling engine sorts the multiple inference tasks according to the numerical value of the task priorities, in descending order of task priorities, to obtain a sorting result. Finally, the single instance inference scheduling engine determines the highest priority inference task among the multiple inference tasks according to the sorting result. The single instance inference scheduling engine then processes the highest priority inference task to obtain a matching intermediate result, and detects the evaluation index parameters generated in the process of obtaining the intermediate result. Then, it determines whether to switch the target single instance from the Prefill running state to the Decode running state according to the evaluation index parameters.

[0097] In this way, the cluster device can ensure that high-priority inference tasks are processed first by a single instance, thereby ensuring that critical inference tasks are allocated and processed in a timely manner, and effectively guaranteeing key performance indicators such as the first response time of inference tasks.

[0098] Based on the first and / or second embodiments of this application, a third embodiment of this application is proposed herein. In this third embodiment, content that is the same as or similar to the above embodiments can be referred to the above description and will not be repeated hereafter. Furthermore, after the step of "distributing multiple inference tasks to matching target macro instances" in step S20 above, the method for processing inference tasks in this application may further include steps B10 to B30:

[0099] Step B10: Determine the running status of each of the multiple single instances in the target macro instance;

[0100] Step B20: If all the described running states are determined to be decoding running states, determine the single instance with the minimum instance load parameter;

[0101] Step B30: Switch the running state of the single instance with the smallest instance load parameter from the decoding running state to the pre-filling running state, and execute the step of distributing multiple inference tasks to the target single instance.

[0102] For example, after the cluster inference service system distributes multiple inference tasks to the target macro instance, the macro instance scheduler within the target macro instance can first detect the running status of each of its multiple individual instances. Then, if the macro instance scheduler detects that the running status of each of the multiple individual instances is in the Decode running state, it further determines the instance load parameters of each of the multiple individual instances and determines the individual instance with the smallest instance load parameter. Finally, the macro instance scheduler switches the Decode running state to the Prefill running state of the individual instance with the smallest instance load parameter, and distributes multiple inference tasks to the individual instance with the smallest instance load parameter, so that the individual instance in the Prefill running state can process the multiple inference tasks to obtain the inference results.

[0103] In this way, the cluster device can ensure that newly received inference tasks can be processed, thereby further improving the processing efficiency of inference tasks.

[0104] This application also provides a system for processing reasoning tasks; please refer to [reference needed]. Figure 4 The processing system for the reasoning task includes:

[0105] Task receiving module 10 is used to receive multiple inference tasks;

[0106] The task distribution module 20 is used to distribute multiple inference tasks to the matching target macro instance, and to filter multiple single instances in the target macro instance to determine the target single instance;

[0107] The inference execution module 30 is used to distribute multiple inference tasks to the target single instance, and to process the multiple inference tasks through the target single instance to obtain inference results.

[0108] In one feasible implementation, the task distribution module 20 is further configured to:

[0109] Detect the request receiving switch status corresponding to each of the multiple macro instances, wherein the request receiving switch status includes an on state and a off state;

[0110] The target macro instance is determined based on the request receiving switch status of each of the aforementioned requests, and multiple inference tasks are distributed to the target macro instance, wherein the request receiving switch status of the target macro instance is in the on state.

[0111] In one feasible implementation, the task distribution module 20 is further configured to:

[0112] Determine the running state of each of the multiple single instances in the target macro instance;

[0113] Based on the aforementioned running states, multiple single instances to be filtered are determined from among multiple single instances, wherein the running state of the single instances to be filtered is a pre-filled running state.

[0114] Determine the instance load parameters that match each of the multiple single instances to be filtered, and identify the single instance with the smallest instance load parameter as the target single instance.

[0115] In one feasible implementation, the inference execution module 30 is further configured to:

[0116] The inference task is pre-filled using the target single instance to obtain intermediate results;

[0117] Determine the inference evaluation index generated by the target single instance during the pre-filling calculation process;

[0118] If the inference evaluation index is detected to reach the preset evaluation index threshold, the running state of the target single instance is controlled to switch from the pre-filled running state to the decoding running state;

[0119] The intermediate results are decoded and processed by a single target instance in the decoding running state to obtain the inference result.

[0120] In one feasible implementation, the inference execution module 30 is further configured to:

[0121] Determine the priority of the request task corresponding to each of the multiple inference tasks;

[0122] The multiple inference tasks are sorted according to the priority of each request task to obtain a sorting result;

[0123] Based on the sorting results, the inference task with the highest priority is determined, and the step of pre-filling the inference task with the target single instance to obtain intermediate results is executed.

[0124] In one feasible implementation, the inference execution module 30 is further configured to:

[0125] Determine the memory usage parameters, task completion parameters, and computational intensity parameters generated by the target single instance during the pre-filling computation process;

[0126] Determine the corresponding index weights for the video memory usage parameters, the task completion parameters, and the computational intensity parameters;

[0127] The inference evaluation index is obtained by weighting the memory usage parameter, the task completion parameter, and the computational intensity parameter according to the weights of each of the aforementioned indicators.

[0128] In one feasible implementation, the task distribution module 20 is further configured to:

[0129] Determine the running state of each of the multiple single instances in the target macro instance;

[0130] Given that all the aforementioned running states are in the decoding running state, determine the single instance with the minimum instance load parameter;

[0131] The running state of the single instance with the smallest instance load parameter is switched from the decoding running state to the pre-filling running state, and the step of distributing multiple inference tasks to the target single instance is executed.

[0132] The reasoning task processing system provided in this application, employing the reasoning task processing method in the above embodiments, can solve the technical problem of low processing efficiency of large models for reasoning tasks in related technologies. Compared with the prior art, the beneficial effects of the reasoning task processing system provided in this application are the same as those of the reasoning task processing method provided in the above embodiments, and other technical features of the reasoning task processing system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0133] This application provides a clustering device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the processing method of the inference task in the above embodiment 1.

[0134] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a cluster device suitable for implementing embodiments of this application. The cluster device in these embodiments may include, but is not limited to, a cluster device internally configured with multiple macro instances, or a mobile terminal, data storage control terminal, PC, or other terminal connected to an electronic control unit associated with the cluster device. Figure 5 The cluster device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0135] like Figure 5As shown, the cluster device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory 1002 or a program loaded from storage device 1003 into random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the cluster device. The processing unit 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. Input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to input / output interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the cluster device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagram shows a cluster device with various systems, it should be understood that it is not required to implement or have all of the systems shown. Alternatively, more or fewer systems may be implemented.

[0136] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0137] The cluster device provided in this application, employing the reasoning task processing method in the above embodiments, can solve the technical problem of low processing efficiency of large models for reasoning tasks in related technologies. Compared with the prior art, the beneficial effects of the cluster device provided in this application are the same as those of the reasoning task processing method provided in the above embodiments, and other technical features in this cluster device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0140] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the processing method of the reasoning task in the above embodiments.

[0141] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0142] The aforementioned computer-readable storage medium may be included in the cluster device or may exist independently and not assembled into the cluster device.

[0143] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the cluster device, the cluster device causes the cluster device to: receive multiple inference tasks; distribute the multiple inference tasks to matching target macro instances, and filter multiple single instances in the target macro instances to determine a target single instance; distribute the multiple inference tasks to the target single instance, and process the multiple inference tasks through the target single instance to obtain inference results.

[0144] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0147] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the processing method of the above-described reasoning task, which can solve the technical problem of low processing efficiency of large models for reasoning tasks in related technologies. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the processing method of the reasoning task provided in the above embodiments, and will not be repeated here.

[0148] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the processing method for the reasoning task as described above.

[0149] The computer program product provided in this application can solve the technical problem of low processing efficiency of large models for inference tasks in related technologies. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the inference task processing method provided in the above embodiments, and will not be repeated here.

[0150] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for processing reasoning tasks, characterized in that, The inference task processing method is applied to a cluster device, which contains multiple macro instances, each macro instance containing multiple single instances. The inference task processing method includes: Receive multiple inference tasks; Multiple inference tasks are distributed to matching target macro instances, and multiple single instances in the target macro instances are filtered to determine the target single instance; Multiple inference tasks are distributed to the target single instance, and the target single instance processes the multiple inference tasks to obtain the inference result.

2. The method for processing reasoning tasks as described in claim 1, characterized in that, The step of distributing multiple inference tasks to matching target macro instances includes: Detect the request receiving switch status corresponding to each of the multiple macro instances, wherein the request receiving switch status includes an on state and a off state; The target macro instance is determined based on the request receiving switch status of each of the aforementioned requests, and multiple inference tasks are distributed to the target macro instance, wherein the request receiving switch status of the target macro instance is in the on state.

3. The method for processing reasoning tasks as described in claim 2, characterized in that, The step of filtering multiple single instances in the target macro instance to determine the target single instance includes: Determine the running state of each of the multiple single instances in the target macro instance; Based on the aforementioned running states, multiple single instances to be filtered are determined from among multiple single instances, wherein the running state of the single instances to be filtered is a pre-filled running state. Determine the instance load parameters that match each of the multiple single instances to be filtered, and identify the single instance with the smallest instance load parameter as the target single instance.

4. The method for processing reasoning tasks as described in claim 3, characterized in that, The step of processing multiple inference tasks through the target single instance to obtain inference results includes: The inference task is pre-filled using the target single instance to obtain intermediate results; Determine the inference evaluation index generated by the target single instance during the pre-filling calculation process; If the inference evaluation index is detected to reach the preset evaluation index threshold, the running state of the target single instance is controlled to switch from the pre-filled running state to the decoding running state; The intermediate results are decoded and processed by a single target instance in the decoding running state to obtain the inference result.

5. The method for processing reasoning tasks as described in claim 4, characterized in that, Prior to the step of pre-filling the inference task using the target single instance to obtain intermediate results, the method further includes: Determine the priority of the request task corresponding to each of the multiple inference tasks; The multiple inference tasks are sorted according to the priority of each request task to obtain a sorting result; Based on the sorting results, the inference task with the highest priority is determined, and the step of pre-filling the inference task with the target single instance to obtain intermediate results is executed.

6. The method for processing reasoning tasks as described in claim 4, characterized in that, The step of determining the inference evaluation index generated by the target single instance during the pre-filling calculation process includes: Determine the memory usage parameters, task completion parameters, and computational intensity parameters generated by the target single instance during the pre-filling computation process; Determine the corresponding index weights for the video memory usage parameters, the task completion parameters, and the computational intensity parameters; The inference evaluation index is obtained by weighting the memory usage parameter, the task completion parameter, and the computational intensity parameter according to the weights of each index.

7. The method for processing reasoning tasks as described in claim 1, characterized in that, Following the step of distributing the multiple inference tasks to the matching target macro instances, the method further includes: Determine the running state of each of the multiple single instances in the target macro instance; Given that all the aforementioned running states are in the decoding running state, determine the single instance with the minimum instance load parameter; The running state of the single instance with the smallest instance load parameter is switched from the decoding running state to the pre-filling running state, and the step of distributing multiple inference tasks to the target single instance is executed.

8. A cluster device, characterized in that, The apparatus includes: a plurality of macro instances, a plurality of single instances, a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the processing method for the reasoning task as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the processing method for the reasoning task as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the processing method for the reasoning task as described in any one of claims 1 to 7.

Images