Request allocation method, system and equipment based on heterogeneous computing system and medium

Abstract

The invention discloses a request allocation method and system based on a heterogeneous computing system, equipment and a medium, relates to the technical field of computers, and determines reading time required for using key value cache when a plurality of heterogeneous computing power equipment accesses a memory under the condition of preferentially considering the use condition of a key value cache mechanism. Requests are reasonably distributed by considering the performance difference of the computing power information of the heterogeneous computing power equipment, the memory information corresponding to the memory extension and the characteristics of the reasoning task information. Furthermore, the reasonability of request allocation is improved according to a comparison relationship between the preset allocation request quantity and the concurrent request quantity and different strategies for judging whether allocation conditions are met or not. Therefore, the technical problem that more computing power resources are wasted due to the fact that the allocation balance of the reasoning request is poor when a separated memory or a local memory is called can be solved, and the technical effects that the reasoning task information is reasonably allocated to the heterogeneous computing power equipment, so that the allocation balance is improved, the computing power resources are saved, and the cost is reduced are achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a request allocation method, system, device and medium based on a heterogeneous computing system. Background Art

[0002] In a separate memory system, heterogeneous computing devices will access local memory or separate memory when performing inference tasks. The inference request corresponding to the inference task is allocated to the heterogeneous computing device. If the separate memory is called, the latency and bandwidth of the corresponding memory during the access process will reduce the computing efficiency of the inference task, affect the feedback speed of model inference, and reduce the user experience. If the local memory is called for model inference, it will consume more computing resources, and the overall distribution balance of inference requests will be reduced.

[0003] Therefore, how to improve the distribution balance of inference requests to save computing resources is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0004] The present application provides a request allocation method, system, device and medium based on a heterogeneous computing system, so as to at least solve the problem in the related art that the allocation balance of inference requests when calling separate memory or local memory is poor, resulting in a large waste of computing resources.

[0005] The present application provides a request allocation method based on a heterogeneous computing system, comprising: Obtain inference task information, multiple heterogeneous computing devices, and corresponding performance parameters; Determine the read time required for the key-value cache based on the inference task information and multiple performance parameters; and determine the current heterogeneous computing power device based on the multiple read times; Determine the preset number of allocation requests for the inference task based on the target read time and target performance parameters of the current heterogeneous computing power device; The request allocation result of the current heterogeneous computing power device is determined based on the preset number of allocation requests, the number of concurrent requests for inference task information and the allocation conditions.

[0006] The present application also provides a heterogeneous computing system, including a host, a plurality of heterogeneous computing devices, and a plurality of storage devices; The host is connected to multiple heterogeneous computing devices respectively; multiple heterogeneous computing devices call multiple storage devices; The host is used to execute the steps of the above-mentioned request allocation method based on a heterogeneous computing system to complete the request allocation of the reasoning task.

[0007] The present application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned request allocation methods based on a heterogeneous computing system when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned request allocation methods based on a heterogeneous computing system are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned request allocation methods based on heterogeneous computing systems when executed by a processor.

[0010] Through this application, since the reasoning task information and multiple performance parameters determine the reading time required for the key-value cache, it takes into account that the key-value cache mechanism speeds up the reasoning speed in the reasoning task, and gives priority to the use of the key-value cache mechanism. The reading time required for multiple heterogeneous computing devices to access the memory using the key-value cache is determined, so that the reading time of the key-value cache feedback is fully considered in the process of presetting the number of allocation requests. At the same time, the performance parameters of multiple heterogeneous computing devices are added, which mainly include computing power information and memory information of the separated memory, such as latency, bandwidth and other information, to determine the preset number of allocation requests. Considering the performance differences of the computing power information of the heterogeneous computing devices, the characteristics of the memory information corresponding to the memory expansion and the reasoning task information, the requests are reasonably allocated. In addition, considering the key-value cache mechanism, the request allocation result of the current heterogeneous computing device is determined according to the relationship between the preset number of allocation requests, the number of concurrent requests and the allocation conditions, and further considering the comparative relationship between the preset number of allocation requests and the number of concurrent requests, as well as different strategies for whether the allocation conditions are met, the rationality of the request allocation and the user experience are improved. Therefore, the technical problem of poor distribution balance of inference requests when calling separate memory or local memory, which leads to a lot of waste of computing resources, can be solved, so that the reasoning task information can be reasonably distributed to heterogeneous computing devices to improve the distribution balance while saving computing resources and reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A structural diagram of a heterogeneous computing system provided in an embodiment of the present application; Figure 2 A flowchart of a request allocation method based on a heterogeneous computing system provided in an embodiment of the present application; Figure 3 A schematic diagram of a request allocation architecture based on a heterogeneous computing system provided in an embodiment of the present application; Figure 4 A structural diagram of a request allocation device based on a heterogeneous computing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0016] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the request allocation method based on the heterogeneous computing system depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0017] In heterogeneous computing systems, Figure 1 A structural diagram of a heterogeneous computing system provided in an embodiment of the present application, such as Figure 1 As shown, for each heterogeneous computing device (in Figure 1Three heterogeneous computing devices are given, namely heterogeneous computing device 1', heterogeneous computing device 2' and heterogeneous computing device 3'), which can access local memory and extended memory in the separated memory network at the same time, that is, non-local memory devices; the two memory devices are collectively referred to as storage devices. When a new pre-trained language model reasoning task arrives, two processing methods are usually adopted. One is to execute in local memory. Because the memory space of local memory is limited and the key-value cache (KVCache) occupies more memory, in the case of local memory call, the heterogeneous computing device is directly used for model reasoning without using KVCache. The second is to store KVCache in separate memory, and use storage space to convert computing power performance to reason about the model when calling the separate memory.

[0018] The first processing method mentioned above has a lower latency in feeding back inference results, but it consumes a lot of computing resources, and a single heterogeneous computing device can only process a small number of inference requests. Although the second processing method can process more inference requests, it will be affected by the latency and bandwidth of the separate memory, resulting in a longer latency in returning inference results. Therefore, it is necessary to reasonably select the processing method while saving heterogeneous computing resources.

[0019] In the above heterogeneous computing system, training tasks are assigned to multiple heterogeneous computing devices, the host is connected to multiple heterogeneous computing devices respectively, and multiple heterogeneous computing devices call multiple storage devices. Heterogeneous computing power mainly refers to the computing method of using computing units with different types of instruction sets and architectures to form a system. Common types of computing units include central processing units (CPU), graphics processing units (GPU), digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), etc. Heterogeneous computing devices are different types of processors, and their performance varies when processing different types of applications.

[0020] The embodiment of the present application provides a request allocation method based on a heterogeneous computing system, and the method is described in detail in conjunction with the execution flow of the request allocation method based on a heterogeneous computing system.

[0021] Pre-training is a strategy for deep learning model training. Its core is to use large-scale data sets to preliminarily train the model so that the model can learn common feature representations. This process is similar to the basic learning stage of humans before learning new knowledge, accumulating experience through extensive reading and observation.

[0022] Pre-trained language model generally refers to designing language model training tasks based on large-scale corpus (including language training materials such as sentences and paragraphs), training large-scale neural network algorithm structure to learn and implement, and the final large-scale neural network algorithm structure and parameters are pre-trained language model. Other tasks can be used to extract features or fine-tune tasks based on this model to achieve specific task objectives. The idea of ​​pre-training is to first train a task to obtain a set of model parameters, then use this set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain models adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and can extract rich syntactic and semantic information from the text. The pre-trained language model can provide tokens and sentence-level features containing rich semantic information for use in downstream tasks, and can also be fine-tuned directly on the pre-trained model for downstream tasks, so as to obtain downstream-specific models conveniently and quickly.

[0023] The neural network algorithm structure of the pre-trained language model training can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), etc., or it can be a model built with an attention network, such as a transformer structure (transformer), a bidirectional transformer model (Bidirectional Encoder Representations from Transformers, BERT), a generative pre-trained transformer (Generative Pre-trained Transformer, GPT), a contrastive language-image pre-training (Contrastive Language-Image Pre-training, CLIP), etc., which is not limited in this application. An attention network refers to a network model that uses an attention mechanism for training. The model extracts more important feature information from the input sequence by assigning different weights to each part of the input sequence, so that the model ultimately obtains a more accurate output.

[0024] Fine-tuning refers to further training on a task-specific dataset based on the use of a pre-trained model to adjust the model parameters to better suit the target task. During the fine-tuning process, most of the layers of the pre-trained model are usually frozen, and only the newly added layers are trained or a small number of key layers are adjusted. This can not only retain the useful features learned by the pre-trained model, but also quickly adapt to the specific needs of the new task. In addition, choosing the right learning rate and training rounds is also the key to successful fine-tuning. Fine-tuning refers to small-scale training for specific task objectives (downstream tasks) and task data (downstream data) based on the pre-trained model, to achieve minor adjustments to the parameters of the pre-trained model, and finally obtain a model adapted to specific tasks and data.

[0025] Figure 2 A flowchart of a request allocation method based on a heterogeneous computing system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes: S11: Obtain inference task information, multiple heterogeneous computing devices, and corresponding performance parameters.

[0026] It is understandable that reasoning task information usually refers to the data, parameters, configuration information, etc. involved in the reasoning-related tasks. Since the pre-trained language model is mainly formed by transformer stacking, it mainly includes model information and task information. Model information is the model parameters of the pre-trained model, mainly including: How many transformer structures does the model contain? How many attention heads does a transformer have? The dimension of the key vector for each attention head; The dimension of the value vector of each attention head; The maximum generated sequence length supported by the task; The precision of the model parameters, such as half precision, single precision, etc.

[0027] The above model information can be manually input or obtained based on tools provided by the deep learning framework, such as a profiler. There is no limitation here and it can be set according to actual conditions.

[0028] Regarding task information, it mainly includes reasoning request information, such as: User-specified heterogeneous computing device types, such as H100 and MLU370 single computing devices; How many concurrent requests need to be processed at most per second? The requirement with the longest response time.

[0029] The above information depends on user needs and can be manually input.

[0030] The multiple heterogeneous computing devices may be all the heterogeneous computing devices corresponding to the entire heterogeneous computing system, or multiple heterogeneous computing devices selected from all the heterogeneous computing devices. The selection requirements here may be specified, or may be selected by one or more factors such as computing power, latency, bandwidth, etc. to select heterogeneous computing devices that meet the requirements, so as to save computing resources when requesting allocation.

[0031] The performance parameters of heterogeneous computing devices include heterogeneous computing performance parameters and memory communication performance parameters. The heterogeneous computing performance parameters cover the performance indicators of different computing units, such as peak computing information, floating-point calculation accuracy, etc. They can be obtained through the performance manual of heterogeneous computing or related public information, and entered into the module in advance for storage and retrieval when needed. They can also be obtained by recording historical heterogeneous computing execution information, that is, using the profiler that comes with the deep learning framework to obtain it during execution.

[0032] Memory communication performance parameters are used to describe the access efficiency and transmission capacity of memory resources, such as the latency and bandwidth information of separate memory. The latency and bandwidth information of each heterogeneous computing power reading the data in its corresponding separate memory (since separate memory can be used to expand the heterogeneous computing power memory in large quantities using double data rate (DDR), it is assumed that the capacity of the remotely expanded separate memory is unlimited). Separate memory may be implemented using Compute Express Link (CXL), Non-Uniform Memory Access (NUMA) and other methods. Therefore, for the extended memory of each heterogeneous computing power, its latency and bandwidth information are different and need to be collected separately. The corresponding latency and bandwidth information can be obtained using public testing tools such as the system benchmark tool (sysbench).

[0033] S12: Determine the read time required for the key-value cache based on the inference task information and multiple performance parameters; and determine the current heterogeneous computing power device based on the multiple read times.

[0034] The read time required for the key-value cache is the read time when each heterogeneous computing device accesses the separate memory using the key-value cache mechanism, that is, the feedback time of the key-value cache mechanism. The reasoning task information mainly considers the maximum key-value cache data volume corresponding to the reasoning task and the bandwidth and latency parameters corresponding to the separate memory in the performance parameters. The maximum key-value cache data volume is obtained through the maximum floating-point count, which corresponds to the floating-point computing capability of the heterogeneous computing device, reflecting the maximum number of floating-point operations that the device can complete per unit time.

[0035] Here, each heterogeneous computing device uses the read time of the key-value cache when accessing the separated memory, and the latency and bandwidth of different heterogeneous computing devices accessing the separated memory may correspond to different values. It should be noted that there are multiple storage devices in the separated memory, and different heterogeneous computing devices call each storage device to obtain multiple latency and bandwidth. Based on the two factors of latency and bandwidth, the separated memory corresponding to the least latency and bandwidth of different heterogeneous computing devices can be pre-selected to form a matching relationship, so as to facilitate the subsequent determination of the read time, that is, the read time of each heterogeneous computing device corresponds to one. It is also possible to calculate the read time of accessing all separated memories based on the latency and bandwidth generated by each heterogeneous computing device accessing each storage device, and then filter the read time required for the corresponding key-value cache of each heterogeneous computing device, and select the minimum read time as the read time corresponding to each heterogeneous computing device, that is, there is only one such matching relationship formed for the separated memory accessed by different heterogeneous computing devices. The first method of reading time corresponds to less processing time, and the second method of reading time is more accurate. The determination process of the above two reading times can be set according to actual conditions and is not limited here.

[0036] After forming the above matching relationship, the current heterogeneous computing device is determined based on multiple read times. It should be noted that the number of the current heterogeneous computing device is one, that is, the heterogeneous computing device corresponding to the minimum read time is iterated from multiple read times as the current heterogeneous computing device. This facilitates the subsequent allocation of inference requests on the current heterogeneous computing device to improve the subsequent inference speed.

[0037] It should be noted that the current heterogeneous computing power device is updated in real time based on the matching relationship. For example, there are 10 heterogeneous computing power devices, and their corresponding reading time is one each. After selecting them from the current heterogeneous computing power devices and performing subsequent steps S13 and S14, the matching relationship will be updated, and there will be 9 heterogeneous computing power devices left, and the next current heterogeneous computing power device will continue to be selected.

[0038] S13: Determine the preset allocation request quantity of the inference task based on the target reading time and target performance parameters of the current heterogeneous computing power device.

[0039] Specifically, the preset number of allocated requests is based on the number of requests that can be allocated by the current heterogeneous computing power device using KVCache. It is determined based on the read time and performance parameters corresponding to the current heterogeneous computing power, mainly considering the computing power, using the peak computing power information that characterizes the computing power information and the corresponding maximum floating-point calculation amount under the KVCache mechanism. The preset number of allocated requests for the inference task is determined based on the two parameters, the read time, and the response time.

[0040] S14: Determine the request allocation result of the current heterogeneous computing power device according to the preset allocation request number, the concurrent request number of the inference task information and the allocation condition.

[0041] Specifically, the preset number of allocated requests is the number of requests expected to be allocated to the current heterogeneous computing power device. In order to make the request allocation result accurate, in this embodiment, it is determined whether the preset number of allocated requests meets the allocation conditions. If it does, the final request allocation result is determined based on the relationship between the preset number of allocated requests and the number of concurrent requests of the inference task information. At this time, if the actual number of concurrent requests is less than the preset number of allocated requests, the actual number of concurrent requests is selected as the number of request allocations. This scenario mainly considers the remaining number of unallocated concurrent requests and the total number of concurrent requests. If it is not satisfied, it means that the latency requirement cannot be met using the KVCache mechanism, and it is necessary to determine the request allocation result of the current heterogeneous computing power device without using the KVCache mechanism, based on performance parameters, that is, computing power information.

[0042] It can be understood that, in this embodiment, corresponding to the reduction of costs, by allocating heterogeneous computing devices for concurrent requests, compared with randomly placing a few requests in heterogeneous computing devices, resulting in more heterogeneous computing devices, thereby generating more purchase costs, this embodiment adopts a reasonable allocation method to deploy more requests corresponding to reasoning tasks and allocate them to heterogeneous computing devices, so as to achieve the use of fewer heterogeneous computing devices and deploy more reasoning tasks, thereby reducing the purchase cost. On the basis of the existing heterogeneous computing devices, the deployment and allocation of this embodiment can save the power consumption of heterogeneous computing devices and save operating costs.

[0043] Through this application, since the reasoning task information and multiple performance parameters determine the reading time required for the key-value cache, it takes into account that the key-value cache mechanism speeds up the reasoning speed in the reasoning task, and gives priority to the use of the key-value cache mechanism. The reading time required for multiple heterogeneous computing devices to access the memory using the key-value cache is determined, so that the reading time of the key-value cache feedback is fully considered in the process of presetting the number of allocation requests. At the same time, the performance parameters of multiple heterogeneous computing devices are added, which mainly include computing power information and memory information of the separated memory, such as latency, bandwidth and other information, to determine the preset number of allocation requests. Considering the performance differences of the computing power information of the heterogeneous computing devices, the characteristics of the memory information corresponding to the memory expansion and the reasoning task information, the requests are reasonably allocated. In addition, considering the key-value cache mechanism, the request allocation result of the current heterogeneous computing device is determined according to the relationship between the preset number of allocation requests, the number of concurrent requests and the allocation conditions, and further considering the comparative relationship between the preset number of allocation requests and the number of concurrent requests, as well as different strategies for whether the allocation conditions are met, the rationality of the request allocation and the user experience are improved. Therefore, the technical problem of poor allocation balance of inference requests when calling separate memory or local memory, which leads to a lot of waste of computing resources, can be solved, so that the inference task information can be reasonably allocated to heterogeneous computing devices to improve the allocation balance while saving computing resources and reducing costs.

[0044] In some embodiments, the process of selecting multiple heterogeneous computing devices includes: Obtain multiple initial heterogeneous computing power devices and corresponding peak computing power information in a heterogeneous computing system; According to the relationship between the multiple peak computing power information and the preset computing power information, the target initial heterogeneous computing power device that exceeds the preset computing power information is selected from the multiple peak computing power information; The target initial heterogeneous computing power device is used as multiple heterogeneous computing power devices.

[0045] Specifically, there are multiple heterogeneous computing power devices in the heterogeneous computing system, which are used as initial heterogeneous computing power devices. Here, based on the determination of the heterogeneous computing power specified by the user, it is necessary to screen according to the peak computing power information and the preset computing power information, so as to screen out the peak computing power information corresponding to the preset computing power information from the multiple initial heterogeneous computing power devices, and use the initial heterogeneous computing power device corresponding to the peak computing power information exceeding the preset computing power information as the target initial heterogeneous computing power device, that is, the multiple heterogeneous computing power devices in step S11.

[0046] This embodiment provides a process for determining a designated heterogeneous computing power device by screening based on peak computing power information and preset computing power information, thereby improving computing power performance.

[0047] In some embodiments, the read time required for the key-value cache is determined based on the inference task information and multiple performance parameters, including: Obtain the critical key value cache data volume, performance parameter latency information, and bandwidth information required for reasoning task information; The read time required for the corresponding key-value cache when multiple heterogeneous computing devices access the separated memory is determined according to the critical key-value cache data volume, multiple delay information and multiple bandwidth information.

[0048] Specifically, the critical key-value cache data volume required by the reasoning task information is the maximum KVCache data volume required by the reasoning requirement. The performance parameters correspond to the latency and bandwidth of the separated memory.

[0049] The critical key-value cache data volume and the memory characteristic information of the extended memory are used to determine the read time required for the corresponding key-value cache when each heterogeneous computing device accesses the separated memory. Here, the read time determined by the computing power information and the memory characteristic information of the extended memory is taken into account, so that the number of requests can be reasonably allocated in the process of allocating them, improving the user experience.

[0050] In some embodiments, determining the read time required for the corresponding key-value cache when multiple heterogeneous computing devices access the separated memory according to the critical key-value cache data volume, multiple latency information, and multiple bandwidth information includes: Determine multiple critical transmission delays according to the critical key value cache data volume and multiple bandwidth information; A plurality of read times are determined according to a plurality of critical transmission delays and a corresponding plurality of delay information.

[0051] The process of determining the reading time is as follows: ; in, The read time required for each heterogeneous computing device to access the corresponding separate memory key-value cache, To access the latency information of the split memory, To access the bandwidth information of the split memory, Cache data volume for critical key values, For the Heterogeneous computing devices.

[0052] The process provided in this embodiment for determining the reading time required for the corresponding key-value cache when multiple heterogeneous computing devices access separate memories lays the foundation for the subsequent process of determining the number of KVCache allocation requests and ensures reasonable allocation.

[0053] In some embodiments, the process of determining the critical key-value cache data amount required for reasoning task information includes: Obtain model information of the pre-trained model corresponding to the reasoning task information; wherein the model information includes at least the number of transformer structures, the number of attention heads of the transformer structure, the key vector dimension of the attention head, the value vector dimension, the critical sequence length and precision quantity supported by the reasoning task; The critical key-value cache data volume is determined based on the number of transformer structures, the number of attention heads, the key vector dimension, the value vector dimension, the critical sequence length, and the number of precisions.

[0054] Specifically, Table 1 is a table of parameters involved in the request allocation method. As shown in Table 1, the performance parameters include peak computing power information of heterogeneous computing power, latency information and bandwidth information of extended memory. The reasoning task information includes model information and task information.

[0055] Table 1 Parameters involved in request allocation method

[0056] The process of determining the critical key value cache data volume is as follows: ; in, is the number of converter structures, is the number of attention heads, is the key vector dimension, is the dimension of the value vector, is the critical sequence length, is the precision number.

[0057] The process of determining the critical key value cache data volume required for the reasoning task information provided in this embodiment facilitates the subsequent calculation of the maximum read time required for the key value cache, improves the rationality of request allocation, and makes the request allocation reasonable.

[0058] In some embodiments, determining the current heterogeneous computing device according to multiple read times includes: Sort multiple read times from largest to smallest to obtain a sorted read time relationship; The minimum read time is selected from the read time relationship, and the heterogeneous computing device to which the minimum read time belongs is used as the current heterogeneous computing device.

[0059] Specifically, in this embodiment, a read time corresponds to the read time required for a heterogeneous computing device to access the separated memory key-value cache. The multiple read times are sorted from large to small to obtain the sorted read time relationship. Here, a list method or a collection method can be used for sorting. There is no limitation here. The smallest read time is selected, which is to select the heterogeneous computing device to which the smallest read time belongs as the current heterogeneous computing device. That is, , here express Heterogeneous computing devices.

[0060] This embodiment provides a process for selecting the current heterogeneous computing device to iterate the heterogeneous computing device corresponding to the minimum key-value cache data reading time, so as to increase the computing power when allocating the number of requests, so that the remaining time for calling the separated memory is sufficient.

[0061] In some embodiments, determining the preset allocation request quantity of the inference task according to the target read time and target performance parameters of the current heterogeneous computing power device includes: Obtain the first critical computing power information of the key-value cache, the peak computing power information of the target performance parameter, the critical response time of the reasoning task information, and the target reading time; Determine the remaining access time according to the target read time and the critical response time; Determine the number of requests processed per unit time according to the first critical computing power information and the peak computing power information; Determine access times corresponding to the multiple requests according to the number of requests and the remaining access time; The preset number of allocation requests is obtained by normalizing the access time and the target read time corresponding to the multiple requests.

[0062] Specifically, the first critical computing power information is the maximum floating point calculation amount when KVCache exists, and the critical response time is the longest response time requirement in Table 1. The formula for the preset number of allocation requests is as follows: ; in, Read the time for the target, is the critical response time, is the remaining access time, is the first critical computing power information, is the peak computing power information, is the number of requests processed per unit time, is the access time corresponding to multiple requests, Allocate the requested quantity for the preset, It is the current heterogeneous computing power device.

[0063] The above preset number of allocation requests is determined based on the preset number of allocation requests per unit time, i.e., 1 second. The target read time, critical response time, remaining access time, and access time corresponding to multiple requests are all set in seconds.

[0064] Regarding the determination of the first critical computing power information, the formula is as follows: ; in, is the number of converter structures, is the number of attention heads, is the key vector dimension, is the dimension of the value vector, is the critical sequence length.

[0065] The process for determining the preset number of allocated requests for inference tasks provided in this embodiment takes into account the number of requests that can be allocated per second when using key-value cache, determines the remaining access time by the difference between the key-value cache time and the critical response time, and performs subsequent normalization processing to make the entire determination process more accurate, taking into account the computing power information, memory expansion information and inference task characteristic information of heterogeneous computing power devices, so as to ensure reasonable allocation of inference tasks while meeting the maximum response delay performance requirements.

[0066] In some embodiments, the request allocation result of the current heterogeneous computing device is determined according to the preset allocation request number, the concurrent request number of the inference task information, and the allocation condition, including: When the preset number of allocation requests meets the allocation conditions, the final number of allocation requests for the current heterogeneous computing power device is determined according to the relationship between the preset number of allocation requests and the number of concurrent requests; When the preset allocation request quantity does not satisfy the allocation condition, a new preset allocation request quantity of the reasoning task is determined according to the target performance parameter.

[0067] Specifically, when the preset number of allocation requests meets the allocation conditions, the final number of allocation requests is determined based on the relationship between the preset number of allocation requests and the number of concurrent requests. The allocation condition in this application is that the number of requests is 1. If the preset number of allocation requests is greater than or equal to 1, the preset number of allocation requests and the number of concurrent requests are continued to be compared to determine the final number of allocation requests. If the preset number of allocation requests does not meet the allocation conditions, that is, the preset number of allocation requests is less than 1, it means that the use of key-value cache cannot meet the latency requirements, then the current allocation does not use key-value cache, and the new preset number of allocation requests is determined by computing power information.

[0068] This embodiment provides a comparison between the preset number of allocation requests and the allocation conditions to ensure whether the key-value cache can actually be used to meet the latency requirements, to ensure that the current heterogeneous computing power devices meet the latency requirements while using the key-value cache, and the number of requests that can be allocated without using the key-value cache, thereby improving the rationality of the allocation.

[0069] In some embodiments, determining the final number of allocation requests of the current heterogeneous computing power device according to the relationship between the preset number of allocation requests and the number of concurrent requests includes: When the preset number of allocation requests is greater than or equal to the number of concurrent requests, the final number of allocation requests for the current heterogeneous computing power device is determined according to the number of concurrent requests; When the preset number of allocation requests is less than the number of concurrent requests, the final number of allocation requests for the current heterogeneous computing power device is determined according to the preset number of allocation requests.

[0070] Specifically, considering the situation that there is an unequal relationship between the preset number of allocated requests and the number of concurrent requests, when the preset number of allocated requests is greater than or equal to the number of concurrent requests, it means that all concurrent requests can be allocated using one heterogeneous computing device, and at this time, the number of concurrent requests is used as the final number of allocated requests for the current heterogeneous computing device. When the preset number of allocated requests is less than the number of concurrent requests, the preset number of allocated requests is used to determine the final number of allocated requests.

[0071] This embodiment provides a method for determining the final number of allocated requests for the current heterogeneous computing power device when there is an unequal relationship between the preset number of allocated requests and the number of concurrent requests, thereby ensuring that the allocation result is correct.

[0072] In some embodiments, determining a new preset allocation request quantity of the inference task according to the target performance parameter includes: Obtaining the second critical computing power information; Determine a new preset allocation request quantity corresponding to the current heterogeneous computing power device according to the second critical computing power information and the peak computing power information; The final number of allocated requests for the current heterogeneous computing power device is determined based on the relationship between the new preset number of allocated requests and the number of concurrent requests.

[0073] Specifically, considering that the preset number of allocation requests is determined based on the use of key-value cache, when the latency requirement cannot be met at present and the key-value cache is not used, the new preset number of allocation requests is recalculated, and the formula is as follows: ; in, The second critical computing power information, that is, the maximum floating point calculation amount without KVCache; is the peak computing power information, Allocate the requested quantity for the new preset, It is the current heterogeneous computing power device.

[0074] The process of determining the second critical computing power information is as follows: ; in, is the number of converter structures, is the number of attention heads, is the key vector dimension, is the dimension of the value vector, is the critical sequence length.

[0075] After determining the new preset number of allocation requests, it is compared with the number of concurrent requests to determine the final number of allocation requests for the current heterogeneous computing device, which is the same as the above embodiment and will not be described in detail here.

[0076] This embodiment provides a new preset allocation request number for determining the inference task based on the target performance parameters, and realizes the allocation request number without using the key-value cache. While improving the rationality of allocation, the final allocation request number of the current heterogeneous computing power device is determined when there is an unequal relationship between the preset allocation request number and the concurrent request number, thereby ensuring the correctness of the allocation result.

[0077] In some embodiments, determining the final allocation request quantity of the current heterogeneous computing power device according to the preset allocation request quantity includes: If the preset allocation request quantity contains a decimal, the preset allocation request quantity is rounded down to obtain the rounded preset allocation request quantity; The rounded preset allocation request quantity is used as the final allocation request quantity.

[0078] Specifically, if the preset allocation request quantity contains decimals, it needs to be rounded down. Floating-point operations are usually more time-consuming than integer operations. Rounding down can convert floating-point operations into integer operations, thereby improving computing efficiency. Integers usually take up less storage space than floating-point numbers. Rounding down can reduce storage requirements and improve the overall performance of the system.

[0079] In some embodiments, after determining the request allocation result of the current heterogeneous computing power device, the method further includes: Get the final allocation request quantity of the current heterogeneous computing power device; Determine the number of remaining requests based on the number of concurrent requests and the number of final allocated requests; Update the reading time corresponding to multiple heterogeneous computing devices according to the reading time corresponding to the current heterogeneous computing device; Determine a new current heterogeneous computing device based on the updated reading time of the multiple heterogeneous computing devices, and return to the step of determining the preset number of allocated requests for the inference task based on the target reading time and target performance parameters of the current heterogeneous computing device, until the number of concurrent requests is allocated among the multiple heterogeneous computing devices.

[0080] Specifically, after determining the request allocation result of the current heterogeneous computing power device, it is necessary to allocate the next heterogeneous computing power device. It is necessary to determine the current number of remaining requests based on the number of concurrent requests and the final number of allocated requests of the current heterogeneous computing power device, continue to update the reading time relationship of multiple heterogeneous computing power devices to determine the next new current heterogeneous computing power device, and continue to return to step S13 until the number of concurrent requests is allocated.

[0081] The request allocation process provided in this embodiment is implemented in a loop until the concurrent requests of the reasoning task are allocated to multiple heterogeneous computing devices. While considering the computing performance differences, memory expansion, and reasoning task characteristics of the heterogeneous computing devices, reasonable requests are allocated to the heterogeneous computing devices, so that the above-mentioned factors are balanced, computing resources are saved, and costs are reduced.

[0082] In some other embodiments, after determining the request allocation results of the plurality of heterogeneous computing devices, the method further includes: Count the number of first requests corresponding to multiple heterogeneous computing devices; Determine the final number of remaining requests according to the number of concurrent requests and the number of multiple first requests; Determine the remaining heterogeneous computing power devices in the initial heterogeneous computing power device except the target initial heterogeneous computing power device; Sort the reading times of the remaining heterogeneous computing devices, determine the new current heterogeneous computing device, and return to the step of determining the preset number of allocated requests for the inference task based on the target reading time and target performance parameters of the current heterogeneous computing device, so as to complete the allocation of the final remaining number of requests among the remaining heterogeneous computing devices.

[0083] Specifically, after all requests of multiple heterogeneous computing devices in step S11 are allocated based on the number of concurrent requests, if there are still remaining requests, it is necessary to count the final number of remaining requests. After multiple heterogeneous computing devices are used up, continue to screen the remaining heterogeneous computing devices in the original initial heterogeneous computing devices. The screening here can be random use, or it can reduce the preset computing power information, and further screening can also be based on other factors such as other computing power information, which is not limited here. The reading time of the remaining heterogeneous computing devices will be determined and sorted for subsequent allocation, which is the same as the technical solution of step S13, and is not limited here, until the final number of remaining requests is allocated.

[0084] This embodiment takes into account that even after the multiple heterogeneous computing devices specified by the user are allocated, there are still remaining requests, so the screening range of the heterogeneous computing devices is further expanded, thereby improving the allocation efficiency compared to reallocating the multiple heterogeneous computing devices used for the specification.

[0085] Figure 3A schematic diagram of a request allocation architecture based on a heterogeneous computing system provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, it includes 4 modules, namely: 1. Heterogeneous computing system information collection module: This module will regularly collect heterogeneous computing power and separated memory information in the heterogeneous computing system. Once a neural network is issued, the information will be sent to the inference task information collection module.

[0086] 2. Reasoning task information collection module: When a reasoning task is issued, the heterogeneous computing system will collect the task information of the reasoning task, and the information will be sent to the reasoning task information collection module.

[0087] 3. Reasoning task allocation module: Based on the collected heterogeneous computing power, separated memory information, and the collected reasoning task information, this module will calculate the allocation plan for the reasoning task and send it to the reasoning task delivery module.

[0088] 4. Reasoning task dispatching module: According to the reasoning task allocation plan, the reasoning task is deployed to the heterogeneous computing system for actual task execution.

[0089] For the heterogeneous computing system information collection module, the computing power in the heterogeneous computing system and its corresponding separate memory information are collected regularly.

[0090] For the reasoning task information collection module, when a pre-trained language model reasoning task is issued, the reasoning task information needs to be collected and sent to the reasoning task allocation module for calculating the allocation plan for the reasoning task.

[0091] For the inference task allocation module, based on the collected separate memory information and large model inference task information, the module will execute the large language model inference task allocation algorithm for separate memory. While ensuring the longest response time to meet user needs, it will select the appropriate processing method for the arriving inference task and make reasonable request allocation, while trying to save heterogeneous computing resources as much as possible.

[0092] For the inference task sending module, according to the inference task allocation result of the inference task allocation module, the module will send the inference request according to the calculated allocation result and actually deploy it in the separated memory heterogeneous computing system.

[0093] The purpose of the algorithm is to select several user-specified heterogeneous computing powers in the heterogeneous computing system for the arriving inference requests, and output their request allocation results, in the form of [(heterogeneous computing power identifier, whether to use KVCache, how many concurrent requests can be made per second), (heterogeneous computing power identifier, whether to use KVCache, how many concurrent requests can be made per second), …]. For example, if the user chooses to use MLU370 to perform LLAMA2-13B inference tasks with a maximum of 10 concurrent requests per second, the final allocation result may be [(MLU370_1, using KVCache, 6 concurrent requests can be made), (MLU370_3, not using KVCache, 4 concurrent requests can be made)].

[0094] During the allocation process, if there is no available heterogeneous computing device and there are still requests that need to be allocated, it can be directly returned that the demand cannot be met. If there is an available heterogeneous computing device, it is necessary to use KVCache to calculate the number of requests that can be allocated per second. The specific calculation process has been given in the above embodiment. , then round down and add (If this calculation , then another ), and obtain The distribution result is ( , using KVCache, ).if , then directly output the allocation result. , it means that KVCache cannot meet the latency requirement. The specific calculation process of the number of new requests that can be allocated per second when KVCache is not used has been given in the above embodiment. Assign the value down and add (If this calculation , then another ), and obtain The distribution result is ( , without using KVCache, ).if , then directly output the allocation result.

[0095] The combing algorithm outputs [(heterogeneous computing power identifier, whether KVCache is used, how many concurrent requests can be made per second), (heterogeneous computing power identifier, whether KVCache is used, how many concurrent requests can be made per second), …], and returns the final result.

[0096] It should be noted that the determination of the first critical computing power information refers to the floating-point computing amount without using KVCache.

[0097] 1. Key ) and Value ( ) Vector cache: previously calculated and is cached, and only the Key and Value vectors of the new token need to be calculated.

[0098] 2. Attention Mechanism Calculation :All current Tokens perform dot product calculations.

[0099] 3. Attention output calculation: for all The weighted sum of the value vectors of the tokens.

[0100] The computational effort of a single layer ( tokens): Key and Value of the new token: ; Dot product calculation : ; Attention Weighted Sum : ; The total amount of calculation for a single head is: ; The calculation amount of each head is: ; The total computational effort of the layer (computational effort of a single token) is: .

[0101] Cumulative total computation (generated length is ): For the generated sequence length , from 1st to 2nd Tokens are gradually accumulated: ; Split the items: ; The first sum: ; The second sum: ; After the merger: ; Simplifying further: .

[0102] Regarding the determination of the second critical computing power information, that is, when the floating-point calculation amount of KVCache is not used, each time a new token is generated: 1. Key ) and Value ( ) Vector recalculation: Each token needs to recalculate the Key and Value vectors; 2. Attention Mechanism Calculation :All previously generated Calculate the dot product of tokens; 3. Attention output calculation ( ): for all The weighted sum of the value vectors of the tokens.

[0103] The computational effort of a single layer ( tokens): Recalculate Key and Value: ; in, It is also the length of the current sequence. The Key and Value need to be recalculated for each token.

[0104] Dot product calculation : ; Attention Weighted Sum : ; The total amount of calculation for a single head is: ; The calculation amount of each head is: ; The total computational effort of the layer (computational effort of a single token) is: .

[0105] Cumulative total computation (generated length is ): For the generated sequence length , from 1st to 2nd Tokens are gradually accumulated: ; Will From 1 to : ; Using the formula : .

[0106] About KVCache calculation: 1. KVCache storage content: For a multi-layer Transformer architecture: 1)Key ) Vector: used to store the key values ​​of all previously generated tokens; 2) Value ) Vector: used to store the values ​​of all previously generated tokens.

[0107] Single layer storage capacity: The sizes of the Key and Value vectors cached by each attention head are: ; in, is the length of the generated sequence, , The vector dimensions of Key and Value.

[0108] The total size of each attention head cache: ; If you have attention heads, the single-layer cache size is: ; 2. Total KVCache storage: For one Layer Transformer, the total storage capacity of KVCache is: .

[0109] Furthermore, the present application also provides a heterogeneous computing system, such as Figure 1 As shown, it includes a host, multiple heterogeneous computing devices and multiple storage devices; The host is connected to multiple heterogeneous computing devices respectively; multiple heterogeneous computing devices call multiple storage devices; The host is used to execute the steps of the above-mentioned request allocation method based on a heterogeneous computing system to complete the request allocation of the reasoning task.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0111] The embodiment of the present application also provides a request allocation device based on a heterogeneous computing system. Figure 4 The structure diagram of a request allocation device based on a heterogeneous computing system provided in an embodiment of the present application. Figure 4 As shown, the request allocation device based on the heterogeneous computing system includes: Acquisition module 11: Acquisition of reasoning task information, multiple heterogeneous computing devices and corresponding performance parameters; The first determination module 12: determines the read time required for the key-value cache based on the inference task information and multiple performance parameters; and determines the current heterogeneous computing power device according to the multiple read times; The second determination module 13: determines the preset allocation request quantity of the inference task according to the target reading time and target performance parameters of the current heterogeneous computing power device; The third determination module 14 determines the request allocation result of the current heterogeneous computing power device according to the preset allocation request number, the concurrent request number of the inference task information and the allocation condition.

[0112] For the description of the features in the embodiment corresponding to the request allocation device based on a heterogeneous computing system, reference may be made to the relevant description of the embodiment corresponding to the request allocation method based on a heterogeneous computing system, which will not be described in detail here.

[0113] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned request allocation method embodiments based on a heterogeneous computing system.

[0114] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned request allocation method embodiments based on a heterogeneous computing system when running.

[0115] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0116] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned request allocation method embodiments based on a heterogeneous computing system are implemented.

[0117] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned request allocation method embodiments based on a heterogeneous computing system.

[0118] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] The above is a detailed introduction to a request allocation method, system, device and medium based on a heterogeneous computing system provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A request allocation method based on a heterogeneous computing system, characterized in that: include: Obtain inference task information, multiple heterogeneous computing devices, and corresponding performance parameters; Determine the read time required for the key-value cache based on the inference task information and multiple performance parameters; And determine the current heterogeneous computing power device based on multiple reading times; Determine a preset number of allocation requests for the inference task according to the target read time and target performance parameters of the current heterogeneous computing power device; The request allocation result of the current heterogeneous computing power device is determined according to the preset allocation request number, the concurrent request number of the inference task information and the allocation condition.

2. The request allocation method based on heterogeneous computing system according to claim 1, characterized in that: The read time required for the key-value cache is determined based on the inference task information and multiple performance parameters, including: Obtaining the critical key value cache data volume required for the reasoning task information, the latency information and the bandwidth information of the performance parameters; The read time required for the corresponding key-value cache when multiple heterogeneous computing devices access the separate memory is determined according to the critical key-value cache data volume, multiple delay information and multiple bandwidth information.

3. The request allocation method based on heterogeneous computing system according to claim 2, characterized in that: Determining the read time required for the corresponding key-value cache when the plurality of heterogeneous computing devices access the separated memory according to the critical key-value cache data volume, the plurality of delay information and the plurality of bandwidth information, including: Determine multiple critical transmission delays according to the critical key value cache data amount and the multiple bandwidth information; A plurality of the reading times are determined according to the plurality of the critical transmission delays and the corresponding plurality of the delay information.

4. The request allocation method based on a heterogeneous computing system according to any one of claims 1 to 3, characterized in that: Determine the current heterogeneous computing power device based on multiple read times, including: Sorting the plurality of read times from largest to smallest to obtain a sorted read time relationship; The minimum read time is selected from the read time relationship, and the heterogeneous computing power device to which the minimum read time belongs is used as the current heterogeneous computing power device.

5. The request allocation method based on heterogeneous computing system according to claim 4, characterized in that: Determining the preset allocation request quantity of the inference task according to the target reading time and target performance parameters of the current heterogeneous computing power device includes: Obtaining first critical computing power information of the key-value cache, peak computing power information of the target performance parameter, critical response time of the reasoning task information, and the target reading time; Determine the remaining access time according to the target read time and the critical response time; Determine the number of requests processed per unit time according to the first critical computing power information and the peak computing power information; Determining access times corresponding to the multiple requests according to the number of requests and the remaining access time; The preset allocation request quantity is obtained by normalizing the access times corresponding to the multiple requests and the target read time.

6. The request allocation method based on heterogeneous computing system according to claim 5, characterized in that: Determining the request allocation result of the current heterogeneous computing power device according to the preset allocation request number, the concurrent request number of the inference task information and the allocation condition, including: In the case where the preset number of allocation requests meets the allocation condition, determining the final number of allocation requests of the current heterogeneous computing power device according to the relationship between the preset number of allocation requests and the number of concurrent requests; When the preset number of allocation requests does not satisfy the allocation condition, a new preset number of allocation requests for the reasoning task is determined according to the target performance parameter.

7. The request allocation method based on heterogeneous computing system according to claim 6, characterized in that: Determining the final number of allocation requests of the current heterogeneous computing power device according to the relationship between the preset number of allocation requests and the number of concurrent requests includes: When the preset number of allocation requests is greater than or equal to the number of concurrent requests, determining the final number of allocation requests of the current heterogeneous computing power device according to the number of concurrent requests; When the preset number of allocation requests is less than the number of concurrent requests, the final number of allocation requests of the current heterogeneous computing power device is determined according to the preset number of allocation requests.

8. The request allocation method based on heterogeneous computing system according to claim 6, characterized in that: Determining a new preset allocation request quantity of the reasoning task according to the target performance parameter includes: Obtaining the second critical computing power information; Determine a new preset allocation request quantity corresponding to the current heterogeneous computing power device according to the second critical computing power information and the peak computing power information; The final allocation request quantity of the current heterogeneous computing power device is determined based on the relationship between the new preset allocation request quantity and the concurrent request quantity.

9. The request allocation method based on heterogeneous computing system according to claim 7, characterized in that: Determining the final allocation request quantity of the current heterogeneous computing power device according to the preset allocation request quantity includes: If the preset allocation request quantity contains a decimal, the preset allocation request quantity is rounded down to obtain a rounded preset allocation request quantity; The rounded preset allocation request quantity is used as the final allocation request quantity.

10. The request allocation method based on heterogeneous computing system according to claim 1, characterized in that: The process of selecting the plurality of heterogeneous computing devices includes: Obtain multiple initial heterogeneous computing power devices and corresponding peak computing power information in a heterogeneous computing system; According to the relationship between the plurality of peak computing power information and the preset computing power information, selecting the target initial heterogeneous computing power device that exceeds the preset computing power information from the plurality of peak computing power information; The target initial heterogeneous computing power device is used as a plurality of the heterogeneous computing power devices.

11. The request allocation method based on heterogeneous computing system according to claim 1, characterized in that: After determining the request allocation result of the current heterogeneous computing power device, the method further includes: Obtain the final allocation request quantity of the current heterogeneous computing power device; Determine the number of remaining requests according to the number of concurrent requests and the number of final allocated requests; Update the reading times corresponding to the plurality of heterogeneous computing devices according to the reading time corresponding to the current heterogeneous computing device; Determine a new current heterogeneous computing device based on the updated reading time of the multiple heterogeneous computing devices, and return to the step of determining the preset number of allocated requests for the inference task based on the target reading time and target performance parameters of the current heterogeneous computing device until the number of concurrent requests is allocated among the multiple heterogeneous computing devices.

12. A heterogeneous computing system, characterized in that: Includes a host, multiple heterogeneous computing devices, and multiple storage devices; The host is respectively connected to a plurality of heterogeneous computing devices; the plurality of heterogeneous computing devices call a plurality of storage devices; The host is used to execute the steps of the request allocation method based on a heterogeneous computing system as described in any one of claims 1 to 11 above to complete the request allocation of the reasoning task.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the request allocation method based on a heterogeneous computing system as described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the request allocation method based on a heterogeneous computing system as claimed in any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the request allocation method based on a heterogeneous computing system as claimed in any one of claims 1 to 11 are implemented.

Images