A task allocation method, apparatus, device and medium

Abstract

This application provides a task allocation method, apparatus, device, and medium to improve task execution efficiency. In this embodiment, the total number of tasks to be processed is accurately calculated based on the batch size and depth of each input data. The total tasks are divided into multiple task sets for processing batches according to the maximum concurrent task count supported by each computing core, and these sets are allocated to different computing cores for execution. Each task set contains a set where the number of characters corresponding to the tasks is equal to the maximum number of tasks that can be handled, thereby maximizing the utilization efficiency of each computing core and ensuring that it can execute tasks at full capacity.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a task allocation method, apparatus, device and medium. Background Technology

[0002] With the rapid development of artificial intelligence, deep learning, and high-performance computing, general-purpose graphics processing units (GPGPUs) have become the core computing platform for training and inference tasks. In the GPGPU architecture, a large number of lightweight parallel computing cores work together through a high-bandwidth memory (HBM) stack to achieve efficient processing of large-scale tensor data.

[0003] In neural network models, there exists a widely used class of element-wise or vector-level operations, collectively known as vector operators. These operators typically operate on a specific dimension of a tensor, performing operations such as normalization, activation functions, or element-wise addition. Although the computational intensity per operation is relatively low, their frequent occurrence in the model makes their overall performance overhead considerable.

[0004] In related technologies, GPGPU scheduling strategies for Vector-type operators typically employ a "task partitioning by batch" approach. This involves treating all tasks within an input batch as a whole and distributing them across several computing cores for parallel processing. The number of tasks in each batch is determined by the depth (e.g., the number of characters) of the input data in that batch. However, this method suffers from a significant resource utilization bottleneck: when the input batch is small or the depth of each input data is low, the total number of tasks is often far below the maximum concurrent task capacity supported by the GPGPU hardware. This results in a large number of computing cores being idle or underloaded, failing to fully leverage the advantages of parallel computing.

[0005] For example, suppose there are 100 batches of input data, each batch has a depth of 2 (meaning each input data contains only 2 tasks that can be processed in parallel), and each computing core can handle a maximum of 16 tasks simultaneously. In this scenario, using existing scheduling techniques, each computing core can only be allocated 2 tasks at a time, even if it still has 14 tasks available, which cannot be effectively utilized. The next batch of 2 tasks is processed sequentially only after the current batch is completed. This serial processing mode results in severe resource idleness and low execution efficiency. This approach leads to most computing cores being idle or underloaded, making it difficult to fully utilize all available computing resources. Summary of the Invention

[0006] This application provides a task allocation method, apparatus, device, and medium to improve task execution efficiency.

[0007] On one hand, embodiments of this application provide a task allocation method, the method comprising:

[0008] The total number of tasks to be executed is determined based on the batch size of the input data and the depth of each input data; wherein, the batch size is the total number of questions input into the large language model, and the depth is the number of characters contained in the corresponding question;

[0009] Based on the maximum number of tasks each computing core can handle, the total number of tasks is divided to obtain task sets for different processing batches of each computing core; wherein, in multiple task sets corresponding to any computing core, there exists a task set containing the same number of characters corresponding to the tasks as the maximum number of tasks that computing core can handle, and the number of characters corresponding to the tasks contained in any batch of task sets corresponding to any computing core does not exceed the maximum number of tasks that computing core can handle.

[0010] Different computing cores are controlled to process the characters in the task set of their respective processing batches.

[0011] On the one hand, embodiments of this application also provide a task allocation device, the device comprising:

[0012] The determination module is used to determine the total number of tasks to be executed based on the batch number of input data and the depth of each input data; wherein, the batch number is the total number of questions input to the large language model, and the depth number is the number of characters contained in the corresponding question;

[0013] The processing module is used to divide the total number of tasks according to the maximum number of tasks that each computing core can handle, to obtain task sets for different processing batches of each computing core; wherein, in multiple task sets corresponding to any computing core, there is a task set containing the number of characters corresponding to the tasks that is the same as the maximum number of tasks that the computing core can handle, and the number of characters corresponding to the tasks contained in any batch of task sets corresponding to any computing core does not exceed the maximum number of tasks that the computing core can handle; and controls different computing cores to process the characters in the task sets of their respective processing batches.

[0014] Optionally, the processing module is specifically configured to traverse each computing core sequentially according to a preset computing core order, determine the task set of the current processing batch for each computing core; for the currently traversed computing core, determine whether the number of unassigned tasks in the total number of tasks is greater than or equal to the maximum number of tasks that the computing core can handle; if so, assign the maximum number of tasks to the task set of the current processing batch for the computing core; if not, assign all remaining unassigned tasks to the task set of the current processing batch for the computing core; after completing the task assignment for the last computing core, if there are still unassigned tasks, return to the first computing core, start a new processing batch, and continue to execute task partitioning until all tasks are assigned.

[0015] Optionally, the processing module is specifically configured to determine the maximum number of tasks that each computing core can carry by means of the following method: for each computing core, the number of thread bundles contained in the computing core is determined as the maximum number of tasks that computing core can carry.

[0016] Optionally, the processing module is specifically configured to, for each processing batch and for each computing core, allocate one character from a set of multiple characters in the task set of the processing batch corresponding to the computing core to each thread bundle contained in the computing core, and have each thread bundle complete the processing of the allocated character.

[0017] Optionally, the processing module is further configured to monitor the task completion status of each computing core during the parallel execution of tasks by multiple computing cores; if there is a first computing core that has completed a certain batch of tasks and a second computing core that has not completed the batch of tasks, control the first computing core to pause execution for a preset period and delay the start of the first computing core to execute the tasks contained in the next batch of tasks.

[0018] Optionally, the processing module is specifically configured to send a sleep instruction to the first computing core, wherein the sleep instruction carries a preset period for pausing execution.

[0019] On the one hand, embodiments of this application also provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, wherein the artificial intelligence chip executes the computer program to implement the steps of the above-described task allocation method.

[0020] On the one hand, embodiments of this application also provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described task allocation method.

[0021] On the one hand, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described task allocation method.

[0022] In this embodiment, the total number of tasks to be processed is accurately calculated based on the number of batches of input data and the depth of each input data. Based on the maximum number of concurrent tasks that each computing core can support, the total tasks are divided into multiple task sets for processing batches and assigned to different computing cores for execution. Each task set contains a set of tasks whose number of characters is the same as the maximum number of tasks it can handle. This allows each computing core to be fully loaded with tasks, thereby improving resource utilization efficiency. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a structural diagram of an artificial intelligence chip applicable to the embodiments of this application;

[0025] Figure 2 This is a schematic diagram illustrating a task allocation method provided in an embodiment of this application.

[0026] Figure 3 A detailed structural schematic diagram of an artificial intelligence chip provided in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the structure of a task allocation device provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] refer to Figure 1This is a structural diagram of an artificial intelligence chip applicable to an embodiment of this application. The artificial intelligence chip 100 includes at least: video memory 101 and multiple computing units 102, wherein the computing units 102 may be computing cores. The video memory 101 may be high-bandwidth memory (HBM) or other types of memory.

[0031] In this embodiment, the computing core comprises multiple thread bundles. The central processing unit determines the total number of tasks to be executed based on the batch size of the input data and the depth of each input data. Then, according to the maximum number of tasks each computing core can handle, the total number of tasks is divided into several task sets, resulting in different processing batches for each computing core. Different computing cores are then controlled to process the characters within their respective processing batch task sets. The computing core can be a computing unit 102.

[0032] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0033] Artificial intelligence chips 100 can be: Central Processing Unit (CPU), Graphics Processing Unit (GPU), GPGPU, Domain Specific Architecture (DSA), etc.

[0034] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip and details a task allocation method. (See attached image.) Figure 2 This method is performed by a computer device, which includes... Figure 1 The artificial intelligence chip shown is an example of this method, which can be applied to various scenarios, such as image processing, speech processing, and text processing. The method includes the following steps:

[0035] Step 201: Determine the total number of tasks to be executed based on the batch number of input data and the depth of each input data; wherein, the batch number is the total number of questions input to the large language model, and the depth number is the number of characters contained in the corresponding question.

[0036] In this embodiment, the computer device can determine the total number of tasks to be executed based on the batch size of the input data and the depth of each input data. Wherein: the batch size refers to the total number of questions or requests simultaneously input into the large language model; the depth refers to the number of characters contained in each input question; and the process of processing a single character can be referred to as executing a task. For example, the total number of tasks can be determined using the following formula:

[0037]

[0038] in, Where N is the total number of tasks, N is the number of batches, and n is the input data number. This represents the depth number corresponding to the nth input data.

[0039] For example, in a dynamic batch processing scenario, if the current batch contains 3 user requests with corresponding questions containing 128, 256, and 192 characters respectively, then the total number of tasks is 128 + 256 + 192, a total of 576 tasks. Each character computation kernel can be regarded as an independently scheduled task, used to perform Vector-type operator operations such as normalization, activation functions, and residual joins.

[0040] By using a statistical method that determines the total number of tasks based on the number of batches and the depth of the input data in each batch, this solution can more accurately reflect the actual computing load and avoid resource waste or uneven load caused by traditional fixed-granularity scheduling.

[0041] Step 202: Divide the total number of tasks according to the maximum number of tasks that each computing core can handle, to obtain task sets for different processing batches of each computing core; wherein, in multiple task sets corresponding to any computing core, there is a task set in which the number of characters corresponding to the tasks is the same as the maximum number of tasks that the computing core can handle, and the number of characters corresponding to the tasks in any batch of task sets corresponding to any computing core does not exceed the maximum number of tasks that the computing core can handle.

[0042] To improve resource utilization, in this embodiment, the maximum number of tasks each computing core can handle is pre-stored. For example, the maximum number of tasks each computing core can handle can be the same or different. In this embodiment, the total tasks can be divided according to the maximum task handling capacity of each computing core, generating task sets for each computing core in different processing batches. The division process must satisfy the following conditions: the total number of characters corresponding to the tasks in the task set allocated to each computing core in any processing batch must not exceed the maximum number of tasks that computing core can handle; and among all task sets generated for a certain computing core, at least one batch of task sets has a total number of characters exactly equal to the maximum handling capacity of that computing core, thus achieving efficient resource utilization. All tasks are completely allocated without omissions or duplications, ensuring the completeness and uniqueness of task division.

[0043] Step 203: Control different computing cores to process the characters in the task set of the corresponding processing batch.

[0044] In this embodiment, the computer device can coordinate and schedule various computing cores, enabling each core to independently and in parallel process the characters in its assigned batch task set according to the task partitioning results. For example, the computer device can distribute the partitioned task sets to the corresponding computing cores through a unified control mechanism, ensuring that each core reads and processes the characters contained in the current batch task set sequentially or in parallel. Processing operations may include, but are not limited to, character parsing, encoding conversion, pattern matching, encryption / decryption, or feature extraction, with the specific operations depending on the specific application scenario.

[0045] In this embodiment, input data is typically organized in the form of a multidimensional tensor. To facilitate the description of data traversal and computation scheduling, the concept of "axis" is introduced to divide the data structure. The N-axis represents the quantity dimension of the input data, i.e., the batch size, such as a batch containing N questions; while the H-axis represents the depth dimension of each input data point, which can be understood as the characters contained in a single question. In traditional parallel computing, tasks are usually partitioned and computational cores allocated along a single axis (e.g., only traversing the N-axis or H-axis). However, when the size of a certain axis is too small (e.g., N is small, i.e., the batch is small), if independent scheduling is still performed along that axis, the number of parallelizable tasks will be insufficient, resulting in a large number of computational cores being idle and low utilization of computing resources. This embodiment proposes an axis merging traversal mechanism: logically merging the N-axis and H-axis to form a joint traversal dimension (e.g., obtaining all tasks obtained by traversing the N-axis and H-axis), and then performing task partitioning and computational core scheduling on this merged dimension. In this way, even when N or H is small on its own, their product N×H can still achieve a high parallel granularity, thereby effectively improving the utilization of the computing kernel and avoiding resource waste caused by insufficient scale of a single dimension.

[0046] In this embodiment, the total number of tasks to be processed is accurately calculated based on the batch size and depth of each input data. According to the maximum concurrent task count supported by each computing core, the total tasks are divided into multiple task sets for processing batches and assigned to different computing cores for execution. Each task set contains a set where the number of characters corresponding to the tasks matches the maximum number of tasks that can be handled. This maximizes the efficiency of each computing core, allowing it to execute tasks at full capacity and improving resource utilization. This approach also reduces waste of computing cores and thread resources within each core.

[0047] To improve resource utilization, based on the above embodiments, in this embodiment, the step of dividing the total number of tasks according to the maximum number of tasks each computing core can handle, to obtain a task set for different processing batches of each computing core, includes:

[0048] The system iterates through each computing core in the preset order to determine the task set for the current batch of processing for each computing core.

[0049] For the currently traversed computing core, determine whether the number of unassigned tasks in the total number of tasks is greater than or equal to the maximum number of tasks that the computing core can handle.

[0050] If so, then the maximum number of tasks to be carried will be assigned to the task set of the current processing batch of the computing core;

[0051] If not, then all remaining unassigned tasks will be assigned to the task set of the current processing batch of this computing core;

[0052] After completing the task allocation for the last computing core, if there are still unallocated tasks, return to the first computing core, start a new processing batch, and continue to execute task allocation until all tasks are allocated.

[0053] To achieve efficient and orderly task allocation across multiple computing cores, embodiments of this application can employ a round-robin approach, polling and traversing the computing cores in a preset order, to dynamically batch task partitioning. In one possible implementation, the preset computing core order can be determined based on the data access latency or communication path length between each computing core and video memory. For example, computing cores closer to video memory and with lower data read / write latency are prioritized at the beginning of the scheduling sequence. This sorting strategy helps to prioritize the use of computing cores with higher memory access efficiency during task allocation, thereby reducing overall data transfer overhead and improving bandwidth utilization and task processing performance.

[0054] For example, when partitioning tasks, each computing core can be traversed sequentially according to a pre-defined order, determining the task set that the currently traversed computing core needs to undertake in the current processing batch. For the computing core currently being traversed, it can first be determined whether the number of unassigned tasks in the total number of tasks is greater than or equal to the maximum number of tasks that the computing core can handle. The maximum number of tasks that the computing core can handle in a single batch is the upper limit of tasks that the computing core can process. If the number of unassigned tasks in the total number of tasks is greater than or equal to the maximum number of tasks that the computing core can handle, then tasks equal to the maximum number of tasks that the computing core can handle are taken from the remaining unassigned tasks and assigned to the current processing batch task set of the current computing core, ensuring that the computing core runs at full capacity and maximizing resource utilization. If the number of unassigned tasks in the total number of tasks is less than the maximum number of tasks that the computing core can handle, it means that the number of remaining tasks is insufficient to fill the capacity of the computing core. In this case, all remaining unassigned tasks can be assigned to the current computing core to form its current batch task set, avoiding task omissions.

[0055] After the task allocation to the last computing core is completed, if there are still unallocated tasks, the process can automatically revert to the first computing core, start a new processing batch (i.e., enter the next round of traversal), and repeat the above allocation process.

[0056] For example, during task allocation, the system follows a priority scheduling strategy: First, it prioritizes the allocation of tasks corresponding to input data with earlier timestamps to ensure that data arriving earlier can be processed first, satisfying the timeliness and sequential consistency requirements of processing; Second, within the same input data, tasks are allocated sequentially from front to back according to the order of appearance of characters in the language description, that is, tasks for characters in earlier positions are prioritized to ensure the logical coherence and semantic integrity of processing within a single data set.

[0057] The method provided in this application distributes tasks evenly through a round-robin approach, avoiding the phenomenon of some computing cores being overloaded while others are idle, thus improving the overall load balancing level. It also prioritizes making each computing core run at full capacity as much as possible, reducing the idle computing units caused by task fragmentation. This method does not rely on the integer division relationship between the total number of tasks and the number of computing cores, and is applicable to task sets and computing core configurations of any size. It can reduce the time of mutual dependence and synchronization between thread bundles within each computing core, making it more efficient in utilizing computing power.

[0058] To improve resource utilization, based on the above embodiments, in this embodiment, the maximum number of tasks that each computing core can handle is determined in the following way:

[0059] For each computing core, the number of thread bundles contained in that computing core is determined as the maximum number of tasks that computing core can handle.

[0060] In this embodiment, the number of thread bundles contained within each computing core can be determined as the maximum number of tasks that the computing core can handle. Here, a thread bundle refers to a group of threads (typically 32 threads) that can be executed synchronously by a single-instruction multithreaded unit in a parallel computing architecture. Since each thread bundle can be independently scheduled and process a task, the number of tasks that a computing core can execute concurrently is directly related to the number of thread bundles it supports.

[0061] In this embodiment of the application, the number of thread bundles of the computing core is determined as the maximum number of tasks that the corresponding computing core can carry, which means that the computing core can process a maximum of tasks equal to its total number of thread bundles in parallel in a single batch.

[0062] To improve resource utilization, based on the above embodiments, in this embodiment, controlling different computing cores to process characters within the task set of the corresponding processing batch includes:

[0063] For each processing batch, for each computing core, one character from a set of characters in the task set of the processing batch corresponding to that computing core is assigned to each thread bundle contained in that computing core, and each thread bundle completes the processing of the assigned character.

[0064] Within each processing batch, for each computing core, the computer device assigns a single character from multiple characters in the current batch task set of that computing core to each thread bundle within that core, implementing a fine-grained task mapping mechanism where one thread bundle processes one character. Each thread bundle independently performs parsing, transformation, calculation, or feature extraction operations on the character within the execution context of its respective computing core. In one possible implementation, the vector corresponding to the character can be processed.

[0065] In practical applications, since each thread bundle is the basic scheduling unit in a parallel computing architecture and has the ability to execute 32 or more threads synchronously, the computing resources of the entire thread bundle can be focused on the processing task of a single character, and its internal multi-threaded parallelism can be used to accelerate the complex calculations related to that character.

[0066] For example, when processing vector data corresponding to a character, if the vector is structurally arranged from left to right (where left and right refer to the logical order of elements within the vector), this direction is defined as the W-axis. Based on this, the same thread bundle can perform cyclical serial processing along the W-axis: that is, the thread bundle first processes the first 32 components of the vector, then continues processing the next 32 components, proceeding sequentially until the entire vector is calculated. The core idea of ​​this processing method is: for the actual algorithmic logic of the operator, select a suitable computation axis (such as the W-axis) that can be sequentially unfolded within a single thread bundle, and let the thread bundle execute continuously along this axis in a segmented, cyclical manner. Since all threads within the same thread bundle share the control flow and synchronization context, the method provided in this application avoids the coordination and synchronization requirements caused by data segmentation between multiple thread bundles by concentrating the entire vector within a single thread bundle; and, by utilizing the natural synchronization characteristics of threads within a thread bundle, efficient state transfer and intermediate result sharing are achieved on the W-axis, supporting algorithmic operations that require sequential dependencies.

[0067] The embodiments of this application are general optimization schemes for Vector-type operators under the GPGPU architecture, including the training and inference domains, including but not limited to root mean square normalization (RmsNorm), root mean square normalization after addition (AddRmsNorm), and other Vector-type operators. The tasks executed by the thread bundle are usually Vector-type operators.

[0068] To improve resource utilization, based on the above embodiments, the method in this application embodiment further includes:

[0069] During the parallel execution of tasks by multiple computing cores, the task completion status of each computing core is monitored;

[0070] If there is a first computing core that has completed a certain batch of tasks, and there is a second computing core that has not completed the batch of tasks, control the first computing core to pause execution for a preset period, and delay the start of the first computing core to execute the tasks contained in the next batch of tasks.

[0071] During the parallel execution of tasks by multiple computing cores, computer devices can dynamically monitor the task completion status of each computing core to regulate the overall processing pace. To maximize hardware resource utilization, the scheduling strategy prioritizes ensuring that all computing cores and their internal thread bundles are used as fully as possible. That is, tasks are rationally allocated according to the number of thread bundles of each computing core, so that all computing units are running at high load in each round of processing, giving full play to the throughput potential of the parallel architecture.

[0072] When all computing cores are running and the task load reaches saturation, although computing resources are fully utilized, due to physical differences in the chip's internal interconnect topology, some computing cores farther from the video memory face higher communication latency and more congested data paths when accessing data. These remote computing cores may experience accumulated data loading delays in multiple rounds of continuous processing due to continuous competition for insufficient memory bandwidth, leading to a long-tail effect. In other words, overall performance is determined by the slowest few computing cores, severely impacting task execution efficiency. To alleviate this problem, this application introduces an active throttling and bandwidth balancing mechanism. If there is a first computing core that has completed the current batch of tasks, and a second computing core that has not completed the same batch, the first computing core that has completed the current batch and is preparing to enter the next batch can be paused for a preset period. In practical scenarios, the first computing core is usually the one closer to the video memory and with faster data access speed, thus delaying the execution of tasks in the next batch of tasks by that first computing core.

[0073] The method provided in this application can alleviate memory access contention. Computing cores located close to each other typically have lower access latency and higher bandwidth preemption capabilities. If they continue to run at high speed, remote computing cores will frequently encounter link congestion during multiple rounds of data loading, resulting in cumulative latency. The method provided in this application can reduce cumulative latency. By giving up critical transmission windows for remote computing cores, remote computing cores can complete the download and acquisition operations corresponding to the task more efficiently, thereby narrowing the execution progress gap between computing cores in different locations.

[0074] Figure 3 This is a detailed structural diagram of an artificial intelligence chip provided in an embodiment of this application. Figure 3 As can be seen, if an artificial intelligence chip has 16 computing cores (the squares in the diagram), and several computing cores (such as...) Figure 3The four compute cores are linked together through a node and interconnected via links, all connected to the HBM memory stack. Figure 3 As shown, on the left ( Figure 3 The calculation kernel shown on the left and right sides is compared to the right side (shown on the left and right). Figure 3 As shown in the diagram (left and right), the link between the computing cores and the video memory is shorter. Based on this, the computing core on the left usually executes a set of tasks faster than the computing core on the right.

[0075] right Figure 3 In this application, the method provided in the embodiments can effectively improve the download (load) bandwidth of computing cores located further away from video memory, such as... Figure 3 right side ( Figure 3 As shown on the left and right, all computing cores will complete tasks faster, improving overall performance.

[0076] To improve resource utilization, based on the above embodiments, in this embodiment, controlling the first computing core to pause execution for a preset period includes:

[0077] A sleep command is sent to the first computing core, wherein the sleep command carries a preset period for pausing execution.

[0078] In this embodiment, a sleep command can be sent to the first computing core. This sleep command carries a specific pause duration parameter, indicating a preset period for which the computing core will pause execution. This preset period can be dynamically configured according to a system scheduling strategy, such as adaptively adjusting based on current memory access latency, data loading progress of remote computing cores, or on-chip network congestion. For example, a sleep command can be sent to the first computing core after receiving an instruction from it indicating that it has completed executing tasks in the current batch of tasks.

[0079] By precisely controlling the pause time at the instruction level, it is possible to achieve fine-grained management of the rhythm of computing resource usage while ensuring high performance. For example, when the first computing core receives the sleep instruction, it will enter a low-power waiting state and will not initiate new memory accesses or computing operations within a specified period, thereby actively releasing shared resources, such as memory controller bandwidth, on-chip interconnect links, and bus channels, to make room for data transmission to other computing cores that are far from the video memory and have slower memory access speeds.

[0080] Most existing performance optimization techniques are highly dependent on the specific characteristics of operators. For different types of Vector-like operators, customized optimization strategies need to be designed based on their algorithm logic and data access patterns. Common optimization methods include: using double-buffered switching (Ping-Pong) buffering mechanisms of different granularities to hide memory latency; using shared memory, main buffer, or registers to cache intermediate results and reduce repeated memory accesses; enabling vector loading / storage instructions or multi-issue techniques to improve memory throughput and reduce instruction overhead; and performing loop unrolling, block partitioning, or vectorization processing for specific data types or tensor shapes. However, these optimization methods generally suffer from several key problems: lack of generality: each operator often requires independent analysis and tuning, making it difficult to form a unified optimization framework; high migration costs: when the operator structure, input dimension, or data precision changes, the original optimization strategy may become ineffective and needs to be redesigned; resource contention and conflicts: resource contention is prone to occur when multiple optimization techniques coexist. For example, the capacity of Shared Memory is limited, and if it is already occupied by other modules, it cannot support new caching requirements; similarly, excessive register pressure may lead to a decrease in thread concurrency. To solve the above problems, the general performance optimization method for Vector-type operators proposed in this application constructs a unified optimization framework that is loosely coupled with specific operators, independent of input shape and data type, and compatible with mainstream underlying optimization mechanisms. This solution is a general optimization solution with low correlation to the algorithm and does not hinder the use of resources such as Ping-Pong, Shared Memory, and Main Buffer. It is easy to implement, easy to migrate, and has stable performance benefits.

[0081] Based on the same technical concept, this application provides a schematic diagram of the structure of a task allocation device, such as... Figure 4 As shown, the task allocation device 400 includes:

[0082] The determining module 401 is used to determine the total number of tasks to be executed based on the batch number of input data and the depth of each input data; wherein, the batch number is the total number of questions input to the large language model, and the depth number is the number of characters contained in the corresponding question;

[0083] The processing module 402 is used to divide the total number of tasks according to the maximum number of tasks that each computing core can handle, to obtain task sets for different processing batches of each computing core; wherein, in multiple task sets corresponding to any computing core, there is a task set containing the number of characters corresponding to the tasks that is the same as the maximum number of tasks that the computing core can handle, and the number of characters corresponding to the tasks contained in any batch of task sets corresponding to any computing core does not exceed the maximum number of tasks that the computing core can handle; and controls different computing cores to process the characters in the task sets of their respective processing batches.

[0084] Optionally, the processing module 402 is specifically configured to traverse each computing core sequentially according to a preset computing core order, determine the task set of the current processing batch for each computing core; for the currently traversed computing core, determine whether the number of unassigned tasks in the total number of tasks is greater than or equal to the maximum number of tasks that the computing core can handle; if so, assign the maximum number of tasks to the task set of the current processing batch of the computing core; if not, assign all remaining unassigned tasks to the task set of the current processing batch of the computing core; after completing the task assignment for the last computing core, if there are still unassigned tasks, return to the first computing core, start a new processing batch, and continue to execute task partitioning until all tasks are assigned.

[0085] Optionally, the processing module 402 is specifically used to determine the maximum number of tasks that each computing core can carry by means of the following method: for each computing core, the number of thread bundles contained in the computing core is determined as the maximum number of tasks that the computing core can carry.

[0086] Optionally, the processing module 402 is specifically configured to, for each processing batch and for each computing core, allocate one character from a set of multiple characters in the task set of the processing batch corresponding to the computing core to each thread bundle contained in the computing core, and have each thread bundle complete the processing of the allocated character.

[0087] Optionally, the processing module 402 is further configured to monitor the task completion status of each computing core during the parallel execution of tasks by multiple computing cores; if there is a first computing core that has completed a certain batch of tasks and there is a second computing core that has not completed the batch of tasks, control the first computing core to suspend execution for a preset period and delay the start of the first computing core to execute the tasks contained in the next batch of tasks.

[0088] Optionally, the processing module 402 is specifically used to send a sleep command to the first computing core, wherein the sleep command carries a preset period for pausing execution.

[0089] In this embodiment, the total number of tasks to be processed is accurately calculated based on the number of batches of input data and the depth of each input data. Based on the maximum number of concurrent tasks that each computing core can support, the total tasks are divided into multiple task sets for processing batches and assigned to different computing cores for execution. Each task set contains a set of tasks whose number of characters is the same as the maximum number of tasks it can handle. This allows each computing core to be fully loaded with tasks, thereby improving resource utilization efficiency.

[0090] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0091] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 5 As shown, it includes at least one artificial intelligence chip 100 and a memory 501 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 501 is not limited. Figure 5 Taking the connection between the AI ​​chip 100 and the memory 501 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0092] In this embodiment of the application, the memory 501 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 501, the at least one artificial intelligence chip 100 can perform the steps of the task allocation method described above.

[0093] The artificial intelligence chip 100 serves as the control center of the computer device. It connects to various parts of the computer device via various interfaces and lines, and performs task allocation by running or executing instructions stored in the memory 501 and accessing data stored in the memory 501. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 501 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0094] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0095] Memory 501, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 501 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 501 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 501 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0096] Based on the same concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps described above based on the task allocation method.

[0097] Based on the same concept, embodiments of this application provide a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps described above based on the task allocation method.

[0098] Those skilled in the art will understand that embodiments of this application can be provided as methods or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0103] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A task allocation method, characterized in that, The method includes: The total number of tasks to be executed is determined based on the batch size of the input data and the depth of each input data; wherein, the batch size is the total number of questions input to the large language model, and the depth is the number of characters contained in each input question; wherein, the process of processing a character is the execution of a task; Based on the maximum number of tasks each computing core can handle, the total number of tasks is divided to obtain task sets for different processing batches of each computing core; wherein, in multiple task sets corresponding to any computing core, there exists a task set containing the same number of characters corresponding to the tasks as the maximum number of tasks that computing core can handle, and the number of characters corresponding to the tasks contained in any batch of task sets corresponding to any computing core does not exceed the maximum number of tasks that computing core can handle. Control different computing cores to process the characters in the task set of their respective processing batches; The step of dividing the total number of tasks according to the maximum number of tasks each computing core can handle, to obtain a task set for different processing batches for each computing core, includes: The system iterates through each computing core in a pre-defined order to determine the task set for the current batch of processing on each core. The pre-defined order of computing cores is determined based on the data access latency or communication path length between each computing core and the video memory. For the currently traversed computing core, determine whether the number of unassigned tasks in the total number of tasks is greater than or equal to the maximum number of tasks that the computing core can handle. If so, then the maximum number of tasks to be carried will be assigned to the task set of the current processing batch of the computing core; If not, then all remaining unassigned tasks will be assigned to the task set of the current processing batch of this computing core; After completing the task allocation for the last computing core, if there are still unallocated tasks, return to the first computing core, start a new processing batch, and continue to execute the task allocation until all tasks are allocated. The control of different computing cores to process characters within the task set of their respective processing batches includes: Choose a computation axis suitable for sequential expansion within a single thread bundle, and have the thread bundle execute continuously along this axis in a segmented loop; wherein the task executed by the thread bundle is root mean square normalization RmsNorm, root mean square normalization after addition AddRmsNorm, or other Vector-type operators.

2. The method according to claim 1, characterized in that, The maximum number of tasks that each computing core can handle is determined in the following way: For each computing core, the number of thread bundles contained in that computing core is determined as the maximum number of tasks that computing core can handle.

3. The method according to claim 2, characterized in that, The control of different computing cores to process characters within the task set of their respective processing batches includes: For each processing batch, for each computing core, one character from a set of characters in the task set corresponding to that computing core is assigned to each thread bundle contained in that computing core, and each thread bundle processes the assigned character.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: During the parallel execution of tasks by multiple computing cores, the task completion status of each computing core is monitored; If there is a first computing core that has completed a certain batch of tasks, and there is a second computing core that has not completed the batch of tasks, control the first computing core to pause execution for a preset period, and delay the start of the first computing core to execute the tasks contained in the next batch of tasks.

5. The method according to claim 4, characterized in that, The control of the first computing core to pause execution for a preset period includes: A sleep command is sent to the first computing core, wherein the sleep command carries a preset period for pausing execution.

6. A task allocation device, characterized in that, The device includes: The determination module is used to determine the total number of tasks to be executed based on the batch size of the input data and the depth of each input data; wherein, the batch size is the total number of questions input to the large language model, and the depth is the number of characters contained in each input question; wherein, the process of processing a character is the execution of a task; The processing module is used to divide the total number of tasks according to the maximum number of tasks that each computing core can handle, to obtain task sets for different processing batches of each computing core; wherein, in multiple task sets corresponding to any computing core, there is a task set containing the number of characters corresponding to the tasks that is the same as the maximum number of tasks that the computing core can handle, and the number of characters corresponding to the tasks contained in any batch of task sets corresponding to any computing core does not exceed the maximum number of tasks that the computing core can handle; and controls different computing cores to process the characters in the task sets of their respective processing batches; Specifically, the processing module is used to traverse each computing core sequentially according to a preset computing core order, and determine the task set of the current processing batch for each computing core. The preset computing core order is determined based on the data access latency or communication path length between each computing core and the video memory. For the currently traversed computing core, it is determined whether the number of unassigned tasks in the total number of tasks is greater than or equal to the maximum number of tasks that the computing core can carry. If so, the maximum number of tasks is assigned to the task set of the current processing batch of the computing core. If not, all remaining unassigned tasks are assigned to the task set of the current processing batch of the computing core. After completing the task assignment for the last computing core, if there are still unassigned tasks, the module returns to the first computing core, starts a new processing batch, and continues to execute task partitioning until all tasks are assigned. The processing module is specifically used to select a computation axis suitable for sequential expansion within a single thread bundle, and to allow the thread bundle to execute continuously along this axis in a segmented loop; wherein, the task executed by the thread bundle is root mean square normalization RmsNorm, root mean square normalization after addition AddRmsNorm, or other Vector-type operators.

7. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that is executed by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method as described in any one of claims 1-5.

Images