Task execution method, electronic device, medium, program product

Abstract

The present invention provides a task execution method, electronic device, medium, and program product, relating to the field of artificial intelligence technology. The task execution method includes: in response to receiving multiple inference requests, dividing multiple inference tasks generated based on the multiple inference requests into N batches, wherein each inference task includes a computational subtask and a post-processing subtask to be executed sequentially; sequentially executing the N batches of inference tasks to obtain task execution results, wherein executing the nth batch of inference tasks includes: sending the nth batch of computational subtasks to a computing unit; when the computing unit completes execution of the nth batch of computational subtasks, sending the n+1th batch of computational subtasks to the computing unit; and synchronously executing the nth batch of post-processing subtasks while the computing unit executes the n+1th batch of computational subtasks. This task execution method can improve computing resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a task execution method, device, electronic device, medium, and program product. Background Art

[0002] Language models, with their powerful computing capabilities and the ability to train on massive amounts of data, have demonstrated broad application potential in multiple fields. When using language models for semantic reasoning, multiple calculators can collaborate to complete the computational task. For example, two calculators can be used to perform the computational task, with one performing the main reasoning calculations and the other performing post-processing.

[0003] During the implementation of this invention, it was discovered that post-processing calculations must be performed after inference calculations are complete. One approach involves having one calculator perform all the inference calculations for a batch of inference tasks, followed by another calculator performing all the post-processing calculations. Consequently, during the post-processing period, one of the calculators remains idle, wasting computing resources. Summary of the Invention

[0004] In view of the above problems, the present invention provides a task execution method, device, electronic device, medium, and program product.

[0005] According to a first aspect of the present invention, a task execution method is provided, the method comprising: in response to receiving multiple inference requests, dividing multiple inference tasks generated based on the multiple inference requests into N batches, wherein each inference task includes a computing subtask and a post-processing subtask to be executed sequentially, and N is an integer greater than 2; executing N batches of inference tasks in sequence to obtain task execution results, wherein executing the nth batch of inference tasks comprises: sending the nth batch of computing subtasks to a computing unit; when the computing unit completes executing the nth batch of computing subtasks, sending the n+1th batch of computing subtasks to the computing unit, and synchronously executing the nth batch of post-processing subtasks while the computing unit executes the n+1th batch of computing subtasks, n=1,…,N-2.

[0006] The second aspect of the present invention provides a task execution device, including: a batching module, used to divide multiple reasoning tasks generated based on the multiple reasoning requests into N batches in response to receiving multiple reasoning requests, wherein each reasoning task includes a computing subtask and a post-processing subtask to be executed in sequence, and N is an integer greater than 2; an execution module, used to execute N batches of reasoning tasks in sequence to obtain task execution results, wherein executing the nth batch of reasoning tasks includes: sending the nth batch of computing subtasks to the computing unit; when the computing unit completes executing the nth batch of computing subtasks, sending the n+1th batch of computing subtasks to the computing unit, and synchronously executing the nth batch of post-processing subtasks during the process of the computing unit executing the n+1th batch of computing subtasks, n=1,...,N-2.

[0007] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0008] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0009] The fifth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0010] According to an embodiment of the present invention, by dividing the received multiple inference tasks into multiple batches and sequentially passing the computational subtasks of the multiple batches to the computing unit for calculation, this strategy of separating the computations of multiple tasks allows the computing unit to execute the computational subtasks synchronously without waiting for the post-processing subtasks to execute. This approach can improve the utilization of computing resources, especially when processing large-scale data and complex models, and can increase computing speed, thereby improving inference efficiency without compromising model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0012] Figure 1 A system architecture diagram of a task execution method according to an embodiment of the present invention is shown;

[0013] Figure 2 A flowchart of a task execution method according to an embodiment of the present invention is shown;

[0014] Figure 3 A schematic diagram of a task execution method in related art is shown;

[0015] Figure 4 A schematic diagram showing a task execution method according to an embodiment of the present invention is shown;

[0016] Figure 5 A flowchart of a task execution method according to another embodiment of the present invention is shown;

[0017] Figure 6 A schematic diagram showing a task execution method according to another embodiment of the present invention is shown;

[0018] Figure 7 It shows a structural block diagram of a task execution device according to an embodiment of the present invention;

[0019] Figure 8 A block diagram of an electronic device suitable for implementing a task execution method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0021] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0023] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0024] An embodiment of the present invention provides a task execution method, comprising: in response to receiving multiple inference requests, dividing multiple inference tasks generated based on the multiple inference requests into N batches, wherein each inference task includes a computing subtask and a post-processing subtask to be executed sequentially, and N is an integer greater than 2; executing N batches of inference tasks in sequence to obtain task execution results, wherein executing the nth batch of inference tasks comprises: sending the nth batch of computing subtasks to a computing unit; when the computing unit completes executing the nth batch of computing subtasks, sending the n+1th batch of computing subtasks to the computing unit, and synchronously executing the nth batch of post-processing subtasks during the process of the computing unit executing the n+1th batch of computing subtasks, n=1,…,N-2.

[0025] Figure 1 A system architecture diagram of a task execution method according to an embodiment of the present invention is shown.

[0026] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a processing unit 101 and a computing unit 102. Data can be transmitted between the processing unit 101 and the computing unit 102 via a network, which may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0027] The processing unit 101 and the computing unit 102 may be various types of computing devices, including but not limited to a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), and the like.

[0028] For example, the processing unit 101 may be a CPU and the computing unit 102 may be a GPU. For example, the processing unit 101 may be a CPU and the computing unit 102 may be a CPU. For example, the processing unit 101 may be a GPU and the computing unit 102 may be a GPU.

[0029] In the scenario of the embodiment of the present invention, the processing unit 101 and the computing unit 102 can work together to complete the reasoning task of semantic reasoning using the language model, and are not limited to semantic reasoning tasks. Any type of computing task scenario involving multiple computing stages is applicable to the method of the embodiment of the present invention.

[0030] For example, taking a complete reasoning task including pre-processing, calculation, and post-processing operations as an example, processing unit 101 and calculation unit 102 can collaborate to complete the reasoning task: processing unit 101 performs pre-processing and / or post-processing operations, while calculation unit 102 focuses on completing the primary calculation task, that is, primarily performing forward calculation operations. After completing pre-processing, processing unit 101 sends the pre-processing results and the calculation task to calculation unit 102. Calculation unit 102 performs forward calculations and, after completing the forward calculations, returns the calculation results to processing unit 101 so that processing unit 101 can perform post-processing operations based on the calculation results.

[0031] The following will be based on Figure 1 The architecture described by Figures 2 to 8 The task execution method of the embodiment of the invention is described in detail.

[0032] Figure 2 FIG. 1 shows a flow chart of a task execution method according to an embodiment of the present invention. Figure 2 As shown, the task execution method of this embodiment includes operations S201 to S202.

[0033] In operation S201, in response to receiving multiple inference requests, multiple inference tasks generated based on the multiple inference requests are divided into N batches, where each inference task includes a computation subtask and a post-processing subtask to be executed sequentially, and N is an integer greater than 2;

[0034] In operation S202, N batches of inference tasks are executed in sequence to obtain task execution results, wherein executing the nth batch of inference tasks includes:

[0035] Send the nth batch of computing subtasks to the computing unit;

[0036] When the computing unit completes the nth batch of computing subtasks, the n+1th batch of computing subtasks is sent to the computing unit, and while the computing unit is executing the n+1th batch of computing subtasks, the nth batch of post-processing subtasks is synchronously executed, where n=1,…,N-2.

[0037] According to embodiments of the present invention, the aforementioned task execution method can be applied in scenarios where semantic reasoning is performed using a language model. That is, the aforementioned reasoning task can be a semantic reasoning task. It should be noted that the aforementioned task execution method is not limited to semantic reasoning scenarios; any scenario involving a two-stage computational task (where the computation of the latter stage depends on the results of the computation of the previous stage) is also applicable.

[0038] Exemplarily, the semantic reasoning task of using a language model for semantic reasoning includes a calculation subtask and a post-processing subtask to be executed in sequence. The calculation subtask mainly completes the forward calculation, which is the main calculation task that the language model needs to complete and involves a large amount of calculation. In this process, the language model will perform a complete forward calculation based on the input basic word vector to generate multiple candidate word vectors and the probability distribution of each of the multiple candidate word vectors. This probability distribution contains the probability value of the next possible generated word unit. After the calculation subtask is completed, the post-processing subtask is executed. The post-processing subtask can be to select a target word vector from multiple candidate word vectors based on a predetermined sampling strategy as the vector of the output word unit, and convert the target word vector selected based on the probability distribution into text to obtain a semantic reasoning result.

[0039] According to embodiments of the present invention, inference tasks can be collaboratively performed using a processing unit and a computing unit. The processing unit and computing unit can be various types of computing units, including but not limited to CPUs, GPUs, and FPGAs. The processing unit and computing unit can be the same or different. The computing unit performs the computational subtasks, while the processing unit performs the post-processing subtasks.

[0040] The processing unit and the computing unit use different computing units. For example, the processing unit uses a CPU, and the computing unit uses a GPU or FPGA. This can take advantage of the CPU's suitability for complex but less computationally intensive tasks, while the GPU or FPGA's suitability for simpler but computationally intensive tasks. This allows the GPU or FPGA to perform computational subtasks while the CPU performs post-processing subtasks.

[0041] The task execution method of this embodiment is executed by a processing unit.

[0042] In the above operation S201, in response to receiving multiple reasoning requests (for example, multiple semantic reasoning requests initiated by multiple users), the processing unit generates multiple reasoning tasks based on the multiple reasoning requests, which may be one reasoning task based on one request, and divides the multiple reasoning tasks into N batches.

[0043] To improve language model inference efficiency, the above method consolidates multiple inference requests into a batch for unified processing. Specifically, when multiple users simultaneously initiate inference requests to the model, the system does not immediately initiate a separate inference process for each request. Instead, these requests are temporarily "queued" and grouped into a larger batch according to specific rules. The model then processes the entire batch of input data at once to perform inference tasks. This approach reduces scheduling overhead. In the single-inference request processing model, each inference requires a series of scheduling operations, such as task allocation and resource allocation, which consume a certain amount of time and computing resources. By consolidating multiple requests into a batch, the system only needs to perform a single scheduling operation to process multiple inference tasks simultaneously, thus saving scheduling time. Furthermore, this batch processing method improves the utilization of computing units. Computing units (such as GPUs) generally better utilize their parallel computing capabilities when processing large amounts of data. When multiple inference requests are consolidated into a batch, the device can more fully utilize its computing resources, avoiding the resource waste associated with processing individual requests, thereby improving overall inference efficiency.

[0044] Furthermore, the task execution method can further improve resource utilization by sequentially executing N batches of inference tasks through the execution logic of the operation S202.

[0045] The execution of any (e.g., nth batch) of inference tasks includes: sending the nth batch of computational subtasks to a computing unit so that the computing unit executes the nth batch of computational subtasks; and, after the computing unit completes the nth batch of computational subtasks, sending the n+1th batch of computational subtasks to the computing unit; and synchronously executing the nth batch of post-processing subtasks while the computing unit executes the n+1th batch of computational subtasks. The task execution method of the present invention can improve the utilization of computing resources.

[0046] Figure 3 A schematic diagram of a task execution method in related art is shown. Figure 4 The following is a schematic diagram showing a task execution method according to an embodiment of the present invention. Figure 3 、 Figure 4 The technical principle of how the method according to the embodiment of the present invention can improve the utilization rate of computing resources is explained.

[0047] like Figure 3 As shown, taking the case where the processing unit adopts CPU and the computing unit adopts GPU as an example, in the related technology, after receiving a batch of inference tasks, the GPU first performs all computing tasks. After completing all computing tasks, the GPU needs to transfer the calculation results to the CPU for post-processing, and then the CPU performs all post-processing tasks.

[0048] Thus, while the CPU is performing post-processing tasks, the GPU is idle. Because these post-processing operations primarily involve data transfer and simple logical operations, rather than complex computational tasks, this idle state means that the computing unit's computing resources are not fully utilized during this period, resulting in a waste of computing resources.

[0049] To improve computing resource utilization, the method in this embodiment of the present invention divides the received inference tasks into batches, and sequentially passes the computational subtasks of these batches to the GPU for computation. By separating the computations of multiple tasks, GPU utilization is increased, further improving the efficiency of language model inference.

[0050] like Figure 4 As shown in the figure, after the GPU completes the first batch of computational subtasks, the CPU executes the first batch of post-processing subtasks and simultaneously sends the second batch of computational subtasks to the GPU, allowing them to execute synchronously. After the second batch of computational subtasks completes, the CPU executes the second batch of post-processing subtasks, and so on, executing each batch of tasks in sequence until all tasks are completed.

[0051] As can be seen, the above method divides the received multiple inference tasks into multiple batches, and the computational subtasks of these batches are sequentially passed to the computing unit for calculation. This strategy of separating the calculations of multiple tasks allows the computing unit to execute the computational subtasks synchronously without waiting for the post-processing subtasks to execute. This method can improve the utilization of computing resources, especially when processing large-scale data and complex models, and can improve the calculation speed, thereby improving inference efficiency without compromising model performance.

[0052] According to an embodiment of the present invention, when the above-mentioned task execution method is applied to a scenario where semantic reasoning is performed using a language model, each reasoning task further includes a pre-processing subtask that is executed before the calculation subtask.

[0053] Based on this, the above method further includes: before sending the nth batch of computation subtasks to the computing unit, executing the nth batch of pre-processing subtasks. The pre-processing subtasks primarily include embedding the user input text to generate a vector representation, i.e., converting it into a data format that can be processed by the model for the model to process.

[0054] Figure 5 FIG. 1 shows a flow chart of a task execution method according to another embodiment of the present invention. Figure 5As shown, the task execution method of this embodiment, after dividing multiple reasoning tasks into N batches, includes sequentially executing pre-processing, calculation, and post-processing operations for the semantic reasoning tasks of any batch; wherein the pre-processing operation is completed by executing the pre-processing subtask, the calculation operation is completed by executing the calculation subtask, and the post-processing operation is completed by executing the post-processing subtask.

[0055] For example, in the scenario of using a language model for semantic reasoning, the language model will perform pre-processing, calculation, and post-processing operations in sequence based on the information input by the user (such as text). Among them, the pre-processing operation mainly embeds the text input by the user to generate a vector representation. The pre-processing operation may also include data pre-processing of the input text, such as data cleaning (de-duplication, de-anomaly, missing completion, etc.). The calculation operation mainly performs a complete forward calculation based on the word vector generated in the previous step to generate multiple candidate word vectors and the probability distribution of each of the multiple candidate word vectors. The post-processing operation mainly selects the next word element (target word vector) based on a predetermined sampling strategy, and converts the target word vector selected based on the probability distribution into a semantic reasoning result.

[0056] According to an embodiment of the present invention, when the processing unit and the computing unit are used to collaboratively complete the reasoning task, the task execution method may specifically include: the processing unit performs pre-processing and / or post-processing operations, and the computing unit focuses on completing the main computing task, that is, mainly performing forward computing operations. After completing the pre-processing, the processing unit sends the pre-processing results and the computing task to the computing unit. The computing unit performs forward computing and returns the computing results to the processing unit after completing the forward computing so that the processing unit can perform post-processing operations based on the computing results.

[0057] According to an embodiment of the present invention, the above method further includes: when the computing unit executes the n+1th batch computing subtask, when the nth batch post-processing subtask is completed, executing the n+2th batch pre-processing subtask.

[0058] The computational workload involved in a computational unit is relatively large, so executing computational tasks takes a long time. While the computational unit is executing the current batch of computational subtasks, the processing unit can pre-process the next batch of pre-processing subtasks after the processing unit has completed the previous batch of post-processing subtasks. This allows the computational unit to immediately begin executing the next batch of computational subtasks without waiting. This allows for further parallel operation of the two computational units, further improving computational resource utilization.

[0059] Figure 6 The following is a schematic diagram showing a task execution method according to another embodiment of the present invention. Figure 6 The principle that the task execution method according to the embodiment of the present invention can further improve resource utilization is described.

[0060] like Figure 6 As shown, taking the case where the processing unit uses a CPU and the computing unit uses a GPU as an example: the CPU first executes the first batch of pre-processing subtasks; after the first batch of pre-processing subtasks are completed, the GPU executes the first batch of computing subtasks; while the GPU executes the first batch of computing subtasks, the CPU executes the second batch of pre-processing subtasks in advance; after the GPU executes the first batch of computing subtasks, the CPU executes the first batch of post-processing subtasks; at the same time, the GPU executes the second batch of computing subtasks; while the GPU executes the second batch of computing subtasks, after the CPU executes the first batch of post-processing subtasks, the CPU executes the third batch of pre-processing subtasks in advance... and so on, executing each batch of tasks in sequence until all tasks are completed. It can be seen that while the GPU executes the current batch of computing subtasks, the CPU is synchronously using the CPU to synchronously process the next batch of pre-processing subtasks in advance. In this way, after the GPU completes the previous batch of computing subtasks, it can immediately start executing the next batch of computing subtasks without waiting, thereby improving computing resource utilization.

[0061] According to an embodiment of the present invention, in a scenario where a language model is used for semantic reasoning, the reasoning task is used to indicate: calling a predetermined language model to perform semantic reasoning based on the request data carried in the reasoning request to generate a semantic reasoning result, and the semantic reasoning process includes sequentially executing a pre-processing subtask, a calculation subtask, and a post-processing subtask. The language model will perform pre-processing, calculation, and post-processing operations based on the information input by the user, that is, based on the request data carried in the reasoning request initiated by the user, such as text, voice, image, etc. Semantic reasoning can be used to perform task scenarios such as question and answer, translation, and recommendation.

[0062] The semantic reasoning process involves a multi-layer network structure within the language model. This multi-layer network structure can be a combination of various forms. For example, it can include an embedding layer, multiple attention layers, a mapping layer (multi-layer perceptron), a decoding layer, etc. For example, it can also include an embedding layer, multiple encoders or decoders, etc. For example, it can also include an embedding layer, a multi-layer perceptron, and a decoding layer. The embodiments of the present invention do not limit the network architecture of the language model.

[0063] Below, we take the language model architecture including an embedding layer, multiple attention layers, and a mapping layer as an example, and take the execution of the nth batch of inference tasks as an example to illustrate the operations of the pre-processing subtask, calculation subtask, and post-processing subtask in sequence.

[0064] According to an embodiment of the present invention, executing the nth batch pre-processing subtask includes: by executing the nth batch pre-processing subtask, calling the embedding layer in the predetermined language model to encode (word embedding) the nth batch request data, and generating a basic word vector, wherein the nth batch request data is carried in the inference request for the nth batch inference task.

[0065] The request data can be at least one of user-entered text, voice, or images. If the request data is voice or image, the pre-processing subtask also includes converting the voice into text through voice recognition, or recognizing text in the image through text recognition. After pre-processing, each word in the text becomes a vector (a list of numbers).

[0066] According to an embodiment of the present invention, executing the nth batch calculation subtask includes: the computing unit calls the attention layer and the mapping layer in the predetermined language model by executing the nth batch calculation subtask to process the basic word vector and generate multiple candidate word vectors and the probability distribution of each of the multiple candidate word vectors.

[0067] In this structural layer of the language model, an attention layer and a mapping layer form a computational network layer. Multiple computational network layers process the input basic word vectors using a multi-head attention mechanism. Each computational network layer extracts features and interacts with the input data, generating multiple candidate word vectors and their respective probability distributions. This probability distribution includes the probability of the next possible word unit being generated.

[0068] The operations performed at each computing network layer include the following:

[0069] First, the attention layer calculates the correlation between each pair of word vectors to generate a correlation matrix. Specifically, the basic word vectors are mapped into a query, key, and value matrix. The similarity between the query and key is calculated using a dot product, and then scaled to obtain attention weights. The attention weights are then used to weight the values and sum them to generate a new vector representation for each word (including global context information), generating multiple candidate word vectors.

[0070] Afterwards, the output of the attention layer is subjected to nonlinear transformation and feature enhancement through a mapping layer, such as a multilayer perceptron (MLP). The specific processing includes: the MLP independently processes the representation of each word unit, and maps the features to a higher-dimensional space through a linear transformation (with an activation function in the middle), and then projects them back to the original dimension to enhance the expressive power of the model and output the probability distribution of multiple candidate word vectors.

[0071] According to an embodiment of the present invention, executing the nth batch post-processing subtask includes: determining a target word vector from multiple candidate word vectors based on probability distribution by executing the nth batch post-processing subtask, and converting the target word vector into the nth batch semantic reasoning result.

[0072] After the forward computation is complete, post-processing can involve determining a target word vector from multiple candidate word vectors based on a probability distribution, or selecting the next word element based on a predetermined sampling strategy. The sampling strategy can be arbitrary, such as random sampling, top-k sampling, or top-p sampling. Post-processing also involves converting the target word vector selected based on the probability distribution into a semantic reasoning result (i.e., de-embedding), generating diverse text content.

[0073] In this way, after completing pre-processing, calculation, and post-processing, a complete inference task is completed. This inference task is continuously repeated in an autoregressive manner, with each output serving as the next input (because the language model needs to use previous context to predict the next token) until a termination condition is met, such as reaching a preset maximum generation length or generating a specific terminator (such as a period or line break). Through this process, the language model gradually constructs a text sequence, taking into account the semantics and context of the previous text at each step, thereby generating coherent and natural text content.

[0074] According to an embodiment of the present invention, the on-chip storage of the computing unit is pre-divided into N storage areas corresponding to N batches of inference tasks; the N storage areas are used to store process data during the execution of the N batches of inference tasks in a one-to-one correspondence, and the process data includes input data and output data. The input data is the pre-processing result generated by executing the pre-processing subtask, and the pre-processing result serves as the basic data required for the computing unit to execute the computing subtask; after the computing unit completes the computing subtask, the computing result is generated, and the computing result serves as the output data.

[0075] Based on this, for any nth batch of inference tasks, the above method also includes: when the nth batch pre-processing subtask is completed, generating the nth batch of basic data required for the computing unit to execute the nth batch calculation subtask, and sending the nth batch of basic data to the computing unit so that the nth batch of basic data is stored in the nth storage area among the N storage areas; reading the nth batch of calculation results generated by the computing unit executing the nth batch calculation subtask based on the nth batch basic data from the nth storage area.

[0076] After the above operations, the basic data and calculation results corresponding to the nth batch are stored in the nth storage area of the computing unit. In this way, the N storage areas store the process data of the N batches of reasoning tasks in a one-to-one correspondence.

[0077] According to an embodiment of the present invention, in an actual computing environment, memory resources are often limited and scattered. By pre-allocating memory to computing units, conflicts between reading and writing different batches of data are avoided, thereby avoiding blockages in the data processing process. This allows the processing of different batches to be performed synchronously, improving computing efficiency. In addition, pre-dividing the memory area also avoids memory fragmentation caused by random dynamic memory allocation methods, thereby improving storage utilization (memory fragmentation will lead to a decrease in memory utilization and increase memory access latency). In the case of limited storage space, the method of the embodiment of the present invention has good applicability and improves memory utilization. Since there is no memory fragmentation, the entire storage space can be fully utilized, reducing the performance loss caused by memory fragmentation, thereby improving computing speed.

[0078] According to an embodiment of the present invention, each storage area is further pre-divided into a first area and a second area; the nth batch of basic data is stored in the first area of the nth storage area; and the nth batch of computation results is stored in the second area of the nth storage area. That is, after pre-dividing N storage areas to store the process data of N batches of inference tasks in a one-to-one correspondence, each storage area is further divided into two independent storage areas to achieve independent storage of input and output data. This memory partitioning approach is intended to optimize data storage and computation processes, improving the overall system efficiency.

[0079] For example, if the processing unit uses a CPU and the computing unit uses a GPU, after the nth batch of pre-processing results (i.e., basic data) is transferred from the CPU to the GPU, the basic data for the nth batch is written to the first area of the nth storage area. The GPU reads the basic data for the nth batch from the first area of the nth storage area, performs calculations, generates the calculation results for the nth batch, and writes the calculation results for the nth batch to the second area of the nth storage area. The CPU reads the calculation results for the nth batch from the second area of the nth storage area and performs post-processing calculations for the nth batch. During this process, while the GPU reads the basic data for the nth batch from the first area of the nth storage area and performs calculations, the first area can be cleared to allow the writing of the next batch of basic data. After the CPU reads the calculation results for the nth batch from the second area of the nth storage area, the second area can be cleared to allow the writing of the next batch of calculation results. This entire process repeats in this manner until the inference task is fully completed. This iterative calculation and data processing method ensures an efficient and orderly inference process. If the first area and the second area are not divided in advance, when the GPU reads the basic data of the nth batch from the nth storage area for calculation, the storage area still needs to be occupied to store the calculation results of the nth batch. The storage area cannot be cleared until the calculation results of the nth batch are read. Only then can the basic data of the next batch be written. This requires extended waiting time and increases the conflict between reading and writing.

[0080] According to an embodiment of the present invention, to avoid conflicts between reads and writes, pre-allocated input and output memory stores input data and calculation results in separate memory areas, thereby avoiding data processing blockages. Furthermore, input data preparation and storage can be performed synchronously during the calculation process, reducing memory access conflicts and waiting time, optimizing the data scheduling process, and making data input, calculation, and output smoother, reducing unnecessary waiting and delays. Furthermore, data is divided into different batches for sequential calculation execution, and writing and reading different batches can be performed in parallel, avoiding frequent memory read and write conflicts, reducing memory access delays, and improving data transmission efficiency, thereby indirectly accelerating calculation speed. Furthermore, the computing unit and the first processor can focus on their respective tasks, reducing the frequency of context switching. Because context switching consumes a large amount of time and computing resources, reducing the overhead of context switching can make the calculation process more efficient. This effectively covers the time required for each post-processing operation, effectively improving computing unit utilization.

[0081] According to an embodiment of the present invention, for any nth batch of inference tasks, the nth batch of computational subtasks includes multiple computational units. Furthermore, multiple computational units can be configured to execute the nth batch of computational subtasks in parallel. By having multiple computational units execute the same batch of computational subtasks in parallel, computational efficiency can be further improved.

[0082] According to an embodiment of the present invention, multiple computational subtasks in the nth batch correspond to multiple inference tasks, and one computational subtask corresponds to one inference task. For example, if 10 users initiate 10 inference requests, the 10 inference tasks generated based on these 10 inference requests are divided into two batches. The inference tasks of these two batches are then executed sequentially using the method of an embodiment of the present invention to obtain the task execution results. For any batch containing 5 inference tasks, these 5 inference tasks correspond to 5 computational subtasks, and these 5 computational subtasks can be executed in parallel by multiple computing units.

[0083] Furthermore, each computational subtask is divided into multiple task stages. For example, in a scenario where a language model is used for semantic reasoning, executing any computational subtask of the nth batch includes: the computation unit performs computations by calling multiple computational network layers (one attention layer and one mapping layer constitute one computational network layer) in a predetermined language model. The computational task of one computational network layer corresponds to one task stage. The execution of multiple task stages has a sequential dependency relationship. After the computation of the current task stage (current computational network layer) is completed, the result needs to be input into the next task stage (next computational network layer) for computation. That is, the execution of the subsequent task stage depends on the execution result of the previous task stage. In this way, multiple task stages are executed sequentially until the computational subtask is completed.

[0084] According to an embodiment of the present invention, executing multiple computing subtasks of the nth batch in parallel by multiple computing units may include the following operations:

[0085] Multiple computing units sequentially execute multiple task phases of a current computing subtask from among multiple computing subtasks, where the i-th computing unit executes the i-th task phase. For example, a processing unit executes the first task phase. After the first task phase completes, a computing unit executes the second task phase. After the second task phase completes, a third computing unit executes the third task phase, and so on, where i is a positive integer.

[0086] Among them, after the i-th computing unit completes the i-th task stage of the current computing subtask, the i-th computing unit does not need to wait, and can immediately execute the i-th task stage of the next computing subtask, and so on until all computing subtasks are completed.

[0087] For example, a batch of computing subtasks includes three, with two computing units. Each computing subtask can include two task phases. After the processing unit completes the first task phase of the first computing subtask, the computing unit executes the second task phase of the first computing subtask. Furthermore, after the processing unit completes the first task phase of the first computing subtask, it does not need to wait and immediately executes the first task phase of the second computing subtask. After the computing unit completes the second task phase of the first computing subtask, it does not need to wait and immediately executes the second task phase of the second computing subtask. After the processing unit completes the first task phase of the second computing subtask, it does not need to wait and immediately executes the first task phase of the third computing subtask. After the computing unit completes the second task phase of the second computing subtask, it does not need to wait and immediately executes the second task phase of the third computing subtask. In this way, multiple computing units can complete multiple computing subtasks in parallel, which can fully improve the utilization of the computing units and speed up the computing speed.

[0088] Based on the above task execution method, the present invention also provides a task execution device. Figure 7 The device is described in detail.

[0089] Figure 7 A structural block diagram of a task execution device according to an embodiment of the present invention is shown.

[0090] like Figure 7 As shown, the task execution device 700 of this embodiment includes a batch module 701 and an execution module 702 .

[0091] Among them, the batch module 701 is used to divide the multiple inference tasks generated based on the multiple inference requests into N batches in response to receiving multiple inference requests, wherein each inference task includes a computing subtask and a post-processing subtask to be executed sequentially, and N is an integer greater than 2; in one embodiment, the batch module 701 can be used to perform the operation S201 described above, which will not be repeated here.

[0092] Execution module 702 is configured to sequentially execute N batches of inference tasks to obtain task execution results. Execution of the nth batch of inference tasks includes: sending the nth batch of computational subtasks to the computing unit; after the computing unit completes execution of the nth batch of computational subtasks, sending the n+1th batch of computational subtasks to the computing unit; and while the computing unit is executing the n+1th batch of computational subtasks, synchronously executing the nth batch of post-processing subtasks, where n = 1, ..., N-2. In one embodiment, execution module 702 may be configured to execute operation S202 described above and will not be further described herein.

[0093] According to an embodiment of the present invention, each reasoning task further includes a pre-processing subtask that is executed before the calculation subtask.

[0094] The execution module 702 is further configured to execute the nth batch of pre-processing subtasks before sending the nth batch of computing subtasks to the computing unit.

[0095] According to an embodiment of the present invention, the execution module 702 is further configured to: when the computing unit executes the n+1th batch computing subtask, upon completion of the nth batch post-processing subtask, execute the n+2th batch pre-processing subtask.

[0096] According to an embodiment of the present invention, the reasoning task is used to instruct: calling a predetermined language model to perform semantic reasoning based on request data carried in the reasoning request to generate a semantic reasoning result.

[0097] The execution module 702 is used to call the embedding layer in the predetermined language model to encode the nth batch of request data by executing the nth batch pre-processing subtask to generate basic word vectors, wherein the nth batch of request data is carried in the inference request for the nth batch of inference tasks.

[0098] According to an embodiment of the present invention, the execution module 702 is used to instruct the computing unit to call the attention layer and mapping layer in the predetermined language model by executing the nth batch of computing subtasks, so as to process the basic word vector and generate multiple candidate word vectors and the probability distribution of each of the multiple candidate word vectors.

[0099] According to an embodiment of the present invention, the execution module 702 is used to determine the target word vector from multiple candidate word vectors based on the probability distribution by executing the nth batch post-processing subtask, and convert the target word vector into the nth batch semantic reasoning result.

[0100] According to an embodiment of the present invention, the on-chip storage of the computing unit is pre-divided into N storage areas corresponding to N batches of inference tasks;

[0101] According to an embodiment of the present invention, the above-mentioned device further includes a generating module and a reading module.

[0102] The generation module is used to generate the nth batch of basic data required for the computing unit to execute the nth batch of computing subtasks when the nth batch of pre-processing subtasks is completed, and send the nth batch of basic data to the computing unit so that the nth batch of basic data is stored in the nth storage area among the N storage areas.

[0103] The reading module is used to read from the nth storage area the nth batch of calculation results generated by the calculation unit executing the nth batch of calculation subtasks based on the nth batch of basic data.

[0104] According to an embodiment of the present invention, each storage area is pre-divided into a first area and a second area; the nth batch of basic data is stored in the first area of the nth storage area; and the nth batch of calculation results is stored in the second area of the nth storage area.

[0105] According to an embodiment of the present invention: the nth batch of computing subtasks includes multiple; the computing units include multiple; the multiple computing units execute the nth batch of computing subtasks in parallel.

[0106] According to embodiments of the present invention, any multiple modules in the batch module 701 and the execution module 702 can be combined into a single module, or any one of them can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present invention, at least one of the batch module 701 and the execution module 702 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware through any other suitable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any suitable combination of any of these. Alternatively, at least one of the batch module 701 and the execution module 702 can be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0107] Figure 8 A block diagram of an electronic device suitable for implementing a task execution method according to an embodiment of the present invention is shown.

[0108] like Figure 8 As shown, an electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 802 or programs loaded from a storage unit 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0109] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the programs in the ROM 802 and / or RAM 803 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0110] According to an embodiment of the present invention, electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to bus 804. Electronic device 800 may also include one or more of the following components connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or modem. Communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 810 as needed, so that computer programs read from the removable media can be installed into storage section 808 as needed.

[0111] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0112] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above, and / or one or more memories other than ROM 802 and RAM 803.

[0113] The embodiments of the present invention further include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the task execution method provided by the embodiments of the present invention.

[0114] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when executed by the processor 801. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0115] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0116] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809 and / or installed from a removable medium 811. When the computer program is executed by the processor 801, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0117] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0120] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A task execution method, characterized in that: The method comprises: In response to receiving multiple inference requests, dividing multiple inference tasks generated based on the multiple inference requests into N batches, wherein each of the inference tasks includes a computation subtask and a post-processing subtask to be executed sequentially, and N is an integer greater than 2; the inference tasks are used to instruct: calling a predetermined language model to perform semantic inference based on the request data carried in the inference requests to generate a semantic inference result; Execute N batches of inference tasks in sequence to obtain task execution results. Executing the nth batch of inference tasks includes: By executing the nth batch pre-processing subtask, calling the embedding layer in the predetermined language model to encode the nth batch of request data to generate basic word vectors, wherein the nth batch of request data is carried in the inference request for the nth batch of inference tasks; Sending the nth batch of computing subtasks to a computing unit, so that the computing unit calls the attention layer and the mapping layer in the predetermined language model by executing the nth batch of computing subtasks, processes the basic word vectors, and generates a plurality of candidate word vectors and probability distributions of the plurality of candidate word vectors; When the computing unit completes executing the nth batch of computing subtasks, the n+1th batch of computing subtasks is sent to the computing unit, and during the process of the computing unit executing the n+1th batch of computing subtasks, the nth batch of post-processing subtasks is synchronously executed, where n=1, ..., N-2.

2. The method according to claim 1, characterized in that The method further comprises: During the process of the computing unit executing the (n+1)th batch computing subtask, if the (n)th batch post-processing subtask is completed, the (n+2)th batch pre-processing subtask is executed.

3. The method according to claim 1, characterized in that The execution of the nth batch post-processing subtask includes: By executing the nth batch post-processing subtask, a target word vector is determined from the multiple candidate word vectors based on the probability distribution, and the target word vector is converted into the nth batch semantic reasoning result.

4. The method according to claim 1, wherein The on-chip storage of the computing unit is pre-divided into N storage areas corresponding to N batches of inference tasks; The method further comprises: generating, upon completion of the nth batch pre-processing subtask, an nth batch of basic data required by the computing unit to execute the nth batch computing subtask, and sending the nth batch of basic data to the computing unit so as to store the nth batch of basic data in the nth storage area among the N storage areas; An nth batch of calculation results generated by the calculation unit executing the nth batch of calculation subtasks based on the nth batch of basic data is read from the nth storage area.

5. The method according to claim 4, characterized in that: Each of the storage areas is pre-divided into a first area and a second area; The nth batch of basic data is stored in the first area of the nth storage area; The nth batch of calculation results is stored in the second area of the nth storage area.

6. The method according to any one of claims 1 to 5, characterized in that: The nth batch of computing subtasks includes multiple; The computing unit includes a plurality of; The plurality of computing units execute the nth batch of computing subtasks in parallel.

7. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program or instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images