Inference acceleration method, device and storage medium

Abstract

Embodiments of the present application provide a reasoning acceleration method and device and storage medium. In the reasoning execution phase, in response to the reasoning instruction, the acceleration demand parameter value corresponding to each reasoning operation to be accelerated in the reasoning model in the current reasoning process is determined. In this way, the dynamic change of the acceleration demand parameter value corresponding to each reasoning operation in the reasoning execution phase can be perceived in real time, so that each reasoning operation can be adaptively matched with a suitable acceleration operator following the dynamic change of the acceleration demand parameter, ensuring that each reasoning operation to be accelerated in the reasoning model can be matched with a suitable acceleration operator in each reasoning process. Compared with the traditional scheme of matching the operator in the reasoning initialization phase and fixing it to the reasoning execution phase, adaptive matching of the acceleration operator can be realized, thereby supporting complex and variable reasoning instructions in the reasoning execution phase, and further ensuring the reasoning acceleration performance in each reasoning process.

Description

Technical Field

[0001] This application relates to the field of AI technology, and in particular to a method, device and storage medium for accelerating inference. Background Technology

[0002] With the development of natural language processing and image processing, the MHA (Multi-Head Attention) mechanism has been widely applied to various inference models, especially inference models with similar BERT structures.

[0003] Currently, the typical approach is to match kernels (kernel functions on the GPU) for each MHA in the inference model during the inference initialization phase. The resulting matching results are then fixed and applied during the inference execution phase. In other words, during the inference execution phase, the corresponding kernel is called for each MHA according to the fixed matching results, thereby accelerating inference.

[0004] As inference requirements become increasingly diverse, the aforementioned kernel matching method is too rigid, often resulting in the invocation of inappropriate kernels and unstable inference acceleration performance. Summary of the Invention

[0005] This application provides several aspects of an inference acceleration method, apparatus, and storage medium for improving inference acceleration performance.

[0006] This application provides an inference acceleration method, including:

[0007] During the execution of the inference model, in response to inference instructions, the acceleration requirement parameter values ​​corresponding to the target inference operation in this round of inference are determined;

[0008] Find a target acceleration operator that matches the acceleration requirement parameter value;

[0009] During this round of inference, the target acceleration operator is invoked for the target inference operation to accelerate the target inference operation;

[0010] The target inference operation is any inference operation to be accelerated contained in the inference model.

[0011] Further, finding a target acceleration operator that matches the acceleration requirement parameter value includes:

[0012] Based on the attribute description information corresponding to each acceleration operator in the acceleration operator library, an acceleration operator with attribute description information that meets the acceleration requirement parameter value is searched from the acceleration operator library and used as the target acceleration operator.

[0013] Furthermore, the acceleration requirement parameter values ​​include the parameter values ​​required by the target inference operation under the first type of parameters and the parameter values ​​required by the target inference operation under the second type of parameters. The first type of parameters are configured as data values ​​in the attribute description information of each acceleration operator, and the second type of parameters are configured as data values ​​or data ranges in the attribute description information of a single acceleration operator. Searching for acceleration operators with attribute description information that matches the acceleration requirement parameter values ​​from the acceleration operator library, as the target acceleration operator, includes:

[0014] From the acceleration operator library, find acceleration operators with attribute description information that meet the parameter values ​​required by the target inference operation under the first type of parameters, and use them as candidate acceleration operators;

[0015] From the candidate acceleration operators, find the acceleration operator that has attribute description information that meets the parameter values ​​required by the target inference operation under the second type of parameters, and use it as the target acceleration operator.

[0016] Furthermore, from the acceleration operator library, acceleration operators with attribute description information that conforms to the parameter values ​​required by the target inference operation under the first type of parameters are searched as candidate acceleration operators, including:

[0017] Construct the query key value corresponding to the target inference operation according to the parameter values ​​required by the target inference operation under the first type of parameters;

[0018] Based on the mapping relationship between the identifier of the acceleration operator and the query key value, the candidate acceleration operator corresponding to the target inference operation is determined.

[0019] Further, from the candidate acceleration operators, an acceleration operator with attribute description information that meets the parameter values ​​required by the target inference operation under the second type of parameters is selected as the target acceleration operator, including:

[0020] The first candidate acceleration operator is determined as the target acceleration operator, where the second type of parameter in the attribute description information is configured as a data value and the data value is consistent with the parameter value required by the target inference operation under the second type of parameter.

[0021] Furthermore, the method also includes:

[0022] If the first candidate acceleration operator does not exist, then the second candidate acceleration operator is determined as the target acceleration operator, where the second type of parameter in the attribute description information is configured as a data range and the data range includes the parameter values ​​required by the target inference operation under the second type of parameter.

[0023] Furthermore, the method further includes using special identifiers to indicate data ranges in the attribute description information:

[0024] If the second type of parameter in the attribute description information corresponding to the acceleration operator is a preset identifier, then the second type of parameter configuration of the acceleration operator is determined to be a data range.

[0025] Furthermore, the target inference operation includes a multi-head attention computation operation, where the first type of parameter includes head size and the second type of parameter includes sequence length.

[0026] Furthermore, the query key is a hash key, and the mapping relationship between the identifier of the acceleration operator and the query key is represented by a hash table.

[0027] This application also provides a computing device, including a memory, a processor, and a communication component;

[0028] The memory is used to store one or more computer instructions;

[0029] The processor is coupled to the memory and the communication component and is used to execute one or more computer instructions for performing the aforementioned inference acceleration method.

[0030] This application also provides a computer-readable storage medium for storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the aforementioned inference acceleration method.

[0031] In this embodiment, a method is proposed to determine the acceleration requirement parameter values ​​for each inference operation to be accelerated in the inference model during the inference execution phase, in response to the inference instruction. This allows for real-time monitoring of the dynamic changes in the acceleration requirement parameter values ​​for each inference operation during the inference execution phase. Consequently, suitable acceleration operators can be adaptively matched to each inference operation based on these dynamic changes, ensuring that each inference operation to be accelerated in the inference model can be matched with a suitable acceleration operator in each round of inference. Compared to the traditional approach of matching operators during the inference initialization phase and then fixing them to the inference execution phase, this method enables adaptive matching of acceleration operators, supporting complex and varied inference instructions during the inference execution phase and thus guaranteeing inference acceleration performance in each round of inference. Attached Figure Description

[0032] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0033] Figure 1 A flowchart illustrating an exemplary embodiment of this application for a reasoning acceleration method;

[0034] Figure 2 A logical schematic diagram of an inference acceleration method provided for an exemplary embodiment of this application;

[0035] Figure 3 A schematic diagram illustrating an application scenario provided for an exemplary embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the structure of a computing device provided for another exemplary embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] Inference is the process of making predictions using a trained model. In this paper, the model used for inference is described as an inference model. An inference model can be a neural network model or a machine learning model, etc. With the continuous development of inference requirements, the scale and complexity of inference models are also constantly increasing. To ensure inference efficiency, inference acceleration technology has emerged. However, currently, the matching method for acceleration operators is too rigid, leading to unstable inference acceleration performance. Therefore, in some embodiments of this application, it is proposed that during the inference execution phase, in response to the inference instruction, the corresponding acceleration requirement parameter values ​​for each inference operation to be accelerated in this round of inference are determined. This allows for real-time perception of the dynamic changes in the acceleration requirement parameter values ​​corresponding to each inference operation during the inference execution phase. Consequently, it can adaptively match suitable acceleration operators for each inference operation according to the dynamic changes in the acceleration requirement parameters, ensuring that each inference operation to be accelerated in the inference model can be matched with a suitable acceleration operator in each round of inference. Compared with the traditional approach of matching operators during the inference initialization phase and then fixing them to the inference execution phase, this approach enables adaptive matching of accelerated operators, thereby supporting complex and varied inference instructions during the inference execution phase and ensuring inference acceleration performance in each round of inference.

[0039] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0040] Figure 1 This is a flowchart illustrating an exemplary embodiment of an inference acceleration method provided in this application. The method can be executed by an inference acceleration engine, which can be implemented as software, hardware, or a combination of both. The inference acceleration engine can be integrated into a computing device. (Reference) Figure 1 The method may include:

[0041] Step 100: During the operation of the inference model, in response to the inference instruction, determine the acceleration requirement parameter value corresponding to the target inference operation in this round of inference. The acceleration requirement parameter value corresponding to the target inference operation is not exactly the same in different inference processes.

[0042] Step 101: Find the target acceleration operator that matches the acceleration requirement parameter value;

[0043] Step 102: During this round of inference, the target acceleration operator is invoked for the target inference operation to accelerate the target inference operation;

[0044] The target inference operation is any inference operation in the inference model that needs to be accelerated.

[0045] The inference acceleration method provided in this embodiment can be applied to various scenarios requiring inference acceleration, such as artificial intelligence inference scenarios like natural language processing or image processing. This embodiment does not limit the application scenario. In different application scenarios, the inference functions carried by the inference model can be deployed as needed. For example, in a natural language processing scenario, the inference functions carried by the inference model can be speech recognition, chat functionality, or translation functionality. As another example, in an image processing scenario, the inference functions carried by the inference model can be object detection or image segmentation functionality. This embodiment does not limit the internal model logic of the inference model. Furthermore, this embodiment does not limit the inference mechanism used in the inference model. Typically, the inference model can employ attention mechanisms, self-attention mechanisms, or multi-head attention mechanisms (MHA). Among these, multi-head attention (MHA) is a sequence processing algorithm in the field of artificial intelligence. This algorithm, through a "multi-head" structure, achieves memory management and context association of processed sequences during sequence processing, thereby achieving better sequence understanding results.

[0046] The inference acceleration method provided in this embodiment can be considered an innovative feature within an inference acceleration engine. This embodiment does not limit the deployment location of the inference acceleration engine; it can be deployed on a cloud server or other locations with inference acceleration requirements, such as enterprise data centers. Currently, inference acceleration engines can optimize inference models built on various mainstream artificial intelligence (AI) computing frameworks based on IaaS (Infrastructure as a Service) resources, significantly improving inference performance. The essence of inference acceleration is heterogeneous acceleration, which can be understood as offloading the inference work to hardware resources (also known as heterogeneous acceleration devices) used for inference acceleration, rather than the CPU. This is because such hardware resources have hardware and software attributes more suitable for performing inference work compared to the CPU. In this embodiment, the hardware resources used for inference acceleration may include, but are not limited to, GPUs (Graphics Processing Units) or NPUs (Neural Networks Processing Units), etc., and this embodiment does not limit them.

[0047] In this embodiment, the computational operations performed on each network layer during the execution of the inference model are described as inference operations. These network layers may include, but are not limited to, convolutional layers, pooling layers, etc. The inference model is essentially a neural network model, and the basic knowledge of neural network models will not be elaborated upon here. It is worth noting that this embodiment does not consider all inference operations in the inference model to need acceleration. Instead, this embodiment focuses primarily on inference operations that require acceleration, i.e., those inference operations that need to be executed on the hardware resources used for inference acceleration. Following the various inference mechanisms that the inference model can employ mentioned earlier, the inference operations to be accelerated here mainly include those inference operations used to implement the inference mechanism, such as multi-head attention computation operations, etc., which are not limited in this embodiment.

[0048] In this embodiment, the function running on the hardware resources used for inference acceleration is described as an acceleration operator. For example, for a GPU, the function running on it is a kernel function. Thus, a typical implementation of the acceleration operator in this embodiment can be a kernel. Of course, the acceleration operator in this embodiment can also adopt an implementation adapted to other types of hardware resources, and is not limited to this.

[0049] Figure 2 A logical schematic diagram of an inference acceleration method provided for an exemplary embodiment of this application. (Reference) Figure 2In this embodiment, a variety of different acceleration operators can be provided. The acceleration operators can implement the inference operations in the inference model on the hardware resources used for inference acceleration to accelerate the inference operations. It should be understood that this acceleration is relative to the speed at which the inference operation is executed on the CPU. That is, the speed at which the inference operation is executed on the hardware resources used for inference acceleration is higher than the speed at which the inference operation is executed on the CPU.

[0050] In this embodiment, different acceleration strategies exist in different acceleration operators. Therefore, different acceleration operators are suitable for different acceleration requirements. To address this, this embodiment proposes a novel technical concept for matching acceleration operators to inference operations.

[0051] refer to Figure 1 In step 100, during the execution of the inference model, in response to the inference instruction, the acceleration requirement parameter value corresponding to the target inference operation in this round of inference is determined. The acceleration requirement parameter value indicates the attributes that the acceleration operator required by the target inference operation should possess. In practical applications, the inference instruction may contain an overall acceleration requirement description for the inference model, and the acceleration requirement parameter values ​​corresponding to each inference operation in this round of inference can be determined based on this overall acceleration requirement description information. The relevant logic involved in this process can be pre-set in the inference model; that is, the inference model itself determines the acceleration requirement parameter values ​​corresponding to each of its inference operations in this round of inference. In this case, in step 100, the acceleration requirement parameter value corresponding to the target inference operation in this round of inference can be obtained from the inference model. Of course, in other possible designs, the inference acceleration method provided in this embodiment can also implement the relevant logic involved in the aforementioned process to obtain the acceleration requirement parameter value corresponding to the target inference operation in this round of inference; this embodiment does not limit this.

[0052] During the execution of the inference model, multiple inference instructions can be received. A single inference instruction can trigger one round of inference. In one round of inference, all inference operations contained in the inference model will be executed. For ease of explanation, this embodiment, starting from step 100, uses the target inference operation in the inference model as an example to illustrate the inference acceleration scheme. The target inference operation can be any inference operation to be accelerated contained in the inference model. Furthermore, it should be understood that other inference operations to be accelerated in the inference model can also be accelerated using the same inference acceleration logic, thereby ensuring the acceleration performance in this round of inference.

[0053] This embodiment innovatively proposes that, during the inference model's operation—that is, the inference execution phase—acceleration requirement parameter values ​​are determined for each inference operation to be accelerated in the current inference round, in response to inference instructions. However, as inference needs become increasingly diverse, taking the target inference operation as an example, the acceleration requirement parameter values ​​corresponding to the target inference operation are not entirely the same in different inference processes and may change dynamically.

[0054] For example, if the reasoning function of the inference model is a chat function, the query statement in the current inference instruction might be "What day of the week is it today?", while the query statement in the next inference instruction might be "Please recommend 3 popular restaurants within 5 kilometers". Obviously, the query statements in these two inference instructions are completely different, which will result in different acceleration requirement parameter values ​​for the target inference operation in these two rounds of inference.

[0055] As mentioned earlier, different acceleration operators require different acceleration. Therefore, this embodiment proposes that in step 101, a target acceleration operator matching the acceleration requirement parameter value can be found. Since the acceleration requirement parameter value corresponding to the target inference operation may change dynamically, in each round of inference, a target acceleration operator matching its acceleration requirement parameter value can be found in real time for the target inference operation. This enables adaptive matching between the inference operation and the acceleration operator, ensuring that the acceleration operator matched for the target inference operation in each round of inference is appropriate even when the acceleration requirement parameter value changes dynamically.

[0056] Based on this, refer to Figure 1 In step 102, during the current inference process, a target acceleration operator can be invoked for the target inference operation to execute the target inference operation on the hardware resources used for inference acceleration.

[0057] refer to Figure 1 In step 102, the CPU can call the appropriate acceleration operators for each inference operation to be accelerated in the inference model during the current inference process. These acceleration operators can be used to transfer each inference operation to be accelerated to heterogeneous hardware resources, thereby accelerating the current inference process. In this way, each inference operation to be accelerated can be accelerated by the application in each inference process, which ensures the stability of inference acceleration performance.

[0058] It is worth noting that the inference acceleration method provided in this embodiment is an innovative function in the inference acceleration engine. This innovative function can be integrated and cooperated with other functions in the inference acceleration engine. For example, the inference acceleration engine can merge the inference operations to be accelerated and use the merged inference operations as the target inference operations in this embodiment to reduce the number of times the acceleration operators are started. This embodiment does not limit other acceleration functions in the inference acceleration engine.

[0059] In summary, this embodiment proposes that during the inference execution phase, in response to inference instructions, the corresponding acceleration requirement parameter values ​​for each inference operation to be accelerated in the inference model are determined for each round of inference. This allows for real-time awareness of the dynamic changes in the acceleration requirement parameter values ​​corresponding to each inference operation during the inference execution phase. Consequently, it can adaptively match suitable acceleration operators to each inference operation based on these dynamic changes, ensuring that each inference operation to be accelerated in the inference model can be matched with a suitable acceleration operator in each round of inference. Compared to the traditional approach of matching operators during the inference initialization phase and then fixing them to the inference execution phase, this approach enables adaptive matching of acceleration operators, thereby supporting complex and varied inference instructions during the inference execution phase and ensuring inference acceleration performance in each round of inference.

[0060] In the above or below embodiments, various implementation methods can be used to find the target acceleration operator matched in the current inference process for the target inference operation.

[0061] In one alternative implementation: based on the attribute description information corresponding to each acceleration operator in the acceleration operator library, an acceleration operator with attribute description information that meets the acceleration requirement parameter values ​​can be searched from the acceleration operator library and used as the target acceleration operator.

[0062] In this implementation, the attribute description information of the acceleration operator can be configured when constructing the acceleration operator. This attribute description information can be used to describe the limitations of the acceleration operator's use; that is, it can be used to describe under what acceleration requirements the acceleration operator performs best. This is because different acceleration operators have different acceleration strategies, and the acceleration requirements they support are naturally different. The acceleration requirement parameter values ​​determined for the target inference operation in step 100 precisely reflect the acceleration requirements of the target inference operation in this round of inference. Based on this, in this implementation, a suitable target acceleration operator can be selected for the target inference operation by comparing the acceleration requirement parameter values ​​and the attribute description information.

[0063] In one exemplary matching scheme: a query key can be constructed based on the acceleration requirement parameter value corresponding to the target inference operation. Then, based on the mapping relationship between the acceleration operator's identifier and the query key, the target acceleration operator matching the target inference operation can be found. Specifically, the values ​​of each parameter corresponding to the acceleration requirement parameter value can be extracted in advance from the attribute description information of each acceleration operator, and a corresponding query key can be generated for each acceleration operator. In practical applications, a hash map can be used to implement the aforementioned mapping relationship. Correspondingly, the query key can be a hash key, thus efficiently matching suitable acceleration operators for the target inference operation.

[0064] As speed-up operators have become more abundant and developed, the inventors discovered during their research that some parameters in the attribute description information of speed-up operators are no longer configured as single data values, but may be configured as data ranges. For example, some speed-up operators no longer support only a single sequence length, but can support a general range of sequence lengths. This leads to the problem that the correct speed-up operator cannot be matched when matching according to the aforementioned exemplary matching scheme.

[0065] Therefore, this embodiment proposes an improved matching scheme. In this improved scheme, the acceleration requirement parameter values ​​can include the parameter values ​​required by the target inference operation under a first type of parameter and the parameter values ​​required by the target inference operation under a second type of parameter. The first type of parameter is configured as data values ​​in the attribute description information of each acceleration operator, while the second type of parameter is configured as a data value or a data range in the attribute description information of a single acceleration operator. Based on this, in the improved matching scheme:

[0066] Acceleration operators with attribute description information that meet the parameter values ​​required by the target inference operation under the first type of parameters can be searched from the acceleration operator library and used as candidate acceleration operators.

[0067] From the candidate acceleration operators, acceleration operators with attribute description information that meets the parameter values ​​required by the target inference operation under the second type of parameters can be selected as the target acceleration operators.

[0068] In this improved matching scheme, the matching process is divided into two stages. In the first stage, a portion of acceleration operators can be initially selected from the acceleration operator library based on the first type of parameters as candidate acceleration operators. In the second stage, the most suitable acceleration operator can be selected from the candidate acceleration operators based on the second type of parameters as the target acceleration operator.

[0069] In the first step described above, the matching concept from the aforementioned exemplary matching scheme is retained: a query key is constructed based on the parameter values ​​required by the target inference operation under the first type of parameters; the identifier of the acceleration operator corresponding to the target inference operation is determined based on the mapping relationship between the acceleration operator's identifier and the query key, in order to find candidate acceleration operators. That is, in the first step, only the parameter values ​​required by the target inference operation under the first type of parameters are used as the filtering criteria, thereby allowing the selection of candidate acceleration operators that meet this filtering criterion from the acceleration operator library. Similarly, in the first step, the query key can be a hash key, and the mapping relationship between the acceleration operator's identifier and the query key can be represented using a hash map.

[0070] Regarding the second step mentioned above, the matching between candidate acceleration operators and the target inference operation can be evaluated based on the second type of parameters. Optionally, the first candidate acceleration operator whose second type of parameters in the attribute description information is configured as a data value and whose data value is consistent with the parameter value required by the target inference operation under the second type of parameters can be identified as the target acceleration operator. As mentioned earlier, some acceleration operators may have their second type of parameters configured as a data range in their attribute description information. Therefore, in the second step, the following judgment logic can be added: if no first candidate acceleration operator meets the above requirements, then the second candidate acceleration operator whose second type of parameters in the attribute description information is configured as a data range and whose data range includes the parameter value required by the target inference operation under the second type of parameters can be identified as the target acceleration operator.

[0071] In practical applications, the above two dimensions can be evaluated under a single candidate speedup operator to determine whether the evaluated candidate speedup operator can be used as the target speedup operator.

[0072] Taking any candidate acceleration operator as an example: First, it is determined whether its second type of parameter is configured as a data value and whether the data value is consistent with the parameter value required by the target inference operation under the second type of parameter. If so, it can be directly identified as the target acceleration operator; otherwise, it can be further determined whether its second type of parameter is configured as a data range. If so, it can also be identified as the target acceleration operator. Moreover, to ensure the uniqueness of the target acceleration operator, in this judgment logic, when multiple acceleration operators that meet the judgment logic exist simultaneously, the acceleration operator with the second type of parameter configured as a data value can be selected as the final target acceleration operator. The final optimization mechanism mainly considers that acceleration operators with the second type of parameter configured as a data value have a more focused acceleration strategy and better acceleration performance compared to acceleration operators with the second type of parameter configured as a data range.

[0073] Furthermore, to better support the two matching stages mentioned above, the improved matching scheme can employ a two-layer storage approach to store attribute description information. The first type of parameters in the attribute description information can be stored using the aforementioned mapping relationship, while the second type of parameters can be stored separately. In practical applications, a two-layer storage structure of a hashmap and an array can be used. The hashmap can be used to store the mapping relationship between the identifier of the acceleration operator and the query key, while the array can be used to store the second type of parameters for each acceleration operator.

[0074] Furthermore, in the improved matching scheme, the attribute description information can also use a preset identifier to refer to the data range. That is, when the second type of parameter of the acceleration operator is configured as a data range, the second type of parameter in the attribute description information of the acceleration operator can be recorded as a preset identifier. In addition, in practical applications, the data range is usually a general range, that is, it can cover any parameter value under the second type of parameter. Based on this, in the judgment logic of the aforementioned second step, if the second type of parameter in the attribute description information corresponding to the candidate acceleration operator is a preset identifier and there are no other candidate acceleration operators whose second type of parameter is configured with a data value consistent with the parameter value required by the target inference operation under the second type of parameter, the candidate acceleration operator can be determined as the target acceleration operator. Here, the second type of parameter in the attribute description information corresponding to the candidate acceleration operator is a preset identifier, which means that the second type of parameter of the candidate acceleration operator is configured as a data range.

[0075] In this way, by using the preset identifier, there is no need to analyze the data range, which can greatly reduce the amount of computation consumed by the judgment logic.

[0076] In summary, this embodiment redesigns the representation of the second type of parameters in the attribute description information of the acceleration operator, improves the storage structure of the attribute description information, and improves the matching process of the acceleration operator, thereby enabling the representation, storage, and optimization of more types of acceleration operators, thus improving the inference acceleration performance of the inference model.

[0077] Figure 3 This is a schematic diagram illustrating an application scenario provided for an exemplary embodiment of this application. (Reference) Figure 3 The inference model uses the MHA mechanism. During the operation of the inference model, an inference instruction is received, which specifies the overall head size (headsize = 160) and the sequence length (seqlength = 1024) of the input query statement. This inference instruction can trigger the current round of inference.

[0078] The inference model may contain multiple MHA operations. Taking operation A as an example, the acceleration requirement parameter values ​​for operation A in this round of inference are determined to be headsize=80 and seqlength=1024. It should be understood that the acceleration requirement parameter values ​​for operation A in the previous or next round of inference may be different.

[0079] Based on this, headsize=80 can be stored in the hash key, and a lookup can be performed in the hashmap using the hash key (the hashmap contains a mapping between accelerator operator IDs and hash keys, and the hash keys in the hashmap are pre-constructed based on the headsize supported by each accelerator operator). Hash lookups are very fast; therefore, in this matching process, candidate accelerator operators supporting headsize=80 can be quickly selected from the accelerator operator library for operation A. (Reference) Figure 3 After this matching process, there may be three candidate acceleration operators: acceleration operator a, acceleration operator b, and acceleration operator c.

[0080] After the first matching step is completed, the second matching step is performed using the acceleration parameter value of seqlength=1024, as described above. Figure 3 The acceleration operator 'a' supports a seqlength of 512, acceleration operator 'b' supports a seqlength of 1024, and acceleration operator 'c' supports a seqlength of [128, 1024]. This is a general sequence size. Based on this, in the second matching stage, the following judgment logic can be performed on each candidate acceleration operator: First, determine whether its seqlength is a data value and whether the data value is equal to 1024. If so, it can be directly determined as the target acceleration operator; if not, it can be further determined whether its seqlength is a preset identifier. If so, it can also be determined as the target acceleration operator. Moreover, if multiple acceleration operators that meet the judgment logic exist simultaneously, the acceleration operator whose seqlength is configured as a data value is selected as the final target acceleration operator.

[0081] Based on the above judgment logic, the acceleration operator b can be selected as the target acceleration operator matched by operation A in this round of inference.

[0082] If the seqlength corresponding to operation A is 128, then after the above judgment logic, the acceleration operator c will be selected as the target acceleration operator matched by operation A in this round of inference.

[0083] As can be seen, the inference acceleration method provided in this embodiment can adaptively match appropriate acceleration operators to each inference operation to be accelerated in the inference model during the inference execution stage, thereby supporting the acceleration requirements of dynamically changing inference operations during the inference execution stage and ensuring that sufficiently good inference acceleration performance can be obtained in each round of inference.

[0084] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in this document, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should also be noted that the descriptions such as "first" and "second" in this document are used to distinguish different parameter types, candidate operators, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0085] Figure 4 This is a schematic diagram of the structure of a computing device provided for another exemplary embodiment of this application. For example... Figure 4 As shown, the computing device includes: a memory 40, a processor 41, and a communication component 42.

[0086] Processor 41, coupled to memory 40, is used to execute computer programs in memory 40 for:

[0087] During the execution of the inference model, in response to inference instructions, the acceleration requirement parameter values ​​corresponding to the target inference operation in this round of inference are determined;

[0088] Find a target acceleration operator that matches the acceleration requirement parameter value;

[0089] During this round of inference, the target acceleration operator is invoked for the target inference operation to accelerate the target inference operation;

[0090] The target inference operation is any inference operation to be accelerated contained in the inference model.

[0091] In an optional embodiment, when the processor 41 searches for a target acceleration operator that matches the acceleration requirement parameter value, it may specifically be used to:

[0092] Based on the attribute description information corresponding to each acceleration operator in the acceleration operator library, an acceleration operator with attribute description information that meets the acceleration requirement parameter value is searched from the acceleration operator library and used as the target acceleration operator.

[0093] In an optional embodiment, the acceleration requirement parameter value includes the parameter value required by the target inference operation under a first type of parameter and the parameter value required by the target inference operation under a second type of parameter. The first type of parameter is configured as a data value in the attribute description information of each acceleration operator, and the second type of parameter is configured as a data value or a data range in the attribute description information of a single acceleration operator. When the processor 41 searches for an acceleration operator with attribute description information that matches the acceleration requirement parameter value from the acceleration operator library as the target acceleration operator, it can specifically be used for:

[0094] From the acceleration operator library, find acceleration operators with attribute description information that meet the parameter values ​​required by the target inference operation under the first type of parameters, and use them as candidate acceleration operators;

[0095] From the candidate acceleration operators, find the acceleration operator that has attribute description information that meets the parameter values ​​required by the target inference operation under the second type of parameters, and use it as the target acceleration operator.

[0096] In an optional embodiment, when the processor 41 searches the acceleration operator library for acceleration operators with attribute description information that matches the parameter values ​​required by the target inference operation under the first type of parameters, as candidate acceleration operators, it can specifically be used for:

[0097] Based on the parameter values ​​required by the target inference operation under the first type of parameters, construct the query key value corresponding to the target inference operation;

[0098] Based on the mapping relationship between the identifier of the acceleration operator and the query key value, the identifier of the acceleration operator corresponding to the target inference operation is determined in order to find the candidate acceleration operator.

[0099] In an optional embodiment, when the processor 41 searches for an acceleration operator with attribute description information that meets the parameter values ​​required by the target inference operation under the second type of parameters from the candidate acceleration operators, it can specifically be used as the target acceleration operator:

[0100] The first candidate acceleration operator is determined as the target acceleration operator, where the second type of parameter in the attribute description information is configured as a data value and the data value is consistent with the parameter value required by the target inference operation under the second type of parameter.

[0101] In an alternative embodiment, processor 41 may also be used for:

[0102] If the first candidate acceleration operator does not exist, then the second candidate acceleration operator is determined as the target acceleration operator, where the second type of parameter in the attribute description information is configured as a data range and the data range includes the parameter values ​​required by the target inference operation under the second type of parameter.

[0103] In an optional embodiment, a preset identifier is used to refer to the data range in the attribute description information, and the processor 41 can also be used for:

[0104] If the second type of parameter in the attribute description information corresponding to the first candidate acceleration operator is a preset identifier, then the second type of parameter of the first candidate acceleration operator is determined to be configured as a data range.

[0105] In an optional embodiment, the target inference operation includes a multi-head attention computation operation, wherein the first type of parameter includes head size and the second type of parameter includes sequence length (seq-length).

[0106] In an optional embodiment, the query key is a hash key, and the mapping relationship between the identifier of the acceleration operator and the query key is represented by a hash map.

[0107] Furthermore, such as Figure 4 As shown, the computing device also includes other components such as a power supply component 43. Figure 4 The diagram only shows some components and does not mean that the computing device includes only these components. Figure 4 The components shown.

[0108] It is worth noting that the technical details of the above embodiments of the computing device can be referred to the relevant descriptions in the foregoing method embodiments. To save space, they will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0109] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can perform the steps that can be executed by a computing device in the above method embodiments.

[0110] The above Figure 4 The memory in a computer is used to store computer programs and can be configured to store various other data to support operation on a computing platform. Examples of this data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disks, or optical disks.

[0111] The above Figure 4The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0112] The above Figure 4 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0119] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for accelerating inference, comprising: During the execution of the inference model, in response to inference instructions, the acceleration requirement parameter values ​​corresponding to the target inference operation in this round of inference are determined; The acceleration requirement parameter values ​​include the parameter values ​​required by the target inference operation under the first type of parameters and the parameter values ​​required by the target inference operation under the second type of parameters. The first type of parameters are configured as data values ​​in the attribute description information of each acceleration operator, and the second type of parameters are configured as data values ​​or data ranges in the attribute description information of a single acceleration operator. From the acceleration operator library, find acceleration operators with attribute description information that meet the parameter values ​​required by the target inference operation under the first type of parameters, and use them as candidate acceleration operators; From the candidate acceleration operators, find the acceleration operator that has attribute description information that meets the parameter values ​​required by the target inference operation under the second type of parameters, and use it as the target acceleration operator; During this round of inference, the target acceleration operator is invoked for the target inference operation to accelerate the target inference operation; The target inference operation is any inference operation to be accelerated contained in the inference model.

2. The method according to claim 1, wherein searching the acceleration operator library for an acceleration operator with attribute description information that conforms to the parameter values ​​required by the target inference operation under the first type of parameters, as a candidate acceleration operator, includes: Construct the query key value corresponding to the target inference operation according to the parameter values ​​required by the target inference operation under the first type of parameters; Based on the mapping relationship between the identifier of the acceleration operator and the query key value, the candidate acceleration operator corresponding to the target inference operation is determined.

3. The method according to claim 1, wherein searching from the candidate acceleration operators for an acceleration operator having attribute description information that conforms to the parameter values ​​required by the target inference operation under the second type of parameters, and using it as the target acceleration operator, comprises: The first candidate acceleration operator is determined to be the target acceleration operator, where the second type of parameter in the attribute description information is configured as a data value and the data value is consistent with the parameter value required by the target inference operation under the second type of parameter.

4. The method according to claim 3, further comprising: If the first candidate acceleration operator does not exist, then the second candidate acceleration operator is determined as the target acceleration operator, where the second type of parameter in the attribute description information is configured as a data range and the data range includes the parameter values ​​required by the target inference operation under the second type of parameter.

5. The method according to claim 4, wherein a preset identifier is used to refer to the data range in the attribute description information, the method further comprising: If the second type of parameter in the attribute description information corresponding to the acceleration operator is a preset identifier, then the second type of parameter configuration of the acceleration operator is determined to be a data range.

6. The method according to claim 1, wherein the target inference operation includes a multi-head attention computation operation, the first type of parameter includes head size, and the second type of parameter includes sequence length.

7. The method according to claim 2, wherein the query key is a hash key, and the mapping relationship between the identifier of the acceleration operator and the query key is represented by a hash table.

8. A computing device, comprising a memory, a processor, and communication components; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component and is used to execute one or more computer instructions for performing the inference acceleration method according to any one of claims 1-7.

9. A computer-readable storage medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the inference acceleration method according to any one of claims 1-7.

Images