Inference acceleration method, device and storage medium

By matching the acceleration operator in real time during the inference execution stage, the problem of rigid matching methods of the acceleration operator in the existing technology is solved, and the stable and efficient acceleration performance of the inference model is achieved.

WO2024239868A1PCT designated stage expired Publication Date: 2024-11-28CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD

Patent Information

Application Number
PCT/CN2024/088678
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-22
Filing Date
2024-04-18
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The acceleration operator matching method in the existing inference model is too rigid, resulting in unstable inference acceleration performance and cannot adapt to the diversification and dynamic changes in inference requirements.

Method used

Responding to inference instructions during the inference execution stage, determining the acceleration requirement parameter value in real time, and adapting to match the appropriate acceleration operators. By finding the attribute description information in the acceleration operator library, dynamically matching to ensure that each round of inference operations matches the appropriate acceleration Operator.

Benefits of technology

It realizes the stability and flexibility of inference acceleration performance, supports complex and changeable inference instructions, and ensures efficient acceleration in each round of inference process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088678_28112024_PF_FP_ABST
    Figure CN2024088678_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present application are an inference acceleration method, a device and a storage medium. The method comprises: during an inference execution stage, in response to an inference instruction, separately determining in a current round of inference an acceleration demand parameter value corresponding to each inference operation to be accelerated in an inference model, so as to be aware in real time of dynamic changes of the acceleration demand parameter value corresponding to each inference operation during the inference execution stage, thereby adaptively matching a proper acceleration operator to each inference operation following the dynamic changes of the acceleration demand parameters, and ensuring that each said inference operation in the inference model can be matched to a proper acceleration operator in each round of inference. Compared with conventional solutions of performing operator matching during an inference initialization stage and securing same to an inference execution stage, the present application achieves adaptive matching of acceleration operators, so as to support complex and changeable inference instructions during inference execution stages, thereby ensuring the inference acceleration performance in each round of inference.
Need to check novelty before this filing date? Find Prior Art

Description

Inference acceleration method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 22, 2023, with application number 202310580983.X and invention name “A method, device and storage medium for inference acceleration”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of AI technology, and in particular to an inference acceleration method, device, and storage medium. Background Art

[0003] With the development of fields such as natural language processing and image processing, the MHA (Multi-Head Attention) mechanism has been widely used in various reasoning models, especially reasoning models similar to the BERT structure.

[0004] Currently, kernels (kernel functions on the GPU) are typically matched for each MHA in the inference model during the inference initialization phase. The resulting matching results are then solidified and applied during the inference execution phase. That is, during the inference execution phase, the corresponding kernel is called for each MHA based on the solidified matching results to achieve inference acceleration.

[0005] Inference requirements are constantly diversifying, but the aforementioned kernel matching method is too rigid, often calling inappropriate kernels, resulting in unstable inference acceleration performance.

[0006] Summary of the Invention

[0007] Various aspects of the present application provide an inference acceleration method, device, and storage medium to improve inference acceleration performance.

[0008] This embodiment of the present application provides an inference acceleration method, including:

[0009] During the execution of the inference model, in response to the inference instruction, determining the acceleration requirement parameter value corresponding to the target inference operation in the current round of inference process;

[0010] Searching for a target acceleration operator that matches the acceleration requirement parameter value;

[0011] In this round of reasoning, calling the target acceleration operator for the target reasoning operation to accelerate the target reasoning operation;

[0012] The target reasoning operation is any reasoning operation to be accelerated contained in the reasoning model.

[0013] Further, searching for a target acceleration operator that matches the acceleration requirement parameter value includes:

[0014] Based on the attribute description information corresponding to each acceleration operator in the acceleration operator library, an acceleration operator having the attribute description information that meets the acceleration requirement parameter value is searched from the acceleration operator library as the target acceleration operator.

[0015] Furthermore, the acceleration requirement parameter value includes a parameter value required by the target reasoning operation under first-category parameters and a parameter value required by the target reasoning operation under second-category parameters, the first-category parameters are configured as data values ​​in the attribute description information of each acceleration operator, and the second-category parameters are configured as data values ​​or data ranges in the attribute description information of a single acceleration operator; searching the acceleration operator library for an acceleration operator having attribute description information that meets the acceleration requirement parameter value as the target acceleration operator includes:

[0016] Searching, from the acceleration operator library, an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the first type of parameters as a candidate acceleration operator;

[0017] From the candidate acceleration operators, an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the second type of parameters is searched as the target acceleration operator.

[0018] Furthermore, searching the acceleration operator library for an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the first type of parameters as a candidate acceleration operator includes:

[0019] Constructing a query key value corresponding to the target reasoning operation according to the parameter values ​​required by the target reasoning operation under the first type of parameters;

[0020] Based on the mapping relationship between the identifier of the acceleration operator and the query key value, a candidate acceleration operator corresponding to the target reasoning operation is determined.

[0021] Furthermore, searching, from the candidate acceleration operators, for an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the second type of parameters as the target acceleration operator, includes:

[0022] Determine a first candidate acceleration operator whose second-category parameter configuration in the attribute description information is a data value and the data value is consistent with a parameter value required by the target reasoning operation under the second-category parameter, as the target acceleration operator.

[0023] Furthermore, the method further comprises:

[0024] If the first candidate acceleration operator does not exist, a second candidate acceleration operator whose second type of parameter configuration in the attribute description information is a data range and the data range includes the parameter value required by the target inference operation under the second type of parameter is determined as the target acceleration operator.

[0025] Furthermore, a special identifier is used in the attribute description information to indicate the data range, and the method further includes:

[0026] If the second type of parameter in the attribute description information corresponding to the acceleration operator is a preset identifier, it is determined that the second type of parameter configuration of the acceleration operator is a data range.

[0027] Furthermore, the target inference operation includes a multi-head attention calculation operation, the first type of parameters includes head size, and the second type of parameters includes sequence length.

[0028] Furthermore, the query key value adopts a hash key value, and the mapping relationship between the identifier of the acceleration operator and the query key value key is represented by a hash table.

[0029] An embodiment of the present application further provides a computing device, including a memory, a processor, and a communication component;

[0030] The memory is used to store one or more computer instructions;

[0031] The processor is coupled to the memory and the communication component, and is configured to execute the one or more computer instructions to perform the aforementioned inference acceleration method.

[0032] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the aforementioned inference acceleration method.

[0033] In an embodiment of the present application, it is proposed that during the inference execution phase, the acceleration requirement parameter values ​​corresponding to each inference operation to be accelerated in the inference model are determined in response to the inference instructions. In this way, the dynamic changes of the acceleration requirement parameter values ​​corresponding to each inference operation during the inference execution phase can be perceived in real time, so that the appropriate acceleration operator can be adaptively matched to each inference operation following the dynamic changes of the acceleration requirement parameters, ensuring that each inference operation to be accelerated in the inference model can be matched to an appropriate acceleration operator during each round of inference. Compared with the traditional solution of performing operator matching in the inference initialization phase and solidifying it in the inference execution phase, this can achieve adaptive matching of acceleration operators, thereby supporting complex and changeable inference instructions in the inference execution phase, and thus ensuring the inference acceleration performance during each round of inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] FIG1 is a flow chart of an inference acceleration method provided by an exemplary embodiment of the present application;

[0036] FIG2 is a logical diagram of an inference acceleration method provided by an exemplary embodiment of the present application;

[0037] FIG3 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present application;

[0038] FIG4 is a schematic structural diagram of a computing device provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0039] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] Inference is the process of making predictions using a trained model. Herein, the model used for inference is described as an inference model. An inference model can be a neural network model or a machine learning model, among others. As inference demands continue to grow, the scale and complexity of inference models are also increasing. To ensure inference efficiency, inference acceleration technology has emerged. However, currently, the matching method for acceleration operators is too rigid, resulting in unstable inference acceleration performance. To this end, in some embodiments of the present application, it is proposed that during the inference execution phase, in response to inference instructions, the corresponding acceleration requirement parameter value for each inference operation to be accelerated in the inference model is determined for the current round of inference. This allows for real-time perception of the dynamic changes in the acceleration requirement parameter value corresponding to each inference operation during the inference execution phase. This allows for adaptive matching of the appropriate acceleration operator to each inference operation based on the dynamic changes in the acceleration requirement parameter, ensuring that each inference operation to be accelerated in the inference model is matched to the appropriate acceleration operator during each round of inference. Compared with the traditional solution of matching operators in the inference initialization phase and solidifying them in the inference execution phase, this method can achieve adaptive matching of acceleration operators, thereby supporting complex and changeable inference instructions in the inference execution phase, and thus ensuring the inference acceleration performance in each round of inference.

[0041] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0042] Figure 1 is a flow chart of an inference acceleration method provided by an exemplary embodiment of the present application. The method can be executed by an inference acceleration engine. The inference acceleration engine can be implemented as software, hardware, or a combination of software and hardware. The inference acceleration engine can be integrated into a computing device. Referring to Figure 1, the method may include:

[0043] Step 100: During the execution of the inference model, in response to the inference instruction, determine the acceleration requirement parameter value corresponding to the target inference operation in the current inference process, where the acceleration requirement parameter values ​​corresponding to the target inference operation in different inference processes are not exactly the same;

[0044] Step 101: Find a target acceleration operator that matches the acceleration requirement parameter value;

[0045] Step 102: During this round of reasoning, a target acceleration operator is called for the target reasoning operation to accelerate the target reasoning operation.

[0046] The target reasoning operation is any reasoning operation to be accelerated contained in the reasoning model.

[0047] The inference acceleration method provided in this embodiment can be applied to various scenarios requiring inference acceleration, such as artificial intelligence inference scenarios such as natural language processing or image processing. This embodiment does not limit the application scenarios. In different application scenarios, the inference functions carried by the inference model can be deployed as needed. For example, in a natural language processing scenario, the inference functions carried by the inference model may include speech recognition, chat, or translation. For another example, in an image processing scenario, the inference functions carried by the inference model may include object detection or image segmentation. This embodiment does not limit the internal model logic of the inference model. In addition, this embodiment does not limit the inference mechanism used in the inference model. Typically, the inference model may adopt an attention mechanism, a self-attention mechanism, or a multi-head self-attention mechanism (MHA). The multi-head attention mechanism (MHA) is a sequence processing algorithm in the field of artificial intelligence. This algorithm uses a "multi-head" structure to implement memory management and context association of processed sequences during the sequence processing process, thereby achieving better sequence understanding effects.

[0048] The reasoning acceleration method provided in this embodiment can be used as an innovative function in the reasoning acceleration engine. However, the deployment location of the reasoning acceleration engine is not limited in this embodiment. The reasoning acceleration engine can be deployed on a cloud server, and of course it can also be deployed in other locations where there is a need for reasoning acceleration, such as an enterprise computer room. At present, the reasoning acceleration engine can optimize the reasoning models built by various mainstream artificial intelligence (AI) computing frameworks based on IaaS (Infrastructure as a Service) resources, which can significantly improve the reasoning performance. The essence of reasoning acceleration is heterogeneous acceleration, which can be understood as handing over the reasoning work to the hardware resources used for reasoning acceleration (also known as heterogeneous acceleration devices) instead of the CPU. This is because such hardware resources have hardware and software properties that are more suitable for performing reasoning work compared to the CPU. In this embodiment, the hardware resources used for reasoning acceleration may include but are not limited to GPUs (Graphics Processing Units) or NPUs (Neural Networks Process Units), etc., and this embodiment does not limit this.

[0049] In this embodiment, the calculation operations on each network layer during the operation of the inference model are described as inference operations. The network layers here may include but are not limited to convolutional layers, pooling layers, etc. The essence of the inference model is a neural network model, and the basic knowledge of the neural network model will not be explained in detail here. It is worth noting that this embodiment believes that not all inference operations in the inference model need to be accelerated, and this embodiment focuses mainly on the inference operations to be accelerated, that is, those inference operations that need to be performed on hardware resources used for inference acceleration. Following the various types of inference mechanisms that can be adopted by the inference model in the previous article, the inference operations to be accelerated here can mainly include those inference operations used to implement the inference mechanism, such as multi-head attention calculation operations, etc. This embodiment does not limit this.

[0050] In this embodiment, the functions running on the hardware resources used for inference acceleration are described as acceleration operators. For example, for a GPU, the functions running on it are kernel functions. Thus, a typical implementation form of the acceleration operator in this embodiment can be a kernel. Of course, the acceleration operator in this embodiment can also be implemented in a form compatible with other types of hardware resources, and is not limited to this.

[0051] FIG2 is a logical diagram of an inference acceleration method provided by an exemplary embodiment of the present application. Referring to FIG2 , in this embodiment, a variety of different acceleration operators can be provided. These acceleration operators can implement inference operations in an inference model on hardware resources used for inference acceleration to accelerate the inference operations. It should be understood that this acceleration is relative to the speed at which the inference operations are executed on the CPU. That is, the speed at which the inference operations are executed on the hardware resources used for inference acceleration is higher than the speed at which the inference operations are executed on the CPU.

[0052] In this embodiment, different acceleration operators have different acceleration strategies. Therefore, different acceleration operators are applicable to different acceleration requirements. Therefore, this embodiment proposes a new technical concept for matching acceleration operators for inference operations.

[0053] Referring to FIG. 1 , in step 100 , during the execution of the inference model, in response to an inference instruction, the acceleration requirement parameter value corresponding to the target inference operation during the current inference round can be determined. The acceleration requirement parameter value indicates the properties that the acceleration operator required for the target inference operation should possess. In practical applications, the inference instruction can include an overall acceleration requirement description for the inference model, and the acceleration requirement parameter values ​​corresponding to each inference operation in the inference model during the current inference round can be determined based on this overall acceleration requirement description. The relevant logic involved in this process can be pre-set in the inference model. That is, the inference model itself determines the acceleration requirement parameter values ​​corresponding to each inference operation it contains during the current inference round. In this case, in step 100 , the acceleration requirement parameter value corresponding to the target inference operation during the current inference round can be obtained from the inference model. Of course, in other possible designs, the inference acceleration method provided in this embodiment can also implement the relevant logic involved in the aforementioned process to obtain the acceleration requirement parameter value corresponding to the target inference operation during the current inference round, and this embodiment is not limited to this.

[0054] During the operation of the inference model, multiple inference instructions may be received, and a single inference instruction may trigger a round of inference process. In a round of inference process, all inference operations contained in the inference model will be triggered for execution. In order to facilitate the description of the technical solution, in this embodiment, starting from step 100, the inference acceleration scheme is described using the target inference operation in the inference model as an example, wherein the target inference operation can be any inference operation to be accelerated contained in the inference model. In addition, it should be understood that other inference operations to be accelerated in the inference model can also be accelerated according to the same inference acceleration logic, thereby ensuring the acceleration performance in this round of inference process.

[0055] This embodiment pioneers the idea of ​​determining, during the inference model's execution phase, corresponding acceleration requirement parameter values ​​for each inference operation to be accelerated within the inference model, in response to inference instructions. However, as inference needs continue to diversify, for example, the corresponding acceleration requirement parameter values ​​for target inference operations may vary across different inference processes and may change dynamically.

[0056] For example, if the inference model's reasoning function is a chat function, the query statement in this inference instruction may be "What day is it today?", while the query statement in the next inference instruction may be "Please recommend three popular restaurants within 5 kilometers." Obviously, the query statements in these two inference instructions are completely different, which will result in different acceleration requirement parameter values ​​corresponding to the target inference operation in these two rounds of inference processes.

[0057] As mentioned above, different accelerators use different acceleration requirements. Therefore, this embodiment proposes that in step 101, a target accelerator operator that matches the acceleration requirement parameter value can be searched. Because the acceleration requirement parameter value corresponding to the target inference operation may change dynamically, searching for a target accelerator operator that matches the acceleration requirement parameter value for the target inference operation in real time during each round of inference can achieve adaptive matching between the inference operation and the accelerator operator. In the case of dynamic changes in the acceleration requirement parameter value, the accelerator operator matched to the target inference operation in each round of inference is guaranteed to be appropriate.

[0058] On this basis, referring to FIG1 , in step 102 , a target acceleration operator may be called for a target reasoning operation during this round of reasoning, so as to execute the target reasoning operation on hardware resources used for reasoning acceleration.

[0059] Referring to Figure 2, in step 102, the CPU can call the appropriate acceleration operator matched for each inference operation to be accelerated in the inference model during this round of inference, so that these acceleration operators can be used to transfer each inference operation to be accelerated to heterogeneous hardware resources for execution, thereby achieving acceleration of this round of inference process. In this way, in each round of inference, each inference operation to be accelerated can be accelerated with application, which ensures the stability of inference acceleration performance.

[0060] It is worth noting that the inference acceleration method provided in this embodiment is an innovative function in the inference acceleration engine. This innovative function can be integrated and coordinated with other functions in the inference acceleration engine. For example, the inference acceleration engine can fuse the inference operations to be accelerated, and use the fused inference operations as the target inference operations in this embodiment to reduce the number of times the acceleration operator is started, etc. This embodiment does not limit other acceleration functions in the inference acceleration engine.

[0061] In summary, this embodiment proposes that, during the inference execution phase, the acceleration requirement parameter values ​​corresponding to each inference operation to be accelerated in the inference model are determined in response to inference instructions. This allows for real-time perception of the dynamic changes in the acceleration requirement parameter values ​​corresponding to each inference operation during the inference execution phase. This allows for adaptive matching of appropriate acceleration operators to each inference operation in response to the dynamic changes in the acceleration requirement parameters, ensuring that each inference operation to be accelerated in the inference model is matched with an appropriate acceleration operator during each inference round. Compared to traditional schemes that perform operator matching during the inference initialization phase and solidify it in the inference execution phase, this method enables adaptive matching of acceleration operators, thereby supporting complex and variable inference instructions during the inference execution phase and ensuring inference acceleration performance during each inference round.

[0062] In the above or following embodiments, various implementations may be used to search for a target acceleration operator that matches the target reasoning operation in the current round of reasoning.

[0063] In an optional implementation, based on the attribute description information corresponding to each acceleration operator in the acceleration operator library, an acceleration operator having attribute description information that meets the acceleration requirement parameter value can be searched from the acceleration operator library as a target acceleration operator.

[0064] In this implementation, the attribute description information of the acceleration operator can be configured when the acceleration operator is constructed. The attribute description information can be used to describe the usage restrictions of the acceleration operator, that is, it can be used to describe under what kind of acceleration requirements the acceleration operator will have a better acceleration effect. This is because the acceleration strategies in different acceleration operators are different, and the acceleration requirements they support are naturally different. The acceleration requirement parameter value determined for the target reasoning operation in the aforementioned step 100 can just reflect the acceleration requirement of the target reasoning operation in this round of reasoning. Based on this, in this implementation, the appropriate target acceleration operator can be selected for the target reasoning operation by comparing the acceleration requirement parameter value and the attribute description information.

[0065] In an exemplary matching scheme: a query key value can be constructed based on the acceleration requirement parameter value corresponding to the target reasoning operation, and based on the mapping relationship between the identifier of the acceleration operator and the query key value, the target acceleration operator that matches the target reasoning operation can be found. Among them, the values ​​under each parameter corresponding to the acceleration requirement parameter value can be extracted in advance from the attribute description information of each acceleration operator, and a corresponding query key value can be generated for each acceleration operator. In practical applications, a hash table hashmap can be used to implement the aforementioned mapping relationship. Accordingly, the query key value can use a hash key value hashkey, so that the target reasoning operation can be efficiently matched to a suitable acceleration operator.

[0066] As accelerators continue to expand and develop, the applicant discovered during research that some parameters in the attribute descriptions of accelerators are no longer configured as single data values, but may instead be configured as data ranges. For example, some accelerators no longer support only a single sequence length, but instead support a general range of sequence lengths. This can lead to the problem of not matching the correct accelerator when performing matching according to the aforementioned exemplary matching scheme.

[0067] To this end, this embodiment proposes an improved matching solution. In this improved matching solution, it is proposed that the acceleration requirement parameter values ​​may include the parameter values ​​required by the target reasoning operation under the first type of parameters and the parameter values ​​required by the target reasoning operation under the second type of parameters. The first type of parameters are configured as data values ​​in the attribute description information of each acceleration operator, and the second type of parameters are configured as data values ​​or data ranges in the attribute description information of a single acceleration operator. Based on this, in the improved matching solution:

[0068] An acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the first type of parameters can be searched from the acceleration operator library as a candidate acceleration operator;

[0069] From the candidate acceleration operators, an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the second type of parameters can be searched as the target acceleration operator.

[0070] In this improved matching scheme, the matching process is divided into two steps. In the first step, a part of the acceleration operators can be preliminarily selected from the acceleration operator library based on the first type of parameters as candidate acceleration operators; in the second step, the most suitable acceleration operator can be screened out from the candidate acceleration operators based on the second type of parameters as the target acceleration operator.

[0071] Among them, for the first link mentioned above, the matching concept in the aforementioned exemplary matching scheme is also used: according to the parameter values ​​required for the target reasoning operation under the first type of parameters, the query key value corresponding to the target reasoning operation is constructed; based on the mapping relationship between the identifier of the acceleration operator and the query key value key, the identifier of the acceleration operator corresponding to the target reasoning operation is determined to find the candidate acceleration operator. That is, in the first link, only the parameter values ​​required for the target reasoning operation under the first type of parameters are used as the screening basis, so that the candidate acceleration operators that meet the screening basis can be screened out from the acceleration operator library. Similarly, in the first link, the query key value key can adopt the hash key value hashkey, and the mapping relationship between the identifier of the acceleration operator and the query key value key can be represented by the hash table hashmap.

[0072] As for the second link mentioned above, the matching of the candidate acceleration operator and the target reasoning operation can be evaluated based on the second type of parameters. Optionally, the first candidate acceleration operator whose second type of parameters in the attribute description information are configured as data values ​​and the data values ​​are consistent with the parameter values ​​required for the target reasoning operation under the second type of parameters can be determined as the target acceleration operator. As mentioned above, there may be some acceleration operators whose attribute description information has the second type of parameters configured as data ranges. For this reason, in the second link, the following judgment logic can be added: If there is no first candidate acceleration operator that meets the above requirements, then the second candidate acceleration operator whose second type of parameters in the attribute description information are configured as data ranges and the data range includes the parameter values ​​required for the target reasoning operation under the second type of parameters is determined as the target acceleration operator.

[0073] In practical applications, the above two dimensions can be evaluated for a single candidate accelerator to determine whether the evaluated candidate accelerator can be used as a target accelerator.

[0074] Take any candidate acceleration operator as an example: first determine whether the second-category parameters under it are configured as data values ​​and the data values ​​are consistent with the parameter values ​​required by the target inference operation under the second-category parameters. If so, it can be directly determined as the target acceleration operator; if not, continue to determine whether the second-category parameters under it are configured as data ranges. If so, it can also be determined as the target acceleration operator. In order to ensure the uniqueness of the target acceleration operator, in this judgment logic, when there are multiple acceleration operators that meet the judgment logic at the same time, the acceleration operator with the second-category parameters configured as data values ​​can be selected from them as the final target acceleration operator. The final optimization mechanism is mainly based on the fact that the acceleration operator with the second-category parameters configured as data values ​​has a more focused acceleration strategy and better acceleration performance than the acceleration operator with the second-category parameters configured as data ranges.

[0075] Furthermore, to better support the two matching steps described above, the improved matching solution can also employ a two-tiered storage approach to store attribute description information. The first-category parameters in the attribute description information can be stored using the aforementioned mapping relationship, while the second-category parameters can be stored separately. In practical applications, a two-tiered storage structure consisting of a hashmap and an array can be employed. The hashmap can be used to store the mapping relationship between the accelerator operator identifier and the query key, while the array can be used to store the second-category parameters of each accelerator operator.

[0076] Furthermore, in the improved matching scheme, a preset identifier can also be used in the attribute description information to refer to the data range. That is, when the second-category parameter of the acceleration operator is configured as a data range, the second-category parameter in the attribute description information of the acceleration operator can be recorded as a preset identifier. In addition, in actual applications, the data range is usually a universal range, that is, it can cover any parameter value under the second-category parameter. Based on this, in the judgment logic of the aforementioned second link, if the second-category parameter in the attribute description information corresponding to the candidate acceleration operator is a preset identifier and there is no other candidate acceleration operator whose second-category parameter is configured as a data value consistent with the parameter value required by the target reasoning operation under the second-category parameter, the candidate acceleration operator can be determined as the target acceleration operator. Among them, the second-category parameter in the attribute description information corresponding to the candidate acceleration operator is a preset identifier, which means that the second-category parameter of the candidate acceleration operator is configured as a data range.

[0077] In this way, through the reference of the preset identifier, there is no need to analyze the data range, which can greatly reduce the amount of computing consumed by the judgment logic.

[0078] In summary, in this embodiment, the representation method of the second type of parameters in the attribute description information of the acceleration operator is redesigned, the storage structure of the attribute description information is improved, and the matching process of the acceleration operator is improved, so that more types of acceleration operators can be represented, stored and optimized, thereby improving the inference acceleration performance of the inference model.

[0079] Figure 3 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present application. Referring to Figure 3, the inference model uses the MHA mechanism. During the execution of the inference model, an inference instruction is received, in which the overall header size headsize = 160 and the sequence length of the input query statement seqlength = 1024 are specified. This inference instruction can trigger the current round of inference.

[0080] Among them, the inference model may include multiple MHA operations. Taking operation A as an example, the acceleration requirement parameter values ​​of operation A in this round of inference process can be determined as headsize=80, seqlength=1024. It should be understood that the acceleration requirement parameter values ​​of operation A in the previous or next round of inference process may be different from this.

[0081] Based on this, headsize = 80 can be stored in the hash key value hashkey, and hashkey can be used to perform a search in the hash table hashmap (the hashmap contains the mapping between the accelerator operator ID and the hash key value hashkey, and the hash key value hashkey in the hashmap is also pre-built based on the headsize supported by each accelerator operator). Hash queries are very fast. Therefore, in this matching step, the candidate accelerator operators that support headsize = 80 can be quickly screened from the accelerator operator library for operation A. Referring to Figure 3, after this matching step, three candidate accelerator operators may be screened: accelerator operator a, accelerator operator b, and accelerator operator c.

[0082] After the first matching step is completed, the second matching step is performed using the acceleration requirement parameter value of seqlength = 1024. Referring to Figure 3, accelerator a supports seqlength = 512, accelerator b supports seqlength = 1024, and accelerator c supports seqlength = [128, 1024]. This is a universal sequence size. Based on this, in the second matching step, the following judgment logic can be executed for each candidate accelerator: First, determine whether the seqlength under it is a data value and whether the data value is equal to 1024. If so, it can be directly determined as the target accelerator; if not, it can continue to determine whether the seqlength under it is a preset identifier. If so, it can also be determined as the target accelerator. Moreover, if there are multiple accelerators that meet the judgment logic at the same time, the accelerator with seqlength configured as the data value is selected as the final target accelerator.

[0083] Through the above judgment logic, the acceleration operator b can be selected as the target acceleration operator matched by operation A in this round of reasoning.

[0084] If the seqlength corresponding to operation A is 128, then according to the above judgment logic, the acceleration operator c will be selected as the target acceleration operator matched by operation A in this round of reasoning.

[0085] It can be seen that the inference acceleration method provided in this embodiment can adaptively match appropriate acceleration operators for each inference operation to be accelerated in the inference model during the inference execution stage, thereby supporting the dynamically changing acceleration requirements of the inference operations during the inference execution stage, and ensuring that sufficiently good inference acceleration performance can be obtained in each round of inference.

[0086] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that descriptions such as "first" and "second" in this article are used to distinguish different parameter types, candidate operators, etc., and do not represent a sequence of precedence, nor do they limit "first" and "second" to being different types.

[0087] FIG4 is a schematic diagram of a computing device according to another exemplary embodiment of the present application. As shown in FIG4 , the computing device includes a memory 40 , a processor 41 , and a communication component 42 .

[0088] The processor 41 is coupled to the memory 40 and is configured to execute the computer program in the memory 40 to:

[0089] During the execution of the inference model, in response to the inference instruction, determining the acceleration requirement parameter value corresponding to the target inference operation in the current round of inference process;

[0090] Searching for a target acceleration operator that matches the acceleration requirement parameter value;

[0091] In this round of reasoning, calling the target acceleration operator for the target reasoning operation to accelerate the target reasoning operation;

[0092] The target reasoning operation is any reasoning operation to be accelerated contained in the reasoning model.

[0093] In an optional embodiment, when searching for a target acceleration operator that matches the acceleration requirement parameter value, the processor 41 may specifically be configured to:

[0094] Based on the attribute description information corresponding to each acceleration operator in the acceleration operator library, an acceleration operator having the attribute description information that meets the acceleration requirement parameter value is searched from the acceleration operator library as the target acceleration operator.

[0095] In an optional embodiment, the acceleration requirement parameter value includes a parameter value required by the target reasoning operation under first-category parameters and a parameter value required by the target reasoning operation under second-category parameters. The first-category parameters are configured as data values ​​in the attribute description information of each acceleration operator, and the second-category parameters are configured as data values ​​or data ranges in the attribute description information of a single acceleration operator. When the processor 41 searches the acceleration operator library for an acceleration operator having attribute description information that meets the acceleration requirement parameter value as the target acceleration operator, the processor 41 may specifically:

[0096] Searching, from the acceleration operator library, an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the first type of parameters as a candidate acceleration operator;

[0097] From the candidate acceleration operators, an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the second type of parameters is searched as the target acceleration operator.

[0098] In an optional embodiment, when the processor 41 searches the acceleration operator library for an acceleration operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the first type of parameters as a candidate acceleration operator, the processor 41 may specifically:

[0099] Constructing a query key value corresponding to the target reasoning operation according to the parameter values ​​required by the target reasoning operation under the first type of parameters;

[0100] Based on the mapping relationship between the identifier of the acceleration operator and the query key value key, the identifier of the acceleration operator corresponding to the target reasoning operation is determined to find the candidate acceleration operator.

[0101] In an optional embodiment, the processor 41 searches for an accelerator operator having attribute description information that meets the parameter values ​​required by the target reasoning operation under the second type of parameters from the candidate accelerator operators as the target accelerator operator, which may specifically be used to:

[0102] Determine a first candidate acceleration operator whose second-category parameter configuration in the attribute description information is a data value and the data value is consistent with a parameter value required by the target reasoning operation under the second-category parameter, as the target acceleration operator.

[0103] In an optional embodiment, the processor 41 may also be configured to:

[0104] If the first candidate acceleration operator does not exist, a second candidate acceleration operator whose second type of parameter configuration in the attribute description information is a data range and the data range includes the parameter value required by the target inference operation under the second type of parameter is determined as the target acceleration operator.

[0105] In an optional embodiment, a preset identifier is used in the attribute description information to indicate a data range, and the processor 41 may further be configured to:

[0106] If the second type of parameter in the attribute description information corresponding to the first candidate acceleration operator is a preset identifier, it is determined that the second type of parameter configuration of the first candidate acceleration operator is a data range.

[0107] In an optional embodiment, the target reasoning operation includes a multi-head attention calculation operation, the first type of parameters includes the head size headsize, and the second type of parameters includes the sequence length seq-length.

[0108] In an optional embodiment, the query key value key adopts a hash key value hashkey, and the mapping relationship between the identifier of the acceleration operator and the query key value key is represented by a hash table hashmap.

[0109] Furthermore, as shown in Figure 4 , the computing device further includes other components such as a power supply component 43. Figure 4 only schematically illustrates some components, which does not mean that the computing device only includes the components shown in Figure 4 .

[0110] It is worth noting that the technical details in the above-mentioned embodiments of the computing device can be referred to the relevant description in the aforementioned method embodiment. In order to save space, they will not be repeated here, but this should not cause any loss of the scope of protection of this application.

[0111] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computing device in the above method embodiment.

[0112] The memory in Figure 4 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, images, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0113] The communication component in Figure 4 above is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0114] The power supply assembly in Figure 4 provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0115] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0117] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0119] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0121] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A method for accelerating reasoning, comprising: During the operation of the inference model, in response to the inference instruction, determining the acceleration requirement parameter value corresponding to the target inference operation in the current round of inference process; Finding a target acceleration operator that matches the acceleration requirement parameter value; In this round of reasoning, calling the target acceleration operator for the target reasoning operation to accelerate the target reasoning operation; The target reasoning operation is any reasoning operation to be accelerated contained in the reasoning model.

2. The method according to claim 1, searching for a target acceleration operator that matches the acceleration requirement parameter value, comprising: Based on the attribute description information corresponding to each acceleration operator in the acceleration operator library, an acceleration operator having the attribute description information that meets the acceleration requirement parameter value is searched from the acceleration operator library as the target acceleration operator.

3. The method according to claim 2, wherein the acceleration requirement parameter value comprises a parameter value required by the target reasoning operation under a first type of parameter and a parameter value required by the target reasoning operation under a second type of parameter, the first type of parameter is configured as a data value in the attribute description information of each acceleration operator, and the second type of parameter is configured as a data value or a data range in the attribute description information of a single acceleration operator; Searching the acceleration operator having the attribute description information that meets the acceleration requirement parameter value from the acceleration operator library as the target acceleration operator includes: From the acceleration operator library, searching for an acceleration operator having attribute description information that meets the parameter value required by the target reasoning operation under the first type of parameters as a candidate acceleration operator; From the candidate acceleration operators, an acceleration operator having attribute description information that meets the parameter value required by the target reasoning operation under the second type of parameters is searched as the target acceleration operator.

4. The method according to claim 3, searching, from the acceleration operator library, an acceleration operator having attribute description information that meets the parameter value required by the target reasoning operation under the first type of parameters as a candidate acceleration operator, comprising: According to the parameter values ​​required by the target reasoning operation under the first type of parameters, construct a query key value corresponding to the target reasoning operation; Based on the mapping relationship between the identifier of the acceleration operator and the query key value, a candidate acceleration operator corresponding to the target reasoning operation is determined.

5. The method according to claim 3, searching, from the candidate acceleration operators, an acceleration operator having attribute description information that meets the parameter value required by the target reasoning operation under the second type of parameters as the target acceleration operator, comprising: Determine that a first candidate acceleration operator in which the second type of parameter configuration in the attribute description information is a data value and the data value is consistent with a parameter value required by the target reasoning operation under the second type of parameter is the target acceleration operator.

6. The method according to claim 5, further comprising: If the first candidate acceleration operator does not exist, the second type of parameter configuration in the attribute description information is determined to be a data range. And the second candidate acceleration operator whose data range includes the parameter value required by the target reasoning operation under the second type of parameters is the target acceleration operator.

7. The method according to claim 6, wherein a preset identifier is used in the attribute description information to refer to the data range, and the method further comprises: If the second type of parameter in the attribute description information corresponding to the acceleration operator is a preset identifier, it is determined that the second type of parameter configuration of the acceleration operator is a data range.

8. According to the method of claim 3, the target reasoning operation includes a multi-head attention calculation operation, the first type of parameters includes head size, and the second type of parameters includes sequence length.

9. According to the method of claim 4, the query key value adopts a hash key value, and the mapping relationship between the identifier of the acceleration operator and the query key value is represented by a hash table.

10. A computing device comprising a memory, a processor, and a communication component; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component, and is used to execute the one or more computer instructions to execute the reasoning acceleration method described in any one of claims 1-9.

11. A computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the inference acceleration method described in any one of claims 1-9.

Citation Information

Patent Citations

  • Inference acceleration method and device and storage medium

    CN116776978A

  • Quantification method and device, storage medium and electronic equipment

    CN114692864A

  • Deep learning model reasoning method and device, equipment and storage medium

    CN115600676A

  • Deep learning model reasoning method, machine translation method and device

    CN116011468A

  • Artificial intelligence based problem descriptions

    US20200117739A1

Cited By

  • Web large language model reasoning acceleration optimization method and device based on WebGPU

    CN119338016A

  • Inference apparatus, inference system, and inference task execution method

    WO2026170698A1