Inference acceleration method, device, and storage medium

By dynamically matching the acceleration operator during the operation of the inference model, the problem of unstable inference acceleration performance in the prior art is solved, and efficient inference acceleration under dynamic input size is achieved.

WO2024239875A1PCT designated stage expired Publication Date: 2024-11-28CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/088915
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-04-19
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The existing convolutional layer curing kernel scheme cannot meet the dynamic acceleration requirements of inference models, resulting in unstable inference acceleration performance.

Method used

During the operation of the inference model, in response to the inference instruction, the target input size is determined, and the interval containing the target input size is found from the pre-configured multiple dimension intervals, and the corresponding acceleration operator is associated to achieve dynamic acceleration operator matching.

Benefits of technology

By preconfiguring multiple size spaces and acceleration operators, the adaptive matching of efficient and accurate acceleration operators is achieved when the inference model supports dynamic input sizes, ensuring the inference acceleration performance in each round of inference process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088915_28112024_PF_FP_ABST
    Figure CN2024088915_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide an inference acceleration method, a device, and a storage medium. A plurality of size intervals are pre-configured for an inference model, and adapted acceleration operators are respectively selected for the model having a plurality of sizes in advance. On this basis, during operation of the inference model, a target input size corresponding to a target inference operation during the current round of inference can be determined, a target size interval containing the target input size can be found from the plurality of size intervals configured for the inference model, and the acceleration operator associated with the target size interval is used as a target acceleration operator adapted for the target inference operation in the current round of inference. Therefore, during operation of an inference model, an adapted acceleration operator can be determined only by finding an appropriate size interval for a target inference operation. In this way, when the inference model supports a dynamic input size, adaptive matching of acceleration operators can be efficiently and accurately implemented, thereby improving the inference acceleration performance.
Need to check novelty before this filing date? Find Prior Art

Description

Inference acceleration method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 23, 2023, with application number 202310596293.3 and invention name “A method, device and storage medium for inference acceleration”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to an inference acceleration method, device, and storage medium. Background Art

[0003] Currently, kernels (kernel functions on the GPU) are typically matched for each convolutional layer in the inference model during the inference initialization phase. The resulting matching results are then solidified and applied during the inference execution phase. That is, during the inference execution phase, the corresponding kernels for each convolutional layer are called based on the solidified matching results to achieve inference acceleration.

[0004] As inference requirements continue to diversify, some inference models can support multiple input / output specifications. However, the above solution of solidifying the kernel for the convolutional layer can no longer meet the dynamic acceleration requirements of the inference model, resulting in unstable inference acceleration performance.

[0005] Summary of the Invention

[0006] Various aspects of the present application provide an inference acceleration method, device, and storage medium to improve inference acceleration performance.

[0007] An embodiment of the present application provides an inference acceleration method, comprising: during the operation of an inference model, in response to an inference instruction, determining a target input size corresponding to a target inference operation in a current round of inference, wherein the input sizes corresponding to the target inference operation in different inference processes are not exactly the same; searching for a target size interval containing the target input size from a plurality of size intervals configured for the inference model; and using an acceleration operator associated with the target size interval as the target acceleration operator corresponding to the target inference operation in the current round of inference, so as to accelerate the target inference operation; wherein the target inference operation is any inference operation to be accelerated contained in the inference model.

[0008] An embodiment of the present application also provides a computing device, including a memory, a processor, and a communication component; the memory is used to store one or more computer instructions; the processor is coupled to the memory and the communication component, and is used to execute the one or more computer instructions to execute the aforementioned inference acceleration method.

[0009] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the aforementioned inference acceleration method. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] FIG1 shows a flowchart of an inference acceleration method provided by an exemplary embodiment of the present application.

[0013] FIG2 shows a logical diagram of an inference acceleration method provided by an exemplary embodiment of the present application.

[0014] FIG3 shows a schematic structural diagram of a computing device provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0015] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] Inference is the process of making predictions using a trained model. Herein, the model used for inference is described as an inference model. An inference model can be a neural network model or a machine learning model, among others. As inference needs continue to grow, the scale and complexity of inference models are also increasing. To ensure inference efficiency, inference acceleration technology has emerged. However, currently, inference acceleration performance is unstable. To this end, in some embodiments of the present application, multiple dimension spaces are pre-configured for the inference model, and adaptive acceleration operators are pre-selected for each of the multiple dimension models. Based on this, during the execution of the inference model, in response to an inference instruction, the target input dimension corresponding to the target inference operation in the current inference round can be determined. A target dimension interval containing the target input dimension can be found from the multiple dimension intervals configured for the inference model, and the acceleration operator associated with the target dimension interval is used as the target acceleration operator adapted for the target inference operation in the current inference round. Therefore, by pre-configuring multiple dimension spaces and the acceleration operators associated with each dimension space for the inference model, during the execution of the inference model, the adaptive acceleration operator can be determined by simply searching for the appropriate dimension interval for the target inference operation. In this way, when the inference model supports dynamic input size, adaptive matching of acceleration operators can be achieved efficiently and accurately, thereby ensuring the inference acceleration performance in each round of inference.

[0017] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0018] FIG1 is a flow chart of an inference acceleration method provided by an exemplary embodiment of the present application. The method can be executed by an inference acceleration engine. The inference acceleration engine can be implemented as software, hardware, or a combination of software and hardware. The inference acceleration engine can be integrated into a computing device. Referring to FIG1 , the method may include: step 100, during the operation of the inference model, in response to the inference instruction, determining the target input size corresponding to the target inference operation in the current round of inference process, wherein the input size corresponding to the target inference operation in different inference processes is not exactly the same; step 101, from the multiple size intervals configured for the inference model, searching for the target size interval containing the target input size; step 102, using the acceleration operator associated with the target size interval as the target acceleration operator corresponding to the target inference operation in the current round of inference process to accelerate the target inference operation; wherein the target inference operation is any inference operation to be accelerated contained in the inference model.

[0019] The inference acceleration method provided in this embodiment can be applied to various scenarios requiring inference acceleration, such as image processing and other AI inference scenarios. This embodiment does not limit the application scenarios. In different application scenarios, the inference functions carried by the inference model can be deployed as needed. For example, in an image processing scenario, the inference functions carried by the inference model can be object detection or image segmentation.

[0020] The reasoning acceleration method provided in this embodiment can be used as an innovative function in the reasoning acceleration engine. However, this embodiment does not limit the deployment location of the reasoning acceleration engine. The reasoning acceleration engine can be deployed on a cloud server, and of course it can also be deployed in other locations where there is a need for reasoning acceleration, such as an enterprise computer room. At present, the reasoning acceleration engine can optimize the reasoning models built by various mainstream AI computing frameworks based on IaaS (Infrastructure as a Service) resources, which can significantly improve the reasoning performance. The essence of reasoning acceleration is heterogeneous acceleration, which can be understood as handing over the reasoning work to the hardware resources used for reasoning acceleration (also known as heterogeneous acceleration period) instead of the CPU. This is because such hardware resources have hardware and software properties that are more suitable for performing reasoning work compared to the CPU. In this embodiment, the hardware resources used for reasoning acceleration may include but are not limited to GPU (Graphics Processing Unit, graphics processor) or NPU (Neural Networks Process Units, neural network processing unit), etc., and this embodiment does not limit this.

[0021] In this embodiment, the computational operations performed on each network layer during the execution of the inference model are described as inference operations. These network layers may include, but are not limited to, convolutional layers and pooling layers. The inference model is essentially a neural network model, and the basics of neural network models will not be elaborated upon here. It is worth noting that this embodiment focuses on the inference operations to be accelerated within the inference model, i.e., those that need to be executed on hardware resources used for inference acceleration.

[0022] In this embodiment, the functions running on the hardware resources used for inference acceleration are described as acceleration operators. For example, for a GPU, the functions running on it are kernel functions. Thus, a typical implementation form of the acceleration operator in this embodiment can be a kernel. Of course, the acceleration operator in this embodiment can also be implemented in a form compatible with other types of hardware resources, and is not limited to this.

[0023] FIG2 is a logical diagram of an inference acceleration method provided by an exemplary embodiment of the present application. Referring to FIG2 , in this embodiment, a variety of different acceleration operators can be provided. These acceleration operators can implement inference operations in an inference model on hardware resources used for inference acceleration to accelerate the inference operations. It should be understood that this acceleration is relative to the speed at which the inference operations are executed on the CPU. That is, the speed at which the inference operations are executed on the hardware resources used for inference acceleration is higher than the speed at which the inference operations are executed on the CPU.

[0024] In this embodiment, different acceleration operators have different acceleration strategies. Therefore, different acceleration operators will achieve better acceleration performance when adapted to their own usage requirements. To this end, this embodiment proposes a new technical concept for matching acceleration operators with inference operations.

[0025] Referring to Figure 1, in step 100, during the execution of the inference model, in response to an inference instruction, a target input size corresponding to a target inference operation in the current inference round can be determined. The input size can be used to represent the size information of the input data to be processed by the inference operation. In practical applications, the inference instruction can include the overall input size for the inference model, and the input size corresponding to each inference operation in the inference model in the current inference round can be determined based on this overall input size. The logic for determining the input size for the inference operation can be pre-installed in the inference model, that is, the inference model itself determines the input size corresponding to each inference operation it contains in the current inference round. In this case, in step 100, the target input size corresponding to the target inference operation in the current inference round can be obtained from the inference model. Of course, in other possible designs, the inference acceleration method provided in this embodiment can also implement the logic for determining the input size for the inference operation, thereby obtaining the target input size corresponding to the target inference operation in the current inference round, which is not limited in this embodiment.

[0026] During the operation of the inference model, multiple inference instructions may be received, and a single inference instruction may trigger a round of inference process. In a round of inference process, all inference operations contained in the inference model will be triggered for execution. In order to facilitate the description of the technical solution, in this embodiment, starting from step 100, the inference acceleration scheme is described using the target inference operation in the inference model as an example, wherein the target inference operation can be any inference operation to be accelerated contained in the inference model. In addition, it should be understood that other inference operations to be accelerated in the inference model can also be accelerated according to the same inference acceleration logic, thereby ensuring the acceleration performance in this round of inference process.

[0027] This embodiment pioneers the idea of ​​determining the input size corresponding to each inference operation to be accelerated in the inference model during the execution of the inference model, i.e., during the inference execution phase, in response to an inference instruction. It is understood that the input size corresponding to each inference operation included in the inference model is affected by the overall input size specified in the inference instruction. Therefore, when the inference model supports dynamic input size, the input size corresponding to the target inference operation in different inference processes will also be different. In other words, the target input size corresponding to the target inference operation can change dynamically.

[0028] For example, when the reasoning function of the inference model is target detection, the inference instruction may contain 3 pictures with a height of 30 and a width of 20, while the next inference instruction may contain 2 pictures with a height of 40 and a width of 30. Obviously, the overall input sizes in these two inference instructions are completely different, which will cause the input sizes corresponding to the target inference operation in these two rounds of inference to be different.

[0029] In this embodiment, the input size can be represented by the product of the number of samples, the sample input height, and the input width. Of course, this embodiment is not limited to this, and other methods can also be used to represent the input size. For inference operations, the sample here can be understood as the feature representation embedding assigned to the inference operation in a round of inference tasks.

[0030] As mentioned earlier, different accelerator operators achieve better acceleration performance when they are adapted to their specific needs. Therefore, finding an accelerator operator that matches the target input size for the target inference operation during the current round of inference ensures optimal inference acceleration performance.

[0031] To this end, this embodiment proposes that before the inference model is run, multiple size intervals can be configured for the inference model, and appropriate acceleration operators can be assigned to the multiple size intervals. That is, the pre-configuration of the acceleration operator can be completed during the inference initialization phase. It should be understood that the multiple size intervals configured for the inference model in this embodiment can cover the various input size conditions required for each inference operation contained in the inference model. In other words, any input size corresponding to each inference operation in the inference model in each round of inference can fall into a size interval configured for the inference model. The technical details related to pre-configuration will be described in detail later.

[0032] Based on this, referring to Figures 1 and 2, in step 101, a target size interval containing the target input size can be searched from the multiple size intervals configured for the inference model. For example, if the target input size corresponding to the target inference operation in this round of inference operation is 900, and the multiple size intervals configured for the inference model are [1, 32], [33, 64], ... [897, 928], ..., etc., then in step 101, the target size interval corresponding to the target inference operation can be found to be [897, 928].

[0033] As mentioned above, in this embodiment, acceleration operators have been pre-assigned to the multiple size intervals corresponding to the inference model. Based on this, referring to Figure 1, in step 102, the acceleration operator associated with the target size interval is used as the target acceleration operator corresponding to the target inference operation in the current inference round, thereby accelerating the target inference operation. This way, in each inference round, each inference operation to be accelerated can receive the appropriate acceleration, ensuring the stability of inference acceleration performance.

[0034] It is worth noting that the inference acceleration method provided in this embodiment is an innovative function in the inference acceleration engine. This innovative function can be integrated and coordinated with other functions in the inference acceleration engine. For example, the inference acceleration engine can fuse the inference operations to be accelerated, and use the fused inference operations as the target inference operations in this embodiment to reduce the number of times the acceleration operator is started, etc. This embodiment does not limit other acceleration functions in the inference acceleration engine.

[0035] In summary, in this embodiment, multiple size spaces are pre-configured for the inference model, and adaptive acceleration operators are pre-selected for the multiple size models. Based on this, during the operation of the inference model, the target input size corresponding to the target inference operation in this round of inference can be determined in response to the inference instruction, and the target size interval containing the target input size can be found from the multiple size intervals configured for the inference model, and the acceleration operator associated with the target size interval is used as the target acceleration operator adapted for the target inference operation in this round of inference. Therefore, by pre-configuring multiple size spaces and the acceleration operators associated with each size space for the inference model, during the operation of the inference model, it is only necessary to find a suitable size interval for the target inference operation to determine the adaptive acceleration operator. In this way, when the inference model supports dynamic input size, the adaptive matching of the acceleration operator can be achieved efficiently and accurately, thereby ensuring the inference acceleration performance in each round of inference.

[0036] In the above or following embodiments, before the inference model is run, suitable acceleration operators may be searched for the multiple size intervals corresponding to the inference model respectively; and a mapping relationship between the size intervals and the acceleration operators may be established.

[0037] In practical applications, the mapping relationship between the size intervals and acceleration operators constructed by the inference model can be stored in a database for future use.

[0038] In some possible implementations, the mapping relationship can be called from the database during the execution of the inference model as a basis for adaptively selecting an acceleration operator for the target inference operation.

[0039] In other possible implementations, the mapping relationship between size intervals and acceleration operators can be read from a database before the inference model runs; based on the mapping relationship between size intervals and acceleration operators, an inference configuration file corresponding to the inference model is generated, so that the mapping relationship between size intervals and acceleration operators is used based on the inference configuration file during the inference model run. In other words, the preconfiguration process of size intervals and acceleration operators can be completed during the inference initialization phase. This preconfiguration process can serve as a step in generating the inference configuration file during the inference initialization phase. The inference configuration file is a data object used to record various parameters related to the inference model, which can record the matching results of the acceleration operator. As described in the background technology, the acceleration operator selected for each convolutional layer in the inference model is currently fixed in the inference configuration file. In this way, during the inference execution phase, a fixed acceleration operator will be used for acceleration of a single convolutional layer in each round of inference according to the mapping relationship between the convolutional layer and the acceleration operator recorded in the inference configuration file. Based on the acceleration operator matching method proposed in this embodiment, during the generation of the aforementioned inference configuration file, the mapping relationship between the size intervals constructed for the inference model and the acceleration operators is recorded in the inference configuration file. Thus, during the execution of the inference model, the inference acceleration method provided in this embodiment can be executed based on the mapping relationship recorded in the inference configuration file. In practical applications, during the execution of the inference model, after step 100 of this embodiment, the inference configuration file can be read, and the size interval corresponding to the target input size of the target inference operation in the current round of inference process, as well as the target acceleration operator associated with this size interval, can be queried from the inference configuration file, thereby selecting the appropriate acceleration operator for the target inference operation in real time.

[0040] Of course, in this embodiment, other implementation schemes may also be used to store the mapping relationship between the size intervals created for the inference model and the acceleration operators, and this embodiment is not limited thereto.

[0041] In this embodiment, various implementations can be used to search for appropriate accelerators for the multiple size intervals corresponding to the inference model. One implementation may include selecting a representative size within a first size interval; searching an accelerator library for an accelerator that is appropriate for the representative size, and using the accelerator as the accelerator for the first size interval; wherein the first size interval is any one of the multiple size intervals.

[0042] In this exemplary implementation, a representative size can be selected from a single size interval. For example, a boundary value within the size interval can be selected. For example, if the size interval is [897, 928], "928" can be selected as the representative size. Based on this, an appropriate accelerator operator can be searched for the representative size from the accelerator operator library and used as the accelerator operator for the size interval.

[0043] The process of searching for an appropriate accelerator for the representative size can employ the existing scheme for selecting an accelerator for the convolutional layer during the inference initialization phase. It should be understood that existing accelerator selection schemes are essentially static selection schemes, while the process of searching for an appropriate accelerator for the representative size in this embodiment can also be considered a static selection scheme. This static selection scheme will not be discussed in detail here; existing static selection schemes and future static selection schemes are all applicable to the process of searching for an appropriate accelerator for the representative size in this embodiment.

[0044] It's also important to note that the same accelerator is typically used for each size within a size interval. In practice, the size interval specifications can be adjusted to ensure that all sizes within the interval achieve adequate acceleration using the same accelerator. If the size interval specifications are too large, causing the selected accelerator to fail to achieve adequate acceleration for some sizes, the size interval specifications can be adjusted down. This allows for efficient and accurate determination of the appropriate accelerator for a single size interval by executing the static matching logic only once.

[0045] Of course, this embodiment can also employ other implementation methods to search for suitable accelerators for multiple size intervals corresponding to the inference model. For example, for a single size interval, static matching logic can be executed for each size within it. The accelerator selected the most times within that size interval, or the accelerator with the highest acceleration efficiency index for a single size, is then determined as the accelerator suitable for that size interval. This embodiment is not limited to this.

[0046] In addition, in this embodiment, multiple implementations can also be used to configure multiple size intervals for the inference model. In one implementation, before the inference model is run, the required input size range corresponding to the inference model can be determined; and multiple size intervals can be divided from the required input size range.

[0047] In this implementation, the source of the required input size range for the inference model is not limited. In practical applications, the required input size range for the inference model can be directly specified by the user. Alternatively, the required input size range can be determined based on the upper limit of the size supported by the hardware resources used by the inference model. For example, if the hardware resource used by the inference model is a GPU, the required input size range for the inference model can be determined based on the upper limit of the size supported by the GPU's video memory.

[0048] After determining the input size requirement range corresponding to the inference model, multiple size intervals can be divided from the input size requirement range. An exemplary division scheme may be: dividing the input size requirement range into multiple size intervals according to a specified step size. For example, if the specified step size is 32 and the input size requirement range is [1,1024], the divided size intervals may be: [1,32], [33,64]…[897,928]…. It can be seen that the step size of each size interval is 32. Of course, this division scheme is exemplary, and this embodiment is not limited to this. Other division schemes may also be used to divide the inference model into multiple sizes, for example, dividing according to the interval boundary values ​​manually entered by the user, etc., which are not exhaustively listed here.

[0049] In addition, in the process of dividing the input and output size intervals, this embodiment also proposes: if the inference model supports multiple numbers of input channels and / or multiple numbers of output channels, then the various numbers of input channels and output channels supported by the inference model are combined to generate multiple combination results; under the multiple combination results, the input size requirement range is divided into multiple size intervals respectively, so as to generate multiple size intervals respectively under the multiple combination results corresponding to the inference model. Among them, input channel and input channel are both conventional concepts in neural network models and will not be explained in detail here. Here, this embodiment proposes to use the multiple combination results between the number of input channels and the number of input channels as indexes, and under each index, respectively perform the operation of dividing the input size requirement range into multiple size intervals. Continuing the example of using 32 as the step size for size interval division in the above text, if the number of input channels supported by the inference model is 1 or 3, and the number of output channels supported is 1, then two combination results [1,1] and [3,1] will be generated; for this reason, in this embodiment, the size division operation can be performed respectively under these two results, and the multiple size intervals configured for the inference model will be:

[0050] Under the [1,1] index --- [1,32], [33,64]... [897,928]...

[0051] Under the [3,1] index --- [1, 32], [33, 64]... [897, 928]...

[0052] Accordingly, in this embodiment, each size interval generated by each combination result is assigned an associated acceleration operator. It is understood that for the same size interval, the acceleration operator adapted to it may be different under different combination results. Furthermore, different partitioning methods can be used to divide the size intervals under different combination results, and are not limited to the partitioning method described in the above example.

[0053] When searching for a target acceleration operator for a target inference operation, the number of input and output channels corresponding to the target inference operation in the current inference round is determined. A target size interval containing the target input size is searched for from the multiple size intervals resulting from the combination of the input and output channel numbers. In other words, using the number of input and output channels corresponding to the target inference operation in the current inference round as an index, a set of size intervals is found, and the target size interval is found from within this set of size intervals.

[0054] In addition to the aforementioned number of input channels and output channels, this embodiment also incorporates other parameters that influence the selection of acceleration operators as a reference for dividing size intervals. For example, at least one model of hardware resources associated with the inference model for inference acceleration can be used. Based on this, the number of input channels, number of output channels, and models of associated hardware resources supported by the inference model are combined to produce a variety of combinations. In other words, various other specified parameter combinations related to the selection of acceleration operators can be used as indexes to guide the division of size intervals and the search for acceleration operators for inference operations.

[0055] In summary, in this embodiment, multiple size intervals can be configured for the inference model in advance, and a suitable acceleration operator can be selected for each size interval. These preprocessing results can be applied to the inference execution stage to support adaptive adjustment of the acceleration operator used for each inference operation according to the dynamic acceleration requirements of the inference model during the inference execution stage, thereby ensuring that each inference operation can obtain sufficient acceleration effect in each round of inference, thereby improving the acceleration performance of the inference model.

[0056] It should be noted that some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations, such as 101, 102, etc., are merely used to distinguish between different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0057] In an embodiment of the present application, multiple size spaces are pre-configured for the inference model, and adaptive acceleration operators are pre-selected for the multiple size models. Based on this, during the operation of the inference model, the target input size corresponding to the target inference operation in this round of inference can be determined in response to the inference instruction, and the target size interval containing the target input size can be found from the multiple size intervals configured for the inference model, and the acceleration operator associated with the target size interval is used as the target acceleration operator adapted for the target inference operation in this round of inference. Therefore, by pre-configuring multiple size spaces and the acceleration operators associated with each size space for the inference model, during the operation of the inference model, it is only necessary to find a suitable size interval for the target inference operation to determine the adaptive acceleration operator. In this way, when the inference model supports dynamic input size, the adaptive matching of the acceleration operator can be achieved efficiently and accurately, thereby improving the inference acceleration performance.

[0058] FIG3 is a schematic diagram of a computing device according to another exemplary embodiment of the present application. As shown in FIG3 , the computing device includes a memory 30 , a processor 31 , and a communication component 32 .

[0059] The processor 31 is coupled to the memory 30 and is used to execute the computer program in the memory 30, so as to: during the operation of the inference model, in response to the inference instruction, determine the target input size corresponding to the target inference operation in the current round of inference process, and the input size corresponding to the target inference operation in different inference processes is not exactly the same; from multiple size intervals configured for the inference model, search for a target size interval containing the target input size; use the acceleration operator associated with the target size interval as the target acceleration operator corresponding to the target inference operation in the current round of inference process to accelerate the target inference operation; wherein the target inference operation is any inference operation to be accelerated contained in the inference model.

[0060] In one embodiment, the processor 31 may be further configured to: before the inference model is run, search for respectively adapted acceleration operators for multiple size intervals corresponding to the inference model; and establish a mapping relationship between the size intervals and the acceleration operators.

[0061] In one embodiment, when the processor 31 searches for suitable acceleration operators for multiple size intervals corresponding to the inference model, it can be specifically used to: select a representative size in the first size interval; search for an acceleration operator that is suitable for the representative size from the acceleration operator library as the acceleration operator adapted for the first size interval; wherein the first size interval is any one of the multiple size intervals.

[0062] In one embodiment, the processor 31 may be further configured to: determine an input size requirement range corresponding to the inference model before the inference model is run; and divide the input size requirement range into a plurality of size intervals.

[0063] In one embodiment, when the processor 31 divides the input size requirement range into multiple size intervals, it can be used to: if the inference model supports multiple numbers of input channels and / or multiple numbers of output channels, then combine the various numbers of input channels and output channels supported by the inference model to generate multiple combination results; under multiple combination results, divide the input size requirement range into multiple size intervals respectively to generate multiple size intervals respectively under the multiple combination results corresponding to the inference model.

[0064] In one embodiment, when the processor 31 searches for a target size interval containing a target input size from a plurality of size intervals configured for the inference model, it can be used to: determine the number of input channels and the number of output channels corresponding to the target inference operation in this round of inference process; and search for a target size interval containing the target input size from a plurality of size intervals generated by the combination result of the number of input channels and the number of output channels.

[0065] In one embodiment, when the processor 31 combines various input channel numbers and output channel numbers supported by the inference model to generate multiple combination results, it can be used to: determine at least one model of hardware resources for inference acceleration associated with the inference model; combine the number of input channels, the number of output channels and the model of the associated hardware resources supported by the inference model to generate multiple combination results.

[0066] In one embodiment, the mapping relationship between the size interval and the acceleration operator is stored in a database, and the processor 31 can also be used to: read the mapping relationship between the size interval and the acceleration operator from the database before the inference model is run; generate an inference configuration file corresponding to the inference model based on the mapping relationship between the size interval and the acceleration operator, so as to use the mapping relationship between the size interval and the acceleration operator based on the inference configuration file during the operation of the inference model.

[0067] In one embodiment, the input size is the product of the number of samples, the input height of the samples, and the input width.

[0068] For example, as shown in Figure 3, the computing device further includes other components such as a power supply component 33. Figure 3 only schematically illustrates some components, which does not mean that the computing device only includes the components shown in Figure 3.

[0069] It is worth noting that the technical details in the above-mentioned embodiments of the computing device can be referred to the relevant description in the aforementioned method embodiment. In order to save space, they will not be repeated here, but this should not cause any loss of the scope of protection of this application.

[0070] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computing device in the above method embodiment.

[0071] The memory in Figure 3 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, images, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0072] The communication component in Figure 3 above is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0073] The power supply assembly in Figure 3 provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0074] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0075] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0076] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0078] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0080] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A method for accelerating reasoning, comprising: During the execution of the inference model, in response to the inference instruction, determining a target input size corresponding to a target inference operation in the current round of inference, wherein the target inference operation has different corresponding input sizes in different inference processes; Searching for a target size interval including the target input size from a plurality of size intervals configured for the inference model; The acceleration operator associated with the target size interval is used as the target acceleration operator corresponding to the target reasoning operation in the current round of reasoning process to accelerate the target reasoning operation; The target reasoning operation is any reasoning operation to be accelerated contained in the reasoning model.

2. The method according to claim 1, further comprising: Before the inference model is run, searching for suitable acceleration operators for multiple size intervals corresponding to the inference model respectively; According to the acceleration operators adapted to the multiple size intervals respectively, a mapping relationship between the size intervals and the acceleration operators is constructed.

3. The method according to claim 2, searching for suitable acceleration operators for the multiple size intervals corresponding to the inference model, respectively, comprises: Select a representative size in the first size interval; Searching an acceleration operator adapted to the representative size from an acceleration operator library as an acceleration operator adapted to the first size interval; The first size interval is any one of the multiple size intervals.

4. The method according to claim 1, further comprising: Before the inference model is run, determining the input size requirement range corresponding to the inference model; The plurality of size intervals are divided from the input size requirement range.

5. The method according to claim 4, dividing the plurality of size intervals from the input size requirement range comprises: If the inference model supports multiple numbers of input channels and / or multiple numbers of output channels, combinations are performed according to the various numbers of input channels and output channels supported by the inference model to generate multiple combination results; Under the multiple combination results, the input size requirement range is divided into multiple size intervals respectively, so as to generate multiple size intervals respectively under the multiple combination results corresponding to the inference model.

6. The method according to claim 5, searching for a target size interval including the target input size from a plurality of size intervals configured for the inference model, comprising: Determine the number of input channels and the number of input channels corresponding to the target reasoning operation in this round of reasoning; From a plurality of size intervals generated by a combination result of the number of input channels and the number of output channels, a target size interval including the target input size is searched.

7. The method according to claim 5, combining various numbers of input channels and output channels supported by the inference model to generate multiple combination results, including: Determine at least one model of hardware resources for reasoning acceleration associated with the reasoning model; The number of input channels, the number of output channels and the models of the associated hardware resources supported by the inference model are grouped to produce a variety of combined results.

8. The method according to claim 2, wherein the mapping relationship between the size interval and the acceleration operator is stored in a database, and the method further comprises: Before the inference model is run, the mapping relationship between the size interval and the acceleration operator is read from the database; An inference configuration file corresponding to the inference model is generated based on the mapping relationship between the size interval and the acceleration operator, so that the mapping relationship between the size interval and the acceleration operator is used based on the inference configuration file during the operation of the inference model.

9. The method of claim 1, wherein the input size is the product of the number of samples, the input height of the samples, and the input width.

10. A computing device comprising a memory, a processor, and a communication component; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component, and is used to execute the one or more computer instructions to execute the reasoning acceleration method described in any one of claims 1-9.

11. A computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the inference acceleration method described in any one of claims 1-9.

Citation Information

Patent Citations

  • Inference acceleration method and device and storage medium

    CN116776979A

  • CNN-SVM resource efficient acceleration architecture based on FPGA

    CN112306951A

  • Image retrieval method and device and image retrieval equipment

    CN114625903A

  • Adaptive reasoning acceleration method and device, computer equipment and storage medium

    CN114997401A

  • Heterogeneous acceleration method, equipment, system and storage medium

    CN115858154A