Inference acceleration method and device, electronic equipment and storage medium

Abstract

The invention discloses a reasoning acceleration method and device, electronic equipment and a storage medium, and relates to the field of computers, in particular to the technical field of artificial intelligence such as large models and deep learning. Firstly, pruning sensitivity corresponding to any element in each weight matrix in an ith layer of a model is determined based on input features of a target scene, sparseness corresponding to the ith layer is determined based on pruning sensitivity corresponding to all elements in each weight matrix in the ith layer, and then pruning sensitivity corresponding to any element in each weight matrix in the ith layer is determined based on the sparseness corresponding to the ith layer. The method comprises the following steps of: performing sparse processing on a weight matrix of an ith layer to obtain a weight matrix after sparse processing, then quantifying and adjusting elements which are not subjected to sparse processing in the weight matrix after sparse processing in the ith layer to obtain a target weight matrix in the ith layer, and finally processing input features in sequence based on the target weight matrix of each layer to obtain a target weight matrix in the ith layer. And obtaining a reasoning result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as large models and deep learning, specifically to inference acceleration methods, devices, electronic devices and storage media. Background Technology

[0002] Currently, the weights of large models generated through training are typically stored in high-precision floating-point form. However, this storage method limits the inference speed of the model under limited hardware bandwidth. Therefore, proposing an inference acceleration method to improve the inference speed of large models is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This disclosure provides an inference acceleration method, apparatus, electronic device, and storage medium. The specific solutions are as follows: According to one aspect of this disclosure, a reasoning acceleration method is provided, comprising: Based on the input features of the target scene, determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model, where i is an integer less than or equal to N, and N is the number of layers contained in the model; Based on the pruning sensitivity of each element in the weight matrix of the i-th layer, the sparsity of the i-th layer is determined. Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix. The elements in the sparse weight matrix of the i-th layer that have not been sparsified are quantized and adjusted to obtain the target weight matrix of the i-th layer. Based on the target weight matrix of each layer, the input features are processed sequentially to obtain the inference result.

[0004] According to another aspect of this disclosure, a reasoning acceleration device is provided, comprising: The first determining module is used to determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model based on the input features of the target scene, where i is an integer less than or equal to N, and N is the number of layers contained in the model. The second determining module is used to determine the sparsity of the i-th layer based on the pruning sensitivity corresponding to all elements in each weight matrix of the i-th layer. The first processing module is used to perform sparsification processing on the weight matrix of the i-th layer based on the sparsity corresponding to the i-th layer, so as to obtain the sparsified weight matrix. The second processing module is used to quantize and adjust the elements in the sparse weight matrix of the i-th layer that have not been sparsified, so as to obtain the target weight matrix in the i-th layer. The third processing module is used to process the input features sequentially based on the target weight matrix of each layer to obtain the inference result.

[0005] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in the above embodiments.

[0006] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0007] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0008] The inference acceleration method, apparatus, electronic device, and storage medium disclosed herein have the following beneficial effects: First, based on the input features of the target scene, the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model is determined. Then, based on the pruning sensitivity corresponding to all elements in each weight matrix of the i-th layer, the sparsity corresponding to the i-th layer is determined. Next, based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain a sparsified weight matrix. Then, the elements in the sparsified weight matrix of the i-th layer that have not undergone sparsification are quantized and adjusted to obtain the target weight matrix of the i-th layer. Finally, based on the target weight matrices of each layer, the input features are processed sequentially to obtain the inference result. Thus, by performing layered sparsification on the model based on the sparsity corresponding to each layer, the accuracy and precision of the sparsification process are improved. Next, quantizing the elements in the sparsified weight matrix of each layer that have not undergone sparsification effectively prevents the quantization process from affecting the sparsification result. Finally, based on the quantized target weight matrix, the input features are processed, thereby improving the model inference speed.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A schematic flowchart illustrating an inference acceleration method provided in an embodiment of this disclosure; Figure 2 This diagram illustrates the impact of sparsity on model performance in the inference acceleration method proposed in this disclosure. Figure 3 This is a schematic diagram illustrating the storage and computation of the sparsified weight matrix in the inference acceleration method proposed in this disclosure; Figure 4 This is a schematic diagram illustrating the architecture optimization of the inference acceleration method proposed in this disclosure; Figure 5 A schematic flowchart illustrating an inference acceleration method provided in another embodiment of this disclosure; Figure 6 A schematic flowchart illustrating an inference acceleration method provided in another embodiment of this disclosure; Figure 7 A schematic flowchart illustrating an inference acceleration method provided in another embodiment of this disclosure; Figure 8 A schematic flowchart illustrating an inference acceleration method provided in another embodiment of this disclosure; Figure 9 This is a schematic diagram illustrating the grouping and quantization of the weight matrix in the inference acceleration method proposed in this disclosure; Figure 10 This is a schematic diagram of the structure of an inference acceleration device provided in an embodiment of the present disclosure; Figure 11 This is a block diagram of an electronic device used to implement the inference acceleration method of the embodiments of this disclosure. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] This disclosure relates to the fields of artificial intelligence technologies such as large models and deep learning.

[0013] Artificial Intelligence (AI) is a new technological science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0014] Large models, also known as Foundation Models, are models that extract knowledge from hundreds of millions of corpora or images, learn, and then produce large models with hundreds of millions of parameters.

[0015] Deep learning (DL) learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound.

[0016] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0017] The following description, with reference to the accompanying drawings, outlines an inference acceleration method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.

[0018] Figure 1 This is a schematic flowchart of an inference acceleration method provided in an embodiment of the present disclosure.

[0019] like Figure 1 As shown, the inference acceleration method includes: Step 101: Based on the input features of the target scene, determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model.

[0020] Where i is an integer less than or equal to N, and N is the number of layers in the model.

[0021] It should be noted that the target scenarios for applying the inference acceleration method proposed in this disclosure can be determined according to actual needs. For example, the target scenarios can be scenarios with high latency requirements, such as online inference scenarios for commercial advertising and commercial creative generation scenarios. This disclosure does not limit these scenarios.

[0022] It should be noted that the specific architecture and type of the model used in the inference acceleration method proposed in this disclosure can be set according to actual needs. For example, it can be a large model, etc., and this disclosure does not limit it in this regard.

[0023] The pruning sensitivity can be used to represent the importance of elements in the weight matrix, thereby determining which elements can be sparsified (i.e., set to zero). In other words, the lower the pruning sensitivity of an element, the lower its importance, and the more likely it is to be sparsified, i.e., pruned.

[0024] In some possible implementations, when determining the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model based on the input features of the target scene, the gradient of the model's loss function with respect to any element can be determined first. Then, based on the gradient of the loss function with respect to any element and the input features of the target scene, the pruning sensitivity corresponding to any element can be determined. This provides the conditions for subsequently determining the sparsity of each layer of the model to perform hierarchical sparsification by determining the pruning sensitivity corresponding to each element of each weight matrix in each layer of the model.

[0025] The input features can be structured information input to the model in the target scenario. Specific input features can be determined based on the specific scenario. For example, in an online advertising inference scenario, input features can be user behavior features, advertising features, and interaction features; in a business creative generation scenario, input features can be basic business features such as business objectives, products, and selling points, etc. This disclosure does not impose any limitations on these.

[0026] The specific loss function can be set according to actual needs. For example, the loss function can be the Hessian matrix algorithm, etc., and this disclosure does not limit it.

[0027] Specifically, taking the Hessian matrix as the loss function as an example, the process of determining the pruning sensitivity corresponding to any element will be explained in detail: Assume the weight vector before sparsification is The weights after sparsification are ( (The change in weights). Error introduced by sparsification. It can be approximated by Taylor expansion as ,in, The gradient is the first-order partial derivative of the error with respect to the weights. The Hessian matrix; This is a higher-order term and can be ignored.

[0028] The optimization objective can be to find a set of weight elements that, when set to zero, minimize the error term. Based on the assumption of element independence and ignoring higher-order terms, the following can be obtained by solving using the Lagrangian function: For Each element within In terms of zeroing operations (i.e.) The error term it introduces is: ,in, This indicates that the input features will be related to The part where the operation occurs; It has the L2 norm. Therefore, for a given weight matrix... any element The pruning guidelines applied in this disclosure can be: ,in, As input features, For loss function pairs The magnitude of the gradient. In other words, the pruning sensitivity corresponding to any element in the weight matrix can be determined using this pruning criterion formula.

[0029] Step 102: Determine the sparsity of the i-th layer based on the pruning sensitivity of each element in the weight matrix of the i-th layer.

[0030] Here, sparsity can be used to represent the sparsity ratio corresponding to the sparsification process of the i-th layer. For example, the sparsity can be 50%, 75%, etc., and this disclosure does not limit it.

[0031] Layers with lower sparsity are more important to the target scene, and sparsification of these layers may cause a significant loss in model performance; therefore, they can be identified as sensitive layers. Conversely, layers with higher sparsity are less important to the target scene, and sparsification of these layers has a smaller impact on model performance; therefore, they can be identified as non-sensitive layers.

[0032] In other words, by determining the sparsity of each layer in the model, we can provide a basis for subsequent layered sparsification of the model.

[0033] In this disclosure, when determining the sparsity of the i-th layer based on the pruning sensitivity corresponding to each element in each weight matrix of the i-th layer, the importance of each weight matrix of the i-th layer can be determined first based on the pruning sensitivity corresponding to each element in each weight matrix of the i-th layer, and then the sparsity of the i-th layer can be determined based on the importance of all weight matrices in the i-th layer. This disclosure does not limit this.

[0034] For example, with Figure 2 For example, this paper illustrates the impact of sparsity in different layers of the model on model performance. Figure 2 This diagram illustrates the impact of sparsity on model performance in the inference acceleration method proposed in this disclosure.

[0035] Figure 2 In the diagram, the vertical axis represents model metrics (such as loss value, error rate, etc., with higher values ​​indicating worse performance), the thick gray horizontal line represents the performance baseline (performance without pruning), sensitive layers are layers with higher importance to the target scene, and non-sensitive layers are layers with lower importance to the target scene.

[0036] like Figure 2 As shown, the sparsity of the sensitive layer is 50%, which is a sparsity ratio of 2:4, which is lower than the sparsity of the non-sensitive layer, which is 75% (i.e., a sparsity ratio of 1:4).

[0037] When sensitive layers are pruned with a sparsity of 50%, the curve drops rapidly from around 0.6, indicating that the model performance deteriorates sharply once sensitive layers are pruned (sparseness changes from 0 to 50%). Therefore, for sensitive layers with high importance, a smaller sparsity can be set.

[0038] When pruning non-sensitive layers with a sparsity of 75%, the curve decreases after pruning (sparseness from 0 to 75%), but eventually recovers to a level close to the baseline. This indicates that non-sensitive layers have many redundant weights, so a higher sparsity can be set. Furthermore, after pruning, the model can recover most of its performance through compensation from the remaining weights or other layers, with minimal impact on model performance. Therefore, a larger sparsity can be set for non-sensitive layers with low importance.

[0039] Step 103: Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix.

[0040] In this disclosure, after determining the sparsity of the i-th layer in the model, the weight matrix of the layer can be sparsified based on the sparsity of the layer to obtain the sparsified weight matrix, thereby realizing the hierarchical sparsification of the model and improving the accuracy and precision of the sparsification process.

[0041] It should be noted that the specific processing procedure for sparsification of the i-th layer can be determined based on the sparsity of that layer. For example, if the sparsity of the i-th layer is 50%, when sparsifying that layer, taking groups of 4 elements as an example, two elements out of every four elements can be pruned to 0. When the sparsity of that layer is 75%, three elements out of every four elements can be pruned to 0, and so on. This disclosure does not impose any limitations on this.

[0042] Step 104: Quantize and adjust the elements in the sparse weight matrix of the i-th layer that have not been sparsified to obtain the target weight matrix of the i-th layer.

[0043] The target weight matrix can be the weight matrix of the i-th layer after the model has undergone sparsification and quantization.

[0044] In this disclosure, after the model is sparsified, the elements in the weight matrix of the sparsed layer that have not been sparsified can be quantized and adjusted to ensure that the quantization process does not affect the sparsification result.

[0045] It should be noted that the method for quantizing elements that have not undergone sparsity processing can be determined according to actual needs. For example, the elements can be quantized using the INT4 quantization method, etc., and this disclosure does not limit this.

[0046] INT4 quantization is a model quantization technique that converts floating-point parameters (weights, activation values) that originally occupied 32 or 16 bits in the model into integers (INT4) that occupy only 4 bits for storage and computation. INT4 stands for 4-bit Integer.

[0047] In some possible implementations, after obtaining the target weight matrix in the i-th layer, the number of non-zero elements contained in each index block of any target weight matrix in the i-th layer can be determined. Then, based on the number of non-zero elements contained in each index block, the 2-bit index code corresponding to each index block is determined. Thus, by determining the 2-bit index code corresponding to each index block, the index calculation in the model inference process can be effectively reduced, thereby improving the model inference speed.

[0048] Specifically, when determining the 2-bit index encoding for each index block based on the number of non-zero elements contained within each index block, data segmentation can be performed first, that is, splitting the sparsed weight matrix according to the index block size. For example, since 2 bits can represent four states, the matrix columns can be divided into "4 columns / index block" and the matrix rows into "M rows / segment". Each segment contains M rows × K index blocks, where K = total number of columns / 4; M is the number of parallel processing rows for the computation unit, such as 8 rows, which can be set as needed and is not limited here.

[0049] Then, for each index block within each segment, the state of non-zero elements within that block is represented by 2 bits: 00: non-zero elements appear out of nowhere (this block can be skipped without accessing the value and column index); 01: 1 non-zero element; 10: 2~3 non-zero elements; 11: 4 non-zero elements.

[0050] After encoding, a 2-bit index table is formed and stored in a cache (such as shared memory). The original 32-bit column index can store only the offset of the column containing the non-zero element within the index block (such as 0~3, which only requires 2 bits and can be stored together with the 2-bit index table).

[0051] Finally, when the computation unit needs to access the non-zero element of the i-th row during model inference, it can first determine the row segment to which it belongs by the row number i (i / M = segment number), read the 2-bit index table of the segment from the cache, and then traverse the index blocks within the segment. It can quickly filter out the "blocks with non-zero elements" (01, 10, or 11) by using 2-bit encoding. Then, for the filtered blocks, it calculates the actual address of the value and column index by "segment start position + block offset + intra-block offset". Therefore, it is not necessary to traverse the complete row pointer and column index. Since the index block size is fixed (e.g., 4 columns), the intra-block offset only requires 2 bits, so it can be directly combined with the 2-bit encoding for calculation without the need for the complex addressing of the 32-bit index. This can effectively reduce the computational complexity and overhead during model inference and improve the model inference speed.

[0052] The following is combined Figure 3 The storage and calculation of the sparsified weight matrix after sparsification of the model are illustrated with an example. Figure 3 This diagram illustrates the storage and computation of the sparsified weight matrix in the inference acceleration method proposed in this disclosure.

[0053] Figure 3 In the table, four elements are grouped together for cyclic indexing. The "index" records the position of non-zero elements in the matrix. For example, "1, 2, 0, 3" in the index table indicates that the elements at these positions have not been pruned. The "weight" retains the non-zero elements in the weight matrix. "vec of I" and "vec of D" represent the input features of the target scene. "refactor of W" represents the reconstructed weight matrix after pruning. In "refactor of W", the dashed blocks represent zero elements after sparsification, and the light-colored blocks represent non-zero elements.

[0054] like Figure 3 The left side of the diagram illustrates the storage and computation of the sparsified weight matrix in the sensitive layer, with a sparsity of 50%, or a sparsity ratio of 2:4. When storing the sparsified weight matrix, the input contains four non-zero elements (indices "1, 2, 0, 3"). Arrows point to the four valid elements in "refactor of W". Each valid element is multiplied by the input features (vec of I, vec of D) using a "mul" operation, and the output is the computation result. Dashed lines represent pruned zero elements; these positions are not included in the computation.

[0055] like Figure 3 The right side of the diagram shows the storage and calculation of the sparsified weight matrix in the non-sensitive layer, with a sparsity of 75%, or a sparsity ratio of 1:4. It should be noted that the storage and calculation of this part are similar to those on the left side, and will not be elaborated further here.

[0056] Step 105: Based on the target weight matrix of each layer, process the input features sequentially to obtain the inference result.

[0057] In this disclosure, after sparsification and quantization of the model, the input features are processed sequentially based on the target weight matrix of each layer to obtain the corresponding inference results, thereby improving the inference speed of the model.

[0058] In some possible implementations, before processing the input features sequentially based on the target weight matrices of each layer, the target weight matrix and / or input features can first be segmented according to the number of storage blocks and thread block size in the shared memory, resulting in multiple element blocks. Then, each element block is randomly shuffled and mapped to determine the shared memory address corresponding to each element within that block. Finally, based on the shared memory address of each element, each element in the weight matrix and / or input features is stored in its corresponding shared memory. Thus, by segmenting the data and randomly shuffling and mapping it, distributing the element blocks across different storage blocks, the storage conflict problem during data reading can be effectively resolved, improving data reading efficiency.

[0059] It should be noted that the specific size and structure of the shared memory can be preset as needed. For example, the shared memory can be shared by the graphics processing unit (GPU), etc., and this disclosure does not limit this.

[0060] It should be noted that storage can also be called "bank", and thread blocks can also be called "block", etc. This disclosure does not limit the terminology.

[0061] The number of storage units and the size of the thread blocks can both be preset according to actual needs. For example, the number of storage units can be 32, and the size of the thread blocks can be 256, etc. This disclosure does not limit these settings.

[0062] In this disclosure, the target weight matrix and / or input features are segmented according to the number of storage blocks contained in the shared memory and the size of the thread blocks to obtain multiple element blocks. This allows the size of each segment to be adapted to the parallel access granularity of the thread blocks and the storage granularity of the storage blocks.

[0063] For example, suppose there are 32 storage banks in the shared memory, and each storage bank can read 32 bits, i.e., 1 float (32-bit single-precision floating-point type), at one time. During model inference, there are 256 threads in the thread block, which need to access 256 data elements at the same time. When segmenting the target weight matrix and / or input features, the continuous data can be split into multiple segments. The size of each segment is equal to the product of the thread block size (256) and the data bit width (32 bits). The elements in each segment are grouped according to the number of storage banks (32), that is, each segment contains 8 storage bank groups (256 threads / 32 storage banks = 8), and each group contains 32 elements (corresponding to 32 storage banks, 1 element per storage bank).

[0064] In this disclosure, after segmenting the target weight matrix and / or input features to obtain multiple element blocks, each element block can be randomly shuffled and mapped, that is, each element block is rearranged (Shuffle).

[0065] The following is combined Figure 4 This paper provides an example of resolving data read conflicts in the inference acceleration method proposed in this disclosure. Figure 4 This is a schematic diagram illustrating the architecture optimization of the inference acceleration method proposed in this disclosure.

[0066] like Figure 4 As shown, the example of input features stored in global memory is used for illustration. First, the input features are segmented to obtain multiple element blocks (such as input0~1, ..., input14~15, input16~17, ..., input30~31). Then, each element block is randomly shuffled, mapped and loaded into shared memory, and the corresponding shared memory address is recorded.

[0067] in, Figure 4 The two vector blocks vec0 and vec1, which share memory, are used by various layers in the model (such as layers 0-15). Layers 0-15 corresponding to different threads can be multiple sets of parallel layer computation units (such as thread-block parallelism). For example, layers 0-7 corresponding to bank0 and bank1 can process different batches of requests simultaneously, thereby realizing multi-request parallel inference and improving the model inference speed.

[0068] Shared memory is shared by all warps within a thread block. Since all warps read the same input feature x, multiple warps work together to load activation x from high-bandwidth memory (HBM) into shared memory, significantly improving data reading speed.

[0069] In this embodiment, firstly, based on the input features of the target scene, the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model is determined. Then, based on the pruning sensitivity corresponding to all elements in each weight matrix of the i-th layer, the sparsity corresponding to the i-th layer is determined. Next, based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain a sparsified weight matrix. Then, the elements in the sparsified weight matrix of the i-th layer that have not undergone sparsification are quantized and adjusted to obtain the target weight matrix of the i-th layer. Finally, based on the target weight matrices of each layer, the input features are processed sequentially to obtain the inference result. Therefore, by determining the pruning sensitivity corresponding to each element in each weight matrix of each layer in the model based on the input features of different scenes, and determining the sparsity corresponding to that layer based on the pruning sensitivity of all elements in each weight matrix of each layer, the model can be layered for sparsification, improving the accuracy and precision of the sparsification process. Next, the elements in the sparsified weight matrix of each layer that were not sparsified are quantized, effectively preventing the quantization process from affecting the sparsification result. Then, based on the quantized target weight matrix, the input features are processed, thereby improving the model's inference speed.

[0070] The following example, using the commercial idea generation scenario as a case study, illustrates the application and effect of the reasoning acceleration method proposed in this disclosure. This scenario mainly involves rewriting and revising user ideas.

[0071] First, the sparsity of each layer in the model for this scenario can be determined using the inference acceleration method proposed in this disclosure. For example, the sparsity of sensitive layers (which are highly important to the scenario) and non-sensitive layers (which are less important to the scenario) can be determined. By assigning smaller sparsity to sensitive layers, performance loss can be reduced, thus ensuring the quality of generated ideas. Larger sparsity can be assigned to non-sensitive layers to reduce redundant data, thereby improving the inference speed of the model when generating ideas. Specifically, sensitive layers, such as attention layers, ensure the semantic and stylistic consistency of the generated ideas. To ensure the quality of the final generated ideas, attention layers are considered sensitive layers and require conservative pruning; therefore, a smaller sparsity is assigned. Non-sensitive layers, such as some fully connected layers, mainly play a feature transformation role in the idea generation process and have high redundancy. Pruning these layers has little impact on the quality of the generated ideas; therefore, a larger sparsity can be assigned.

[0072] Then, based on the sparsity corresponding to each layer, the model is subjected to hierarchical sparsification. The elements in the sparsified weight matrix of each layer that have not been sparsified are quantized to obtain the quantized target weight matrix. This effectively prevents the quantization process from affecting the sparsification result, thereby greatly reducing the loss of model accuracy, improving model computation performance, and increasing model inference speed.

[0073] The current input features (such as basic business features, such as business objectives, product types, and selling points) are processed by the quantified target weight matrix to generate inference results, such as creative copy generated based on the input features. This improves the quality and efficiency of creative generation and thus enhances the user experience.

[0074] Figure 5 This is a schematic flowchart of an inference acceleration method provided in another embodiment of the present disclosure.

[0075] like Figure 5 As shown, the inference acceleration method includes: Step 501: Based on the input features of the target scene, determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model.

[0076] Where i is an integer less than or equal to N, and N is the number of layers in the model.

[0077] The specific implementation of step 501 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0078] Step 502: Determine the impact value of each weight matrix on the target scene based on the pruning sensitivity corresponding to each element in each weight matrix.

[0079] The influence value can be used to represent the importance of the weight matrix to the target scene. In other words, the higher the influence value of the weight matrix to the target scene, the more important the weight matrix is ​​to the target scene. In this disclosure, the specific method for determining the influence value of each weight matrix on the target scene based on the pruning sensitivity corresponding to all elements in each weight matrix can be determined according to actual needs, and this disclosure does not limit it.

[0080] In some possible implementations, when determining the impact value of each weight matrix on the target scene based on the pruning sensitivity corresponding to each element in each weight matrix, the average pruning sensitivity corresponding to each element in each weight matrix can be used as the impact value of each weight matrix on the target scene. This disclosure does not limit this.

[0081] In some possible implementations, when determining the impact value of each weight matrix on the target scene based on the pruning sensitivity corresponding to each element in each weight matrix, the ratio of the number of elements in each weight matrix whose pruning sensitivity is greater than a first threshold to the total number of elements in the weight matrix can also be used to determine the impact value of each weight matrix on the target scene. This disclosure does not limit this aspect.

[0082] The first threshold can be a pruning sensitivity threshold used to determine the importance of elements in the weight matrix. It can be set as needed, and this disclosure does not limit it. That is, elements with a pruning sensitivity greater than the first threshold can be determined to have high importance, and elements with a pruning sensitivity less than or equal to the first threshold can be determined to have low importance.

[0083] When determining the influence value of each weight matrix on the target scene by the ratio of the number of elements in the weight matrix whose pruning sensitivity is greater than the first threshold to the total number of elements in the weight matrix, the greater the influence value of the weight matrix on the target scene is determined by the number of elements in the weight matrix whose pruning sensitivity is greater than the first threshold, and the smaller the influence value of the weight matrix on the target scene is determined by the number of elements in the weight matrix whose pruning sensitivity is greater than the first threshold.

[0084] Therefore, by using various methods such as the average pruning sensitivity of each element in the weight matrix, or the ratio of the number of elements in the weight matrix whose pruning sensitivity is greater than the first threshold to the total number of elements in the weight matrix, the influence of the weight matrix on the target scene is determined, thereby improving the flexibility and reliability of the inference acceleration method proposed in this disclosure.

[0085] Step 503: Determine the sparsity of the i-th layer based on the influence values ​​corresponding to all weight matrices in the i-th layer.

[0086] In this disclosure, after determining the influence value of each weight matrix in each layer of the model, the sparsity of the i-th layer is determined based on the influence values ​​of all weight matrices in the i-th layer. When the influence value of the weight matrix in the i-th layer is higher, the importance of the i-th layer is higher, and its corresponding sparsity is smaller. Conversely, when the influence value of the weight matrix in the i-th layer is lower, the importance of the i-th layer is lower, and its corresponding sparsity is larger. This improves the accuracy and reliability of the determined sparsity of each layer and provides a basis for subsequent hierarchical sparsification of the model.

[0087] Step 504: Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix.

[0088] Step 505: Quantize and adjust the elements in the sparse weight matrix of the i-th layer that have not been sparsified to obtain the target weight matrix of the i-th layer.

[0089] Step 506: Based on the target weight matrix of each layer, process the input features sequentially to obtain the inference result.

[0090] The specific implementation of steps 504 to 506 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0091] In this embodiment, firstly, based on the input features of the target scene, the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model is determined. Then, based on the pruning sensitivity corresponding to all elements in each weight matrix, the influence value of each weight matrix on the target scene is determined. Next, based on the influence values ​​corresponding to all weight matrices in the i-th layer, the sparsity corresponding to the i-th layer is determined. Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain a sparsified weight matrix. Then, the elements in the sparsified weight matrix of the i-th layer that have not undergone sparsification are quantized and adjusted to obtain the target weight matrix in the i-th layer. Finally, based on the target weight matrices of each layer, the input features are processed sequentially to obtain the inference result. Therefore, after determining the pruning sensitivity corresponding to each element in each weight matrix of each layer of the model based on the input features of different scenes, the influence value of the weight matrix is ​​determined based on the pruning sensitivity of all elements in each weight matrix, thereby improving the reliability and accuracy of the influence value determination. Next, based on the influence values ​​of all weight matrices in each layer, the sparsity of that layer is determined, thereby improving the accuracy and reliability of sparsity determination. Then, the model can be subjected to hierarchical sparsification based on the sparsity of each layer, and the unsparsed elements in the sparsified weight matrix are quantized. Finally, the input features are processed based on the quantized weight matrix to obtain the inference results, thus accelerating the computation speed of the model during inference.

[0092] Figure 6 This is a schematic flowchart of an inference acceleration method provided in another embodiment of the present disclosure.

[0093] like Figure 6 As shown, the inference acceleration method includes: Step 601: Based on the input features of the target scene, determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model.

[0094] Where i is an integer less than or equal to N, and N is the number of layers in the model.

[0095] Step 602: Determine the impact value of each weight matrix on the target scene based on the pruning sensitivity corresponding to each element in each weight matrix.

[0096] The specific implementation of steps 601 to 602 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0097] Step 603: Based on the influence values ​​corresponding to the j-th type weight matrix in all layers of the model, determine the j-th reference sparsity for each layer.

[0098] Where j is an integer less than or equal to M, and M is the number of all weight matrices of all types included in the model.

[0099] The type of weight matrix in each layer of the model can be determined according to the actual situation. For example, each layer of the model may include convolutional matrices, key matrices, etc., and this disclosure does not limit this.

[0100] The reference sparsity can be the sparsity determined based on the influence value of the j-th type of weight matrix. In other words, multiple reference sparsities can be determined for each layer based on the influence values ​​of different types of weight matrices in all layers of the model.

[0101] In this disclosure, since each layer of the model contains multiple weight matrices, and these multiple weight matrices may be of different types, when determining the sparsity of a layer based on the influence values ​​corresponding to all weight matrices in each layer, the j-th reference sparsity of each layer can be determined first based on the influence values ​​corresponding to the j-th type of weight matrices in all layers of the model.

[0102] For example, if a model contains 48 layers, each containing a convolutional matrix, sort the influence values ​​of the 48 convolutional matrices in ascending order (the smaller the influence value, the earlier it appears). From the sorted queue, determine the 10 layers containing the convolutional matrix with the smallest influence value (such as the first 5 and last 5 layers of the model). Pruning the convolutional matrices of these layers will cause less damage to the model's accuracy, and the corresponding sparsity will be larger. The reference sparsity from the convolutional matrices of these 10 layers can be set to a larger value.

[0103] For example, each layer contains a key-value matrix. The influence values ​​of all key-value matrices are sorted from smallest to largest. The 10 layers containing the key-value matrices with the smallest influence values ​​(e.g., the first 6 and last 4 layers of the model) are identified. Pruning the key-value matrices in these layers causes less damage to the model's accuracy, and their corresponding sparsity is larger. Therefore, the reference sparsity from the key-value matrices in these 10 layers can be set to a larger value. This process is repeated to determine the reference sparsity from different types of weight matrices in each layer.

[0104] Step 604: Determine the minimum value among all reference sparsities corresponding to the i-th layer as the sparsity corresponding to the i-th layer.

[0105] In this disclosure, after determining the j-th reference sparsity corresponding to each layer, the minimum value among all reference sparsities corresponding to the i-th layer can be determined as the sparsity corresponding to that layer.

[0106] Taking the above example, if the reference sparsity from the convolution matrix in the i-th layer is 30% and the reference sparsity from the key matrix is ​​70%, then the final sparsity of the i-th layer is 30%.

[0107] Therefore, by determining the minimum reference sparsity in each layer as the sparsity corresponding to that layer, it is possible to effectively ensure that pruning does not destroy any type of key weight matrix and avoid compromising model accuracy.

[0108] Step 605: Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix.

[0109] Step 606: Quantize and adjust the elements in the sparse weight matrix of the i-th layer that have not been sparsified to obtain the target weight matrix of the i-th layer.

[0110] Step 607: Based on the target weight matrix of each layer, process the input features sequentially to obtain the inference result.

[0111] The specific implementation of steps 605 to 607 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0112] In this embodiment, firstly, based on the input features of the target scene, the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model is determined. Then, based on the pruning sensitivity corresponding to all elements in each weight matrix, the influence value of each weight matrix on the target scene is determined. Next, based on the influence values ​​corresponding to the j-th type of weight matrix in all layers of the model, the j-th reference sparsity corresponding to each layer is determined. The minimum value among all reference sparsities corresponding to the i-th layer is determined as the sparsity corresponding to the i-th layer. Then, based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix. The elements in the sparsified weight matrix of the i-th layer that have not been sparsified are quantized and adjusted to obtain the target weight matrix in the i-th layer. Finally, based on the target weight matrices of each layer, the input features are processed sequentially to obtain the inference result. Therefore, after determining the impact value of the target scene corresponding to each weight matrix in each layer of the model, multiple reference sparsities are determined for each layer based on the impact values ​​corresponding to different types of weight matrices in all layers. The smallest reference sparsity in that layer is then determined as the sparsity corresponding to that layer. This hierarchical sparsity is performed based on the sparsity corresponding to each layer, improving the reliability and accuracy of the sparsity processing and effectively avoiding damage to model accuracy. Next, the sparsified weight matrices are quantized, and the input features are processed based on the quantized weight matrices, thereby improving the speed of model inference result generation while ensuring model accuracy.

[0113] Figure 7 This is a schematic flowchart of an inference acceleration method provided in another embodiment of the present disclosure.

[0114] like Figure 7 As shown, the inference acceleration method includes: Step 701: Based on the input features of the target scene, determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model.

[0115] Where i is an integer less than or equal to N, and N is the number of layers in the model.

[0116] Step 702: Determine the impact value of each weight matrix on the target scene based on the pruning sensitivity corresponding to each element in each weight matrix.

[0117] The specific implementation of steps 701 to 702 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0118] Step 703: The sum or mean of the influence values ​​corresponding to all weight matrices in the i-th layer is determined as the total influence value corresponding to the i-th layer.

[0119] In this disclosure, after determining the influence value of each weight matrix on the target scene, since each layer in the model contains multiple weight matrices, when determining the sparsity of the i-th layer based on the influence values ​​corresponding to all weight matrices in the i-th layer, the sum or mean of the influence values ​​corresponding to all weight matrices in the i-th layer can be determined as the total influence value corresponding to the i-th layer.

[0120] It should be noted that whether the sum or the mean of the influence values ​​corresponding to all weight matrices in the i-th layer is used to determine the total influence value corresponding to the i-th layer can be determined as needed, and this disclosure does not limit it.

[0121] Step 704: Based on the position of the total influence value corresponding to the i-th layer in the total influence value sequence, determine the sparsity corresponding to the i-th layer.

[0122] The total impact value sequence is obtained by sorting the total impact values ​​of all layers in the model from smallest to largest, or by sorting the total impact values ​​of all layers in the model from largest to smallest. This disclosure does not limit this.

[0123] In this disclosure, after determining the total influence value for each layer, the sparsity of that layer can be determined based on its position in the total influence value sequence. This effectively reduces the impact of sparsification on model accuracy and improves the reliability and accuracy of sparsification.

[0124] For example, taking the total impact value sequence as the total impact values ​​of all layers in the model sorted from largest to smallest, the total impact value sequence can be divided into two parts: the top 30% (the part with larger total impact values) and the bottom 70% (the part with smaller total impact values). The sparsity of the layers with the corresponding total impact values ​​in the top 30% (those with higher importance to the target scene) can be set to a smaller value, while the sparsity of the layers with the corresponding total impact values ​​in the bottom 70% (those with lower importance to the target scene) can be set to a larger value, and so on. This disclosure does not impose any limitations on this.

[0125] Step 705: Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix.

[0126] Step 706: Quantize and adjust the elements in the sparse weight matrix of the i-th layer that have not been sparsified to obtain the target weight matrix of the i-th layer.

[0127] Step 707: Based on the target weight matrix of each layer, process the input features sequentially to obtain the inference result.

[0128] The specific implementation of steps 705 to 707 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0129] In this embodiment, firstly, based on the input features of the target scene, the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model is determined. Then, based on the pruning sensitivity corresponding to all elements in each weight matrix, the influence value of each weight matrix on the target scene is determined. Then, the sum or mean of the influence values ​​corresponding to all weight matrices in the i-th layer is determined as the total influence value corresponding to the i-th layer. Based on the position of the total influence value corresponding to the i-th layer in the sequence of total influence values, the sparsity corresponding to the i-th layer is determined. Then, based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix. Then, the elements in the sparsified weight matrix of the i-th layer that have not been sparsified are quantized and adjusted to obtain the target weight matrix in the i-th layer. Finally, based on the target weight matrices of each layer, the input features are processed sequentially to obtain the inference result. Therefore, after determining the impact value of the target scene corresponding to each weight matrix in each layer of the model, the sum or mean of all weight matrices in each layer is determined as the total impact value corresponding to that layer. Based on the position of the total impact value corresponding to that layer in the total impact value sequence, the sparsity corresponding to that layer is determined, thereby improving the accuracy and precision of sparsity determination. This allows for hierarchical sparsification of the model and ensures the quantification of sparsity results. Based on the processed model, the input features are processed to obtain the inference results. Thus, while ensuring the accuracy and reliability of the model sparsification process, the inference speed of the model is improved.

[0130] Figure 8 This is a schematic flowchart of an inference acceleration method provided in another embodiment of the present disclosure.

[0131] like Figure 8 As shown, the inference acceleration method includes: Step 801: Based on the input features of the target scene, determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model.

[0132] Where i is an integer less than or equal to N, and N is the number of layers in the model.

[0133] Step 802: Determine the impact value of each weight matrix on the target scene based on the pruning sensitivity corresponding to each element in each weight matrix.

[0134] Step 803: Determine the sparsity of the i-th layer based on the influence values ​​corresponding to all weight matrices in the i-th layer.

[0135] Step 804: Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain the sparsified weight matrix.

[0136] The specific implementation of steps 801 to 804 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0137] Step 805: Determine the quantization granularity corresponding to each weight matrix in the i-th layer based on the influence value of each weight matrix in the i-th layer on the target scene and / or the sparsity corresponding to the i-th layer.

[0138] Quantization granularity can be used to represent the numerical precision of the weight matrix. In other words, the finer the quantization granularity, the higher the numerical precision of the weight matrix; the coarser the quantization granularity, the lower the numerical precision of the weight matrix.

[0139] In this disclosure, after determining the sparsity of each layer in the model, the quantization granularity of each weight matrix in the i-th layer can be determined based on the influence value of each weight matrix in the i-th layer on the target scene, and / or the sparsity of the i-th layer. For example, if a weight matrix in the i-th layer has a high influence value on the target scene, it can be determined that the corresponding weight matrix is ​​of high importance to the target scene. In this case, a finer quantization granularity can be set for this weight matrix, and / or if the sparsity of the i-th layer is a small value, it can be determined that the i-th layer has a significant influence on the target scene, i.e., it is of high importance. In this case, a finer quantization granularity can be set for the weight matrix in the i-th layer, and so on. This disclosure does not limit this.

[0140] In some possible implementations, the elements in the sparsed weight matrix can be grouped and quantized. Since the density of these groups affects model performance, considering both performance and accuracy, a finer quantization granularity can be set for weight matrices with higher impact values, and a coarser quantization granularity can be set for weight matrices with lower impact values. For example, ... Figure 9 As shown, Figure 9 This is a schematic diagram illustrating the grouping and quantization of the weight matrix in the inference acceleration method proposed in this disclosure. Figure 9 The entire weight matrix on the left side of the middle column shares a single scaling factor, scale1. This quantization method cannot adapt to the quantization needs of all data and has low precision. Figure 9 The right side of the middle section divides the weight matrix into multiple sub-blocks, each of which is configured with a scaling factor (scale1, scale2, scale3, scale4). By grouping and quantizing the weight matrix, the quantization accuracy is improved. This disclosure does not limit this aspect.

[0141] The scaling factor is a key parameter in the model quantization process, used to establish a linear mapping between high-precision floating-point numbers and low-precision integers. It can also be called the "quantization scaling factor," etc., but this disclosure does not limit its terminology.

[0142] Step 806: Based on the quantization granularity corresponding to each weight matrix, quantize and adjust the elements in the sparse weight matrix of the i-th layer that have not been sparsified, to obtain the target weight matrix in the i-th layer.

[0143] In this disclosure, after determining the quantization granularity corresponding to each weight matrix in each layer, the elements in the sparsely processed weight matrix of the i-th layer that have not undergone sparsification can be quantized and adjusted based on the quantization granularity corresponding to each weight matrix to obtain the target weight matrix in the i-th layer, thereby improving the precision and accuracy of quantization processing and enhancing the model inference performance.

[0144] Step 807: Based on the target weight matrix of each layer, the input features are processed sequentially to obtain the inference result.

[0145] The specific implementation of step 807 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0146] In this embodiment, firstly, based on the input features of the target scene, the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model is determined. Then, based on the pruning sensitivity corresponding to all elements in each weight matrix, the influence value of each weight matrix on the target scene is determined. Next, based on the influence values ​​corresponding to all weight matrices in the i-th layer, the sparsity corresponding to the i-th layer is determined. Based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain a sparsified weight matrix. Then, based on the influence value of each weight matrix in the i-th layer on the target scene and / or the sparsity corresponding to the i-th layer, the quantization granularity corresponding to each weight matrix in the i-th layer is determined. Based on the quantization granularity corresponding to each weight matrix, the elements in the sparsified weight matrix of the i-th layer that have not been sparsified are quantized and adjusted to obtain the target weight matrix in the i-th layer. Finally, based on the target weight matrices of each layer, the input features are processed sequentially to obtain the inference result. Therefore, after determining the influence value of each weight matrix in each layer of the model on the target scene and the corresponding sparsity of each layer, the quantization granularity of each weight matrix in that layer is determined based on the influence value of each weight matrix and / or the sparsity of that layer. The corresponding sparsed weight matrix is ​​then quantized based on the quantization granularity, thereby improving the precision and accuracy of the quantization process. The input features are then processed based on the quantized weight matrix to obtain the inference results, thus improving the inference performance and speed of the model.

[0147] To achieve the above embodiments, this disclosure also proposes an inference acceleration device.

[0148] Figure 10 This is a schematic diagram of the structure of an inference acceleration device provided in an embodiment of the present disclosure.

[0149] like Figure 10 As shown, the inference acceleration device 1000 includes: a first determining module 1001, a second determining module 1002, a first processing module 1003, a second processing module 1004, and a third processing module 1005.

[0150] The first determining module 1001 is used to determine the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model based on the input features of the target scene, where i is an integer less than or equal to N, and N is the number of layers contained in the model. The second determining module 1002 is used to determine the sparsity of the i-th layer based on the pruning sensitivity corresponding to all elements in each weight matrix of the i-th layer. The first processing module 1003 is used to perform sparsification processing on the weight matrix of the i-th layer based on the sparsity corresponding to the i-th layer, so as to obtain the sparsified weight matrix. The second processing module 1004 is used to quantize and adjust the elements in the sparse weight matrix of the i-th layer that have not been sparsified, so as to obtain the target weight matrix in the i-th layer. The third processing module 1005 is used to process the input features sequentially based on the target weight matrix of each layer to obtain the inference result.

[0151] Optionally, the first determining module 1001 described above is specifically used for: Determine the gradient of the model's loss function with respect to any element; Based on the gradient of the loss function with respect to any element and the input features of the target scene, the pruning sensitivity corresponding to any element is determined.

[0152] Optionally, the second determining module 1002 described above is specifically used for: Based on the pruning sensitivity corresponding to each element in each weight matrix, the influence value of each weight matrix on the target scene is determined. Based on the influence values ​​corresponding to all weight matrices in the i-th layer, the sparsity of the i-th layer is determined.

[0153] Optionally, the second determining module 1002 described above is specifically used for any of the following: The average pruning sensitivity of each element in each weight matrix is ​​used to determine the impact value of each weight matrix on the target scene. The ratio of the number of elements in each weight matrix whose pruning sensitivity is greater than the first threshold to the total number of elements in the weight matrix is ​​used to determine the impact value of each weight matrix on the target scene.

[0154] Optionally, the second determining module 1002 described above is further configured to: Based on the influence values ​​corresponding to the j-th type of weight matrix in all layers of the model, the j-th reference sparsity corresponding to each layer is determined, where j is an integer less than or equal to M, and M is the number of all types of weight matrices included in the model; The minimum value among all reference sparsities corresponding to the i-th layer is determined as the sparsity corresponding to the i-th layer.

[0155] Optionally, the second determining module 1002 described above is further configured to: The sum or mean of the influence values ​​corresponding to all weight matrices in the i-th layer is determined as the total influence value corresponding to the i-th layer. Based on the position of the total influence value corresponding to the i-th layer in the total influence value sequence, the sparsity corresponding to the i-th layer is determined. The total influence value sequence is obtained by sorting the total influence values ​​corresponding to all layers in the model from smallest to largest, or by sorting the total influence values ​​corresponding to all layers in the model from largest to smallest.

[0156] Optionally, the second processing module 1004 described above is specifically used for: Based on the influence value of each weight matrix in the i-th layer on the target scene, and / or the sparsity corresponding to the i-th layer, determine the quantization granularity corresponding to each weight matrix in the i-th layer; Based on the quantization granularity corresponding to each weight matrix, the elements in the sparse weight matrix of the i-th layer that have not undergone sparsification are quantized.

[0157] Optionally, the second processing module 1004 described above is further configured to: Determine the number of non-zero elements contained in each index block of any target weight matrix in the i-th layer; The 2-bit index code for each index block is determined based on the number of non-zero elements contained in each index block.

[0158] Optionally, the third processing module 1005 described above is further configured to: Based on the number of storage blocks contained in the shared memory and the size of the thread block, the target weight matrix and / or input features are segmented to obtain multiple element blocks; Each element block is randomly shuffled and mapped to determine the shared memory address corresponding to each element within each element block. Based on the shared memory address corresponding to each element, the elements of the weight matrix and / or input features are stored into the corresponding shared memory.

[0159] It should be noted that the explanation of the aforementioned inference acceleration method embodiment also applies to the inference acceleration device of this embodiment, and therefore will not be repeated here.

[0160] In this embodiment, firstly, based on the input features of the target scene, the pruning sensitivity corresponding to any element in each weight matrix of the i-th layer of the model is determined. Then, based on the pruning sensitivity corresponding to all elements in each weight matrix of the i-th layer, the sparsity corresponding to the i-th layer is determined. Next, based on the sparsity corresponding to the i-th layer, the weight matrix of the i-th layer is sparsified to obtain a sparsified weight matrix. Then, the elements in the sparsified weight matrix of the i-th layer that have not undergone sparsification are quantized and adjusted to obtain the target weight matrix of the i-th layer. Finally, based on the target weight matrices of each layer, the input features are processed sequentially to obtain the inference result. Therefore, by determining the pruning sensitivity corresponding to each element in each weight matrix of each layer in the model based on the input features of different scenes, and determining the sparsity corresponding to that layer based on the pruning sensitivity of all elements in each weight matrix of each layer, the model can be layered for sparsification, improving the accuracy and precision of the sparsification process. Next, the elements in the sparsified weight matrix of each layer that were not sparsified are quantized, effectively preventing the quantization process from affecting the sparsification result. Then, based on the quantized target weight matrix, the input features are processed, thereby improving the model's inference speed.

[0161] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0162] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0163] like Figure 11As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 1102 or loaded from storage unit 1108 into RAM (Random Access Memory) 1103. RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. I / O (Input / Output) interface 1105 is also connected to bus 1104.

[0164] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0165] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as inference acceleration methods. For example, in some embodiments, the inference acceleration method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the inference acceleration method described above may be performed. Alternatively, in other embodiments, computing unit 1101 may be configured to execute inference acceleration methods by any other suitable means (e.g., by means of firmware).

[0166] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0168] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0170] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0171] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0172] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when an instruction processor in the computer program product is executed, performs the inference acceleration method proposed in the above embodiments of this disclosure.

[0173] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0174] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An inference acceleration method, characterized by, The method comprises: determining a pruning sensitivity corresponding to any element in each weight matrix in the i-th layer of the model based on input features of a target scene, wherein i is an integer less than or equal to N, and N is the number of layers included in the model; determining a sparsity corresponding to the i-th layer based on the pruning sensitivities corresponding to all elements in each weight matrix in the i-th layer; performing sparsification processing on the weight matrix of the i-th layer based on the sparsity corresponding to the i-th layer to obtain a sparsely processed weight matrix; quantizing and adjusting the elements in the sparsely processed weight matrix in the i-th layer that have not been subjected to sparsification processing to obtain a target weight matrix in the i-th layer; processing the input features in sequence based on the target weight matrices of the layers to obtain an inference result.

2. The method of claim 1, wherein, The method comprises: determining the gradient of the loss function of the model with respect to the any element; determining the pruning sensitivity corresponding to the any element based on the gradient of the loss function with respect to the any element and the input features of the target scene.

3. The method of claim 1, wherein, The method comprises: determining the influence value of each weight matrix on the target scene based on the pruning sensitivities corresponding to all elements in each weight matrix; determining the sparsity corresponding to the i-th layer based on the influence values corresponding to all weight matrices in the i-th layer.

4. The method of claim 3, wherein, The method comprises any one of the following: determining the influence value of each weight matrix on the target scene as the average of the pruning sensitivities corresponding to all elements in the weight matrix; determining the influence value of each weight matrix on the target scene as the ratio of the number of elements in the weight matrix whose pruning sensitivities are greater than a first threshold to the total number of elements in the weight matrix.

5. The method of claim 3, wherein, The method comprises: determining a j-th reference sparsity corresponding to each layer based on the influence values corresponding to the j-th type of weight matrix in all layers of the model, wherein j is an integer less than or equal to M, and M is the number of types of all weight matrices included in the model; determining the minimum value of all reference sparsities corresponding to the i-th layer as the sparsity corresponding to the i-th layer.

6. The method of claim 3, wherein, The method comprises: determining the total influence value corresponding to the i-th layer as the sum or average of the influence values corresponding to all weight matrices in the i-th layer. determine the sparsity corresponding to the i-th layer based on the position of the total influence value corresponding to the i-th layer in the total influence value sequence, wherein the total influence value sequence is obtained by sorting the total influence values corresponding to all layers in the model from small to large or from large to small.

7. The method of claim 3, wherein, The quantization and adjustment of the elements in the weight matrix of the i-th layer that have not been sparsified include: determining the quantization granularity corresponding to each weight matrix in the i-th layer according to the influence value of each weight matrix in the i-th layer on the target scene and / or the sparsity corresponding to the i-th layer; quantizing the elements in the weight matrix of the i-th layer that have not been sparsified based on the quantization granularity corresponding to each weight matrix.

8. The method of claim 7, wherein, After obtaining the target weight matrix in the i-th layer, the method further includes: determining the number of non-zero elements contained in each index block in any target weight matrix in the i-th layer; determining the 2-bit index code corresponding to each index block based on the number of non-zero elements contained in each index block.

9. The method of any one of claims 1-8, wherein, Before processing the input feature based on the target weight matrix of each layer in turn, the method further includes: segmenting the target weight matrix and / or the input feature according to the number of memory banks and the size of thread blocks contained in the shared memory to obtain a plurality of element blocks; randomly mapping each element block to determine the shared memory address corresponding to each element in the element block; storing each element in the weight matrix and / or the input feature into the corresponding shared memory based on the shared memory address corresponding to each element.

10. An inference acceleration device, comprising: The method includes: a first determining module configured to determine the pruning sensitivity corresponding to any element in each weight matrix in the i-th layer of the model based on the input feature of the target scene, wherein i is an integer less than or equal to N, and N is the number of layers contained in the model; a second determining module configured to determine the sparsity corresponding to the i-th layer based on the pruning sensitivity corresponding to all elements in each weight matrix in the i-th layer; a first processing module configured to sparsify the weight matrix of the i-th layer based on the sparsity corresponding to the i-th layer to obtain a sparsified weight matrix; a second processing module configured to quantize and adjust the elements in the sparsified weight matrix of the i-th layer that have not been sparsified to obtain a target weight matrix in the i-th layer; a third processing module configured to process the input feature based on the target weight matrix of each layer in turn to obtain an inference result.

11. An electronic device, comprising: The method includes: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-9. The computer instructions are used to enable the computer to perform the method of any one of claims 1-9.

13. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by a processor implements the steps of the method of any one of claims 1-9.

Images