Model reasoning method and device, computer equipment and storage medium
By matching the optimized code and data segmentation processing of processing equipment, the utilization rate of processing equipment in the inference process of deep learning model is improved, the problem of low equipment utilization in traditional technology is solved, and faster response speed is achieved.
Patent Information
- Application Number
- CN202310544666.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-15
AI Technical Summary
The utilization rate of processing equipment during the inference process of traditional deep learning models is not high, resulting in inefficiency.
By obtaining the operation optimization code that matches the current processing device, determining the input feature data and parameter data of the network layer based on the input data for segmentation processing, forming a matrix operation group, and using the cache space for processing, optimizing cache access to improve device utilization rate .
This improves the utilization rate of processing devices, reduces the possibility of cache missing, and speeds up the response speed of inference requests.
Smart Images

Figure CN118966348A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a model reasoning method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0002] With the development of deep learning technology, deep learning models are being used more and more widely. For example, deep learning models can be applied in image recognition scenarios, natural language processing scenarios, data push scenarios, etc. How to improve the utilization rate of processing equipment during the deep learning model reasoning process has become an urgent problem to be solved.
[0003] In traditional technology, all types of processing devices use the same set of operation optimization codes to calculate inference results. However, the utilization rate of processing devices is not high under this method. Summary of the invention
[0004] Based on this, it is necessary to provide a model reasoning method, device, computer equipment, computer-readable storage medium and computer program product that can improve the utilization rate of processing equipment in response to the above technical problems.
[0005] In one aspect, the present application provides a model reasoning method. The method includes:
[0006] In response to the inference request, obtain the operation optimization code matching the current processing device, and determine the input feature data of each network layer in the inference model corresponding to the inference request based on the input data carried by the inference request;
[0007] Determine the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device;
[0008] For each network layer, according to the intermediate matrix block information, the input feature data and network parameter data of the corresponding network layer are segmented to obtain corresponding first intermediate matrices and second intermediate matrices;
[0009] Based on each first intermediate matrix and each second intermediate matrix, each matrix operation group is obtained by combination; each matrix operation group is sequentially cached in a cache space, and the matrix operation group cached in the cache space each time is processed by using the operation optimization code to obtain the operation results corresponding to each matrix operation group;
[0010] Based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined.
[0011] In a second aspect, the present application also provides a model reasoning device. The device includes:
[0012] An acquisition module, used to respond to an inference request, acquire an operation optimization code matching the current processing device, and determine input feature data of each network layer in the inference model corresponding to the inference request based on input data carried by the inference request;
[0013] A first determination module is used to determine the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device;
[0014] A segmentation module is used to segment the input feature data and network parameter data of the corresponding network layer according to the intermediate matrix block information for each network layer, so as to obtain corresponding first intermediate matrices and second intermediate matrices;
[0015] The operation module is used to combine and obtain each matrix operation group based on each first intermediate matrix and each second intermediate matrix; cache each matrix operation group in the cache space in sequence, and use the operation optimization code to process the matrix operation group cached in the cache space each time to obtain the operation results corresponding to each matrix operation group;
[0016] The second determination module is used to determine the output data of the corresponding network layer based on the operation results corresponding to each matrix operation group, and determine the inference result corresponding to the inference request based on the output data of each network layer.
[0017] In some embodiments, the acquisition module is specifically used to: obtain target attribute information of the current processing device; match the target attribute information with the attribute information corresponding to each pre-stored optimization code, and use the successfully matched optimization code as the operation optimization code matched with the current processing device.
[0018] In some embodiments, the acquisition module is also used to: for different processing devices, use a preset feature detection tool to detect the architectural features of the targeted processing devices; determine the target processing devices whose model levels meet preset conditions from the different processing devices, and determine the data block information based on the statistical data of the number of registers and the number of channels corresponding to the target processing devices; generate the optimization code corresponding to the targeted processing devices according to the architectural features and the data block information, and store the optimization code corresponding to each processing device.
[0019] In some embodiments, the cache space in the current processing device includes a first cache space and a second cache space, the cache attribute information includes the space capacity of the first cache space and the space capacity of the second cache space, and the cache reference information includes first reference information for row-direction data and second reference information for column-direction data; the access priority of the first cache space is greater than the access priority of the second cache space; the first determination module is specifically used to: determine the intermediate matrix blocking information according to the data blocking information, the cache reference information and the cache attribute information of the cache space in the current processing device, including: determining the column-direction blocking information according to the data blocking information, the first reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the first cache space; determining the row-direction blocking information according to the column-direction blocking information, the data blocking information, the second reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the second cache space; the column-direction blocking information and the row-direction blocking information constitute the intermediate matrix blocking information.
[0020] In some embodiments, the first determination module is specifically used to: merge the row block information, the first reference information, and the number of columns of the input feature data of the corresponding network layer in the data block information to obtain the first data volume of the input feature data that needs to be cached at least in the first cache space; merge the first parameter to be required, the column block information in the data block information, and the number of columns of the input feature data of the corresponding network layer to obtain the second data volume of the network parameter data that can be cached in the first cache space; merge the row block information, the first reference information, the first parameter to be required, and the column block information in the data block information to obtain the third data volume of the calculation result that can be cached in the first cache space; count the first data volume, the second data volume, and the third data volume to obtain a first statistical result; establish a first equation based on the first statistical result and the space capacity of the first cache space; obtain the value of the first parameter to be required according to the first equation, and use the value of the first parameter to be required as the column direction block information.
[0021] In some embodiments, the first determination module is specifically used to: merge the second parameter to be required, the row blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer to obtain a fourth data volume of the input feature data that can be cached in the second cache space; merge the second reference information, the column direction blocking information, the column blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer to obtain a fifth data volume of the network parameter data that needs to be cached at least in the second cache space; merge the second parameter to be required, the row blocking information in the data blocking information, the second reference information, the column direction blocking information, and the column blocking information in the data blocking information to obtain a sixth data volume of the calculation result that can be cached in the second cache space; count the fourth data volume, the fifth data volume, and the sixth data volume to obtain a second statistical result; establish a second equation based on the second statistical result and the space capacity of the second cache space; obtain the value of the second parameter to be required according to the second equation, and use the value of the second parameter to be required as the row direction blocking information.
[0022] In some embodiments, the second determination module is specifically used to: for each matrix operation group, cache the first intermediate matrix in the targeted matrix operation group into the cache space through the first rearrangement code corresponding to the input feature data contained in the operation optimization code; cache the second intermediate matrix in the targeted matrix operation group into the cache space through the second rearrangement code corresponding to the network parameter data contained in the operation optimization code.
[0023] In some embodiments, the second determination module is specifically used to: obtain the number of columns of the input feature data of the corresponding network layer, and when the number of columns is greater than a preset threshold, divide the first intermediate matrix in the targeted matrix operation group into multiple first matrix blocks according to the data blocking information; cache the multiple first matrix blocks into the cache space with column priority, wherein the elements in each first matrix block are cached into the cache space in a column-by-column manner.
[0024] In some embodiments, the second determination module is specifically configured to: when the number of columns is less than or equal to a preset threshold, cache the first intermediate matrix in the targeted matrix operation group into the cache space in a row-by-row cache manner.
[0025] In some embodiments, the second determination module is specifically used to: divide the second intermediate matrix in the targeted matrix operation group into multiple second matrix blocks according to the data blocking information; cache the multiple second matrix blocks into the cache space with columns as priority, wherein the elements in each second matrix block are cached into the cache space in a row-by-row manner.
[0026] In some embodiments, the inference request is an image recognition request, the input data carried by the inference request is an image to be recognized, and the inference result corresponding to the inference request is an image recognition result.
[0027] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0028] In response to the inference request, obtain the operation optimization code matching the current processing device, and determine the input feature data of each network layer in the inference model corresponding to the inference request based on the input data carried by the inference request;
[0029] Determine the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device;
[0030] For each network layer, according to the intermediate matrix block information, the input feature data and network parameter data of the corresponding network layer are segmented to obtain corresponding first intermediate matrices and second intermediate matrices;
[0031] Based on each first intermediate matrix and each second intermediate matrix, each matrix operation group is obtained by combination; each matrix operation group is sequentially cached in a cache space, and the matrix operation group cached in the cache space each time is processed by using the operation optimization code to obtain the operation results corresponding to each matrix operation group;
[0032] Based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined.
[0033] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0034] In response to the inference request, obtain the operation optimization code matching the current processing device, and determine the input feature data of each network layer in the inference model corresponding to the inference request based on the input data carried by the inference request;
[0035] Determine the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device;
[0036] For each network layer, according to the intermediate matrix block information, the input feature data and network parameter data of the corresponding network layer are segmented to obtain corresponding first intermediate matrices and second intermediate matrices;
[0037] Based on each first intermediate matrix and each second intermediate matrix, each matrix operation group is obtained by combination; each matrix operation group is sequentially cached in a cache space, and the matrix operation group cached in the cache space each time is processed by using the operation optimization code to obtain the operation results corresponding to each matrix operation group;
[0038] Based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined.
[0039] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0040] In response to the inference request, obtain the operation optimization code matching the current processing device, and determine the input feature data of each network layer in the inference model corresponding to the inference request based on the input data carried by the inference request;
[0041] Determine the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device;
[0042] For each network layer, according to the intermediate matrix block information, the input feature data and network parameter data of the corresponding network layer are segmented to obtain corresponding first intermediate matrices and second intermediate matrices;
[0043] Based on each first intermediate matrix and each second intermediate matrix, each matrix operation group is obtained by combination; each matrix operation group is sequentially cached in a cache space, and the matrix operation group cached in the cache space each time is processed by using the operation optimization code to obtain the operation results corresponding to each matrix operation group;
[0044] Based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined.
[0045] The above-mentioned model reasoning method, device, computer equipment, storage medium and computer program product can obtain the operation optimization code matching the current processing device in response to the reasoning request, and determine the input feature data of each network layer in the reasoning model corresponding to the reasoning request based on the input data carried by the reasoning request; in the process of calculating the output data of each network layer, the intermediate matrix block information can be determined according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device. For each network layer, the network parameter data and input feature data of the corresponding network layer can be segmented and processed according to the intermediate matrix block information to obtain the corresponding first intermediate matrices and second intermediate matrices; then, based on the first intermediate matrices and the second intermediate matrices, each matrix operation group is combined, and each matrix operation group is cached in the cache space in sequence. This caching method can make full use of the cache space of the current processing device. Finally, the operation optimization code is used to process the matrix operation group cached in the cache space each time to obtain the operation results corresponding to each matrix operation group; based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the reasoning request is determined. The above method can take into account the reuse of both input feature data and network parameter data in the calculation process, reduce the possibility of cache miss, improve the utilization rate of processing equipment, and speed up the response speed of inference requests. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 An application environment diagram of a model reasoning method in an embodiment;
[0047] Figure 2 is a flowchart of a model reasoning method in one embodiment;
[0048] Figure 3 is a schematic diagram of a first intermediate matrix in one embodiment;
[0049] Figure 4 is a schematic diagram of a second intermediate matrix in one embodiment;
[0050] Figure 5 is a schematic diagram of a first intermediate matrix segmentation process in one embodiment;
[0051] Figure 6 is a schematic diagram of a second intermediate matrix segmentation process in one embodiment;
[0052] Figure 7 A schematic diagram of register allocation in one embodiment;
[0053] Figure 8 is a schematic diagram of rearrangement of the first intermediate matrix in one embodiment;
[0054] Fig. 9 is a schematic diagram of rearrangement of the second intermediate matrix in one embodiment;
[0055] Fig.10 is a structural block diagram of a model reasoning device in one embodiment;
[0056] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0058] The model reasoning method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. The model reasoning method provided in the embodiment of the present application can be executed by the terminal 102 or the server 104 alone, or it can be executed by the terminal 102 and the server 104 in collaboration. Taking the execution by terminal 102 alone as an example, terminal 102 is provided with multiple processing devices. In response to the inference request, terminal 102 can obtain the operation optimization code matching the current processing device, and determine the input feature data of each network layer in the inference model corresponding to the inference request based on the input data carried by the inference request; then determine the intermediate matrix blocking information according to the data blocking information, the cache reference information and the cache attribute information of the cache space in the current processing device; for each network layer, according to the intermediate matrix blocking information, the input feature data and network parameter data of the corresponding network layer are segmented and processed to obtain the corresponding first intermediate matrices and second intermediate matrices; then, based on the first intermediate matrices and the second intermediate matrices, matrix operation groups are combined to obtain; each matrix operation group is cached in the cache space in turn, and the matrix operation group cached in the cache space each time is processed using the operation optimization code to obtain the operation results corresponding to each matrix operation group; based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined.
[0059] Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. .
[0060] In one embodiment, Figure 2 As shown, a model inference method is provided, which can be applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:
[0061] Step 202, in response to the inference request, obtain the operation optimization code matching the current processing device, and determine the input feature data of each network layer in the inference model corresponding to the inference request based on the input data carried by the inference request.
[0062] Among them, the terminal can install an application (APP for short), the APP can include an inference model, the object can trigger an inference request on the APP, the APP can process the inference request based on the inference model, obtain the inference result corresponding to the inference request, and display the inference result for the object to view.
[0063] The model inference method provided in the embodiment of the present application can be applied in any inference scenario, and the inference request is a model inference request in the corresponding inference scenario. The inference request carries the data required to be input in the corresponding inference scenario.
[0064] In a specific implementation, the APP may be a program related to image recognition, the inference model may be an image recognition model, the inference request may be an image recognition request, the inference request may carry an image to be recognized, and the inference result corresponding to the inference request may be an image recognition result for the image to be recognized.
[0065] It should be noted that image recognition is only one possible implementation scenario. The model reasoning method provided in the embodiment of the present application can also be applied in other scenarios, and the embodiment of the present application does not limit this.
[0066] Among them, the terminal includes multiple processing devices, and the multiple processing devices constitute a heterogeneous core. Among the multiple processing devices, devices with higher model levels can be called large cores, and devices with lower model levels can be called small cores. The terminal can switch between large and small cores according to actual processing scenarios.
[0067] Among them, after receiving the inference request, the terminal can use the processing device currently in use as the current processing device. The terminal can obtain the target attribute information of the current processing device, and based on the target attribute information, determine the operation optimization code that matches the current processing device from multiple pre-stored optimization codes.
[0068] Exemplarily, the target attribute information may include model information, version information, serial number information, etc. of the current processing device, which is not limited in the embodiments of the present application.
[0069] In a specific implementation, after obtaining the target attribute information of the current processing device, the terminal can match the target attribute information with the attribute information corresponding to each pre-stored optimization code, and use the successfully matched optimization code as the operation optimization code matched with the current processing device.
[0070] The terminal may use a preset feature detection tool to detect the architectural features of the corresponding processing devices for different processing devices, and generate an optimized code corresponding to the corresponding processing devices based on the architectural features.
[0071] In a specific implementation, the terminal can select the lowest-level processing device from different processing devices, obtain the number of registers corresponding to the processing device, and determine the data block information based on the statistical data of the number of registers and the number of channels. For different processing devices, after detecting the architectural characteristics of the corresponding processing device, the terminal generates the optimization code corresponding to the corresponding processing device based on the above data block information and the architectural characteristics.
[0072] Among them, the process of determining data block information and generating optimized code can be executed by the terminal, or by a developer writing software through code, or by a combination of terminal execution and manual execution, which is not limited in the embodiments of the present application.
[0073] The inference model corresponding to the inference request includes multiple network layers, and the terminal can determine the input feature data of each network layer based on the input data carried by the inference request. In a specific implementation, for each network layer in the inference model corresponding to the inference request, the output data of the previous network layer of the network layer can be used as the input feature data of the network layer.
[0074] For example, assume that the inference model corresponding to the inference request contains three network layers, and these three network layers are connected in the order of network layer 1, network layer 2, and network layer 3. For network layer 1, the input data carried by the inference request can be used as the input feature data of network layer 1. For network layer 2, the output data of network layer 1 can be used as the input feature data of network layer 2. For network layer 3, the output data of network layer 2 can be used as the input feature data of network layer 3.
[0075] Step 204, determining the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device.
[0076] Among them, for each network layer, the input feature data of the network layer and the network parameter data of the network layer are usually in matrix form, and the network parameter data of the network layer can be a weight matrix of the network layer. The input feature data of the network layer and the network parameter data can be multiplied to obtain the output data of the network layer. Since the amount of input feature data and network parameter data is relatively large, when calculating the product of the two, the input feature data can be divided into multiple matrices of intermediate size. For the convenience of explanation, these matrices are all called first matrices. In addition, the network parameter data can also be divided into multiple matrices of intermediate size. For the convenience of explanation, these matrices are all called second matrices. The first matrix and the second matrix can be cached in the cache space, and then each first matrix can be divided into multiple smaller matrices. For the convenience of explanation, these matrices are all called first matrix units. In addition, each second matrix can also be divided into multiple smaller matrices. For the convenience of explanation, these matrices are all called second matrix units. The first matrix unit and the second matrix unit can be stored in a register, and the corresponding thread multiplies the first matrix unit and the second matrix unit in the register to obtain the multiplication result of the first matrix unit and the second matrix unit. Based on the multiplication result of the matrix units, the multiplication result of the first matrix and the second matrix can be combined to obtain the multiplication result. Based on the multiplication result of the first matrix and the second matrix, the multiplication result of the input feature data and the network parameter data can be obtained.
[0077] The data block information includes the dimension information of the matrix unit corresponding to the input feature data and the dimension information of the matrix unit corresponding to the network parameter data. Exemplarily, the matrix unit corresponding to the input feature data may be a matrix of Mb×Kb, and the matrix unit corresponding to the network parameter data may be a matrix of Kb×Nb, then the data block information includes Mb, Kb and Nb. For the convenience of explanation, in the embodiment of the present application, Mb is referred to as row block information, Nb is referred to as column block information, and Kb is referred to as multiplication length information.
[0078] In a specific implementation, the terminal can select the lowest-level processing device from different processing devices, obtain the number of registers corresponding to the processing device, and determine the data block information based on the statistical data of the number of registers and the number of channels. Exemplarily, in the data block information, Mb can be 12, Kb can be 4, and Nb can be 8. In this case, the input feature data of the network layer corresponds to a matrix with a matrix unit of 12×4, and the network parameter data of the network layer corresponds to a matrix with a matrix unit of 4×8.
[0079] Among them, the cache space in the current processing device includes a first cache space and a second cache space, the access priority of the first cache space is greater than the access priority of the second cache space, the first cache space can be called L1 space, and the second cache space can be called L2 space.
[0080] The cache reference information may include first reference information of row direction data and second reference information of column direction data. The first reference information is used to indicate the amount of input feature data stored in the L1 space, and the second reference information is used to indicate the amount of network parameter data stored in the L2 space.
[0081] The cache attribute information is description information of the cache space, and illustratively, the cache attribute information includes space capacity, access time, etc. In the case where the cache space in the current processing device includes a first cache space and a second cache space, the cache attribute information includes at least the space capacity of the first cache space and the space capacity of the second cache space.
[0082] The terminal may determine the intermediate matrix block information according to the data block information, the cache reference information, the space capacity of the first cache space, and the space capacity of the second cache space.
[0083] Step 206 , for each network layer, according to the intermediate matrix block information, the input feature data and network parameter data of the corresponding network layer are segmented to obtain corresponding first intermediate matrices and second intermediate matrices.
[0084] Among them, the intermediate matrix blocking information includes column-wise blocking information and row-wise blocking information. The terminal can extract the row-wise blocking information from the intermediate matrix blocking information, and divide the input feature data of the network layer into multiple first intermediate matrices according to the row-wise blocking information; the terminal can extract the column-wise blocking information from the intermediate matrix blocking information, and divide the network parameter data of the network layer into multiple second intermediate matrices according to the column-wise blocking information.
[0085] For example, assume that the input feature data of the network layer is a 960×320 matrix, and the network parameter data of the network layer is a 320×480 matrix. Mb is 12, Kb is 4, Nb is 8, the row direction block information in the intermediate matrix block information is 4*Mb, and the column direction block information in the intermediate matrix block information is 6*Nb. Figure 3 As shown, starting from the first row of input feature data, each 4*Mb row of data forms a first intermediate matrix, and a total of 20 first intermediate matrices can be obtained. Similarly, see Figure 4 As shown, for the network parameter data of the network layer, starting from the first column of the network parameter data, every 6*Nb columns of data constitute a second intermediate matrix, and a total of 10 second intermediate matrices can be obtained.
[0086] Step 208, based on each first intermediate matrix and each second intermediate matrix, each matrix operation group is combined; each matrix operation group is cached in the cache space in sequence, and the matrix operation group cached in the cache space each time is processed using the operation optimization code to obtain the operation results corresponding to each matrix operation group.
[0087] Among them, for each network layer, the terminal can combine each first intermediate matrix and each second intermediate matrix corresponding to the network layer to obtain each matrix operation group, and then calculate the output data of the corresponding network layer based on each matrix operation group.
[0088] In a specific implementation, multiple first intermediate matrices and multiple second intermediate matrices may be combined in pairs, and each obtained combination is used as a matrix operation group.
[0089] Exemplarily, in combination with the above example, assuming that starting from the first row of the input feature data, every 4*Mb row of data constitutes a first intermediate matrix, a total of 20 first intermediate matrices can be obtained. Similarly, for the network parameter data of the network layer, starting from the first column of the network parameter data, every 6*Nb column of data constitutes a second intermediate matrix, and a total of 10 second intermediate matrices can be obtained. For the first first intermediate matrix from top to bottom in the input feature data, the combination of the first intermediate matrix and the first second intermediate matrix from left to right in the network parameter data can be used as the first matrix operation group, and the combination of the first intermediate matrix and the second second intermediate matrix from left to right in the network parameter data can be used as the second matrix operation group, and so on, until the combination of the first intermediate matrix and the 10th second intermediate matrix from left to right in the network parameter data is obtained, that is, the 10th matrix operation group. Similarly, for the second, third, and 20th first intermediate matrices from top to bottom in the input feature data, the combination processing in the above manner is performed, and 200 matrix operation groups can be obtained.
[0090] After obtaining each matrix operation group, the terminal can cache each matrix operation group in the cache space in sequence, and the cache order of each matrix operation group can be flexibly set according to the actual situation. For example, in combination with the above example, the cache can be performed in the order of the first matrix operation group → the second matrix operation group → the third matrix operation group → ... → the 200th matrix operation group.
[0091] For each cached matrix operation group in the cache space, based on the segmentation principle of the first intermediate matrix and the second intermediate matrix, the first intermediate matrix in the matrix operation group contains multiple Mb of data. Figure 3 For example, the first intermediate matrix contains (4*Mb)×320 data. Similarly, the second intermediate matrix in the matrix operation group contains multiple Nb data. Figure 4 For example, the first intermediate matrix contains 320×(6*Nb) data. Each Mb of data in the first intermediate matrix can be divided according to Kb, and K / Kb Mb×Kb matrix units can be obtained. Figure 5 For example, combined with Figure 3 For example, by dividing the first intermediate matrix 1 from the top to the first Mb, 320 / Kb Mb×Kb matrix units can be obtained. Similarly, by dividing the data of each Nb in the second intermediate matrix according to Kb, K / Kb Kb×Nb matrix units can be obtained. Figure 6 For example, combined with Figure 4 For example, the second intermediate matrix 1 is divided into the first Nb from left to right, and 320 / Kb Kb×Nb matrix units can be obtained. Based on each Mb×Kb matrix unit and each Kb×Nb matrix unit, each matrix unit operation group can be combined, and each matrix unit operation group can be sequentially transferred to the register of the current processing device. The matrix unit operation group stored in the register each time is processed using the operation optimization code to obtain the operation results of each matrix unit operation group, and the operation results of each matrix unit operation group can constitute the operation results of the corresponding matrix operation group.
[0092] Step 210, based on the operation results corresponding to each matrix operation group, determine the output data of the corresponding network layer, and based on the output data of each network layer, determine the inference result corresponding to the inference request.
[0093] After obtaining the operation results of each matrix operation group, the terminal combines the operation results of each matrix operation group to obtain the output data of the corresponding network layer. Figure 3 and Figure 4 In the example in , the output data of the network layer is a 960×480 matrix.
[0094] Among them, for each network layer in the inference model corresponding to the inference request, the above method can be used to determine the corresponding output data, and the output data of each network layer can be used as the input feature data of the next network layer. For the next network layer, the above method is used to determine the output data of the next network layer, and the output data of the next network layer can be used as the input feature data of the next network layer. This is repeated to obtain the output data of the last network layer. Based on the output data of the last network layer, the inference result corresponding to the inference request can be determined.
[0095] In the above embodiment, in response to an inference request, an operation optimization code matching the current processing device can be obtained, and based on the input data carried by the inference request, the input feature data of each network layer in the inference model corresponding to the inference request can be determined; in the process of calculating the output data of each network layer, the intermediate matrix blocking information can be determined according to the data blocking information, the cache reference information and the cache attribute information of the cache space in the current processing device. For each network layer, the network parameter data and the input feature data of the corresponding network layer can be segmented and processed according to the intermediate matrix blocking information to obtain the corresponding first intermediate matrices and second intermediate matrices; then, based on the first intermediate matrices and the second intermediate matrices, each matrix operation group is combined and each matrix operation group is cached in the cache space in turn. This caching method can make full use of the cache space of the current processing device. Finally, the operation optimization code is used to process the matrix operation group cached in the cache space each time to obtain the operation results corresponding to each matrix operation group; based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined. The above method can take into account the reuse of both input feature data and network parameter data in the calculation process, reduce the possibility of cache miss, improve the utilization rate of processing equipment, and speed up the response speed of inference requests.
[0096] In some embodiments, obtaining an operation optimization code that matches the current processing device includes: obtaining target attribute information of the current processing device; matching the target attribute information with the attribute information corresponding to each pre-stored optimization code, and using the successfully matched optimization code as the operation optimization code that matches the current processing device.
[0097] The target attribute information may include the model information, version information, serial number information, etc. of the current processing device. Each optimization code is generated in advance for different processing devices and stored in the terminal. The attribute information corresponding to the optimization code includes the device information of the corresponding processing device, and the device information includes the model information, version information, serial number information, etc.
[0098] Among them, after obtaining the target attribute information of the current processing device, the terminal can extract the model information from the target attribute information, match the model information with the model information corresponding to each pre-stored optimization code, and use the successfully matched optimization code as the operation optimization code matching the current processing device.
[0099] In the above embodiment, after receiving the inference request, the target attribute information of the current processing device is obtained; the target attribute information is matched with the attribute information corresponding to each pre-stored optimization code, and the successfully matched optimization code is used as the operation optimization code matched with the current processing device. The operation optimization code can be used to obtain the inference result corresponding to the inference request later. This method of dynamically adapting the optimization code can improve the utilization rate of the processing device and speed up the response speed of the inference request.
[0100] In some embodiments, the model inference method provided by the embodiments of the present application also includes: for different processing devices, using a preset feature detection tool to detect the architectural characteristics of the targeted processing device; determining a target processing device whose model level meets preset conditions from the different processing devices, and determining data blocking information based on the statistical data of the number of registers and the number of channels corresponding to the target processing device; generating an optimization code corresponding to the targeted processing device according to the architectural characteristics and the data blocking information, and storing the optimization code corresponding to each processing device.
[0101] The different processing devices may be processing devices of different models, and the preset characteristic detection tool may be a pre-written code for detecting the architectural characteristics of the processing device. Processing devices of different models have different processing capabilities, and processing devices with higher processing capabilities may be called high-end devices, and processing devices with lower processing capabilities may be called low-end devices. The target processing device may be the processing device with the lowest model level among the different processing devices.
[0102] Among them, the architectural characteristics may include the floating-point peak value of the processing device, instruction issuance width, instruction delay, instruction throughput, multi-instruction issuance limit, cache size at each level, cache delay at each level, and cache bandwidth at each level, etc., which are not limited in the embodiments of the present application.
[0103] Among them, each processing device has a certain number of registers, and the number of registers corresponding to each processing device can be obtained, from which the minimum number of registers can be obtained. The data blocking information can be determined based on the statistical data of the minimum number of registers and the number of channels. The data blocking information determined in this way is suitable for both low-end devices and high-end devices. Since the data blocking information needs to be combined in the subsequent process of generating optimized code, different processing devices use unified data blocking information to improve the generation efficiency of optimized code.
[0104] The channel statistics may include the number of input channels and output channels of the weight matrix corresponding to the common network layer. Since the number of input channels and the number of output channels are usually multiples of 8, such as 8, 16, 32, etc., Nb can be set to 8, so that data padding can be avoided, thereby reducing unnecessary data operations and improving operation efficiency.
[0105] When Nb is set to 8, in order to fully utilize the registers, Mb and Kb can be determined based on the number of registers.
[0106] For example, assuming that the number of registers corresponding to the low-end device is 32, based on the channel number statistics, Nb can be set to 8. In order to maximize the use of registers, based on the number of registers, multiple groups of Mb and Kb that meet the register throughput can be set, such as: Mb is set to 6, Kb is set to 4; Mb is set to 6, Kb is set to 8; Mb is set to 12, Kb is set to 4, etc. Test each group of data to find the combination with the highest utilization of low-end devices. After testing, the combination with the highest utilization is: Mb is 12, Kb is 4, and Nb is 8. Under this combination, the utilization of low-end devices can almost reach the floating point peak.
[0107] For example, for a low-end device with 32 registers, when Mb is 12, Kb is 4, and Nb is 8, the Mb×Kb matrix unit and the Kb×Nb matrix unit can be arranged in the following order: Figure 7 The method shown is placed into the register, Figure 7 In the example, each grid represents a register, each register can store 4 floating-point numbers, 3 registers can be used to store Mb×Kb matrix units, 4 registers can be used to store Kb×Nb matrix units, and 24 registers can be used to store the multiplication result of two matrix units. Specifically, registers V0-V2 can be used to store Mb×Kb matrix units, V3-V6 can be used to store Kb×Nb matrix units, and V8-V31 can be used to store the multiplication result of the above two matrix units. Operation code can be written based on this register setting.
[0108] In the process of generating operation codes for different processing devices, in addition to referring to the above register settings, it is also necessary to refer to the architectural features measured for the corresponding processing devices. For example, the instruction issuance bandwidth indicates how many instructions can be issued at the same time. This parameter can guide how to arrange instructions in the process of generating operation codes. The multi-instruction issuance limit indicates which instructions can be issued at the same time and which instructions cannot be issued at the same time. Based on this parameter, in the process of generating operation codes, instructions that can be issued at the same time are arranged together, and instructions that cannot be issued at the same time are arranged separately to avoid pauses in the processing device pipeline.
[0109] For low-end devices, the following countermeasures can be taken in the process of generating operation codes: no data dependency between adjacent instructions, sufficient unrelated instruction streams, loop unrolling in the K direction, no branch specialization, multiple soft pipelines, and increasing the proportion of calculation instructions. The operation codes generated based on these countermeasures can greatly improve the utilization rate of low-end devices. Operation codes can be used as a component of optimized code.
[0110] Among them, a code generation tool can be used to automatically generate the corresponding optimization code, or a developer can manually write the corresponding optimization code, which is not limited in the present embodiment of the application. After obtaining the optimization code corresponding to each processing device, the optimization code corresponding to each processing device can be stored in the terminal.
[0111] In the above embodiment, during the code development stage, a pre-written feature detection tool can be used to perform architectural feature detection on multiple processing devices of different models, so as to obtain the architectural features corresponding to each of the processing devices of different models. In order to take into account the processing capabilities of both high-end processing devices and low-end processing devices, the data block information is determined based on the statistical data of the number of registers and the number of channels corresponding to the low-end processing devices, and then based on the data block information and the architectural features corresponding to each of the different processing devices, the optimization codes corresponding to each of the different processing devices are generated. These multiple optimization codes can be stored in the terminal, so that after receiving the inference request, the terminal can find the operation optimization code that matches the current processing device from the multiple optimization codes, and then use the operation optimization code to calculate the inference result corresponding to the inference request. This method of dynamically adapting the optimization code can improve the utilization rate of the processing device and speed up the response speed of the inference request.
[0112] In some embodiments, the cache space in the current processing device includes a first cache space and a second cache space, the cache attribute information includes the space capacity of the first cache space and the space capacity of the second cache space, and the cache reference information includes first reference information for row-direction data and second reference information for column-direction data; the access priority of the first cache space is greater than the access priority of the second cache space; the intermediate matrix blocking information is determined according to the data blocking information, the cache reference information and the cache attribute information of the cache space in the current processing device, including: determining the column-direction blocking information according to the data blocking information, the first reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the first cache space; determining the row-direction blocking information according to the column-direction blocking information, the data blocking information, the second reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the second cache space; the column-direction blocking information and the row-direction blocking information constitute the intermediate matrix blocking information.
[0113] The cache space in the current processing device includes an L1 space and an L2 space. The first reference information is used to indicate the amount of input feature data stored in the L1 space, and the second reference information is used to indicate the amount of network parameter data stored in the L2 space. Exemplarily, when the first reference information is 2, it indicates that the L1 space stores at least 2*Mb of data, and when the second reference information is 2, it indicates that the L2 space stores at least 2 second intermediate matrices.
[0114] Among them, the terminal can first determine the column-wise blocking information according to the data blocking information, the first reference information, the number of columns of the input feature data of the corresponding network layer, and the spatial capacity of the first cache space; then determine the row-wise blocking information according to the column-wise blocking information, the data blocking information, the second reference information, the number of columns of the input feature data of the corresponding network layer, and the spatial capacity of the second cache space; finally, based on the column-wise blocking information and the row-wise blocking information, construct the intermediate matrix blocking information.
[0115] In the above embodiment, a method for determining the intermediate matrix blocking information is introduced. The input feature data and network parameter data are segmented and processed based on the intermediate matrix blocking information, and the obtained first intermediate matrices and second intermediate matrices are cached in the cache space of the current processing device. In this way, the space capacity of the cache space can be fully utilized, and the reuse rate of the two types of data can be improved, thereby accelerating the response speed of the inference request.
[0116] In some embodiments, column-wise blocking information is determined based on data blocking information, first reference information, the number of columns of input feature data of the corresponding network layer, and the spatial capacity of the first cache space, including: fusing the row blocking information, the first reference information, and the number of columns of the input feature data of the corresponding network layer in the data blocking information to obtain a first data volume of the input feature data that needs to be cached at least in the first cache space; fusing the first parameter to be requested, the column blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer to obtain a second data volume of network parameter data that can be cached in the first cache space; fusing the row blocking information, the first reference information, the first parameter to be requested, and the column blocking information in the data blocking information to obtain a third data volume of the operation result that can be cached in the first cache space; performing statistics on the first data volume, the second data volume, and the third data volume to obtain a first statistical result; establishing a first equation based on the first statistical result and the spatial capacity of the first cache space; obtaining a value of the first parameter to be requested according to the first equation, and using the value of the first parameter to be requested as the column-wise blocking information.
[0117] The fusion may be a positive correlation operation on the corresponding data, for example, the fusion may be a multiplication of the corresponding data. The statistics may be a gain operation on the corresponding data, for example, the statistics may be an addition, a weighted sum, etc. of the corresponding data. The embodiment of the present application does not limit this.
[0118] Among them, for the first cache space, it is necessary to store data in the input feature data, data in the network parameter data and intermediate operation results. The row block information in the data block information, the first reference information and the number of columns of the input feature data of the corresponding network layer can be multiplied to obtain a first multiplication result, and the first multiplication result represents the first data amount of the input feature data that needs to be cached at least in the first cache space; the first parameter to be requested, the column block information in the data block information and the number of columns of the input feature data of the corresponding network layer can be multiplied to obtain a second multiplication result, and the second multiplication result represents the network parameter data that can be cached in the first cache space. a second data amount of the data; multiplying the row blocking information, the first reference information, the first parameter to be found, and the column blocking information in the data blocking information to obtain a third multiplication result, the third multiplication result represents a third data amount of the operation result that can be cached in the first cache space, adding the first data amount, the second data amount, and the third data amount to obtain a first summation result, establishing a first equation based on the first summation result and the space capacity of the first cache space, since only the first parameter to be found is an unknown number, the value of the first parameter to be found can be determined based on the first equation, and the value of the first parameter to be found is used as the column direction blocking information.
[0119] Exemplarily, in the embodiment of the present application, the elements in the matrix are all floating point numbers, the row blocking information in the data blocking information is Mb, the column blocking information in the data blocking information is Nb, the first reference information is 2, the number of columns of the input feature data of the corresponding network layer is K, the first parameter to be determined is x, and the space capacity of the first cache space is Size1. Then, the first equation established is: ((2×Mb)×K+(x×Nb)×K+(2×Mb)×(x×Nb))×(bytes occupied by each floating point number)=Size1; the value of x can be obtained through the first equation, and the value of x can be used as the column direction blocking information.
[0120] In the above embodiment, a method for determining the column direction blocking information in the intermediate matrix blocking information is introduced. The intermediate matrix blocking information can be determined based on the column direction blocking information. Subsequently, the input feature data and network parameter data are segmented and processed based on the intermediate matrix blocking information, and the obtained first intermediate matrices and second intermediate matrices are cached in the cache space of the current processing device. In this way, the space capacity of the cache space can be fully utilized, and the reuse rate of the two types of data can be improved, thereby accelerating the response speed of the inference request.
[0121] In some embodiments, the row direction blocking information is determined according to the column direction blocking information, the data blocking information, the second reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the second cache space, including: fusing the second parameter to be required, the row blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer to obtain a fourth data volume of the input feature data that can be cached in the second cache space; fusing the second reference information, the column direction blocking information, the column blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer to obtain a fifth data volume of the network parameter data that needs to be cached at least in the second cache space; fusing the second parameter to be required, the row blocking information in the data blocking information, the second reference information, the column direction blocking information, and the column blocking information in the data blocking information to obtain a sixth data volume of the operation result that can be cached in the second cache space; counting the fourth data volume, the fifth data volume, and the sixth data volume to obtain a second statistical result; establishing a second equation based on the second statistical result and the space capacity of the second cache space; obtaining the value of the second parameter to be required according to the second equation, and using the value of the second parameter to be required as the row direction blocking information.
[0122] The fusion may be a positive correlation operation on the corresponding data, for example, the fusion may be a multiplication of the corresponding data. The statistics may be a gain operation on the corresponding data, for example, the statistics may be an addition, a weighted sum, etc. of the corresponding data. The embodiment of the present application does not limit this.
[0123] Among them, for the second cache space, the data that needs to be stored also includes: data in the input feature data, data in the network parameter data and intermediate calculation results. The second parameter to be required, the row blocking information in the data blocking information and the number of columns of the input feature data of the corresponding network layer can be multiplied to obtain a fourth multiplication result, and the fourth multiplication result represents the fourth data amount of the input feature data that can be cached in the second cache space; the second reference information, the column direction blocking information, the column blocking information in the data blocking information and the number of columns of the input feature data of the corresponding network layer can be multiplied to obtain a fifth multiplication result, and the fifth multiplication result represents the minimum amount of data that needs to be cached in the second cache space. a fifth data volume of network parameter data; multiplying the second parameter to be determined, the row blocking information in the data blocking information, the second reference information, the column direction blocking information and the column blocking information in the data blocking information to obtain a sixth multiplication result, the sixth multiplication result representing a sixth data volume of the calculation result that can be cached in the second cache space; adding the fourth data volume, the fifth data volume and the sixth data volume to obtain a second summation result; establishing a second equation based on the second summation result and the space capacity of the second cache space; since only the second parameter to be determined is an unknown number, the value of the second parameter to be determined can be determined based on the second equation, and the value of the second parameter to be determined is used as the row direction blocking information.
[0124] Exemplarily, in the embodiment of the present application, the elements in the matrix are all floating point numbers, the row blocking information in the data blocking information is Mb, the column blocking information in the data blocking information is Nb, the second reference information is 2, the number of columns of the input feature data of the corresponding network layer is K, the first parameter to be determined is y, and the space capacity of the first cache space is Size2. Then, the first equation established is: ((y×Mb)×K+(2×x×Nb)×K+(y×Mb)×(2×x×Nb))×(bytes occupied by each floating point number)=Size2; the value of y can be obtained through the first equation, and the value of y can be used as the row direction blocking information.
[0125] In the above embodiment, a method for determining the row direction blocking information in the intermediate matrix blocking information is introduced. The intermediate matrix blocking information can be determined based on the row direction blocking information. Subsequently, the input feature data and network parameter data are segmented and processed based on the intermediate matrix blocking information, and the obtained first intermediate matrices and second intermediate matrices are cached in the cache space of the current processing device. In this way, the space capacity of the cache space can be fully utilized, and the reuse rate of the two types of data can be improved, thereby accelerating the response speed of the inference request.
[0126] In some embodiments, each matrix operation group is cached in sequence into a cache space, including: for each matrix operation group, a first rearrangement code corresponding to the input feature data contained in the operation optimization code is used to cache the first intermediate matrix in the targeted matrix operation group into the cache space; a second rearrangement code corresponding to the network parameter data contained in the operation optimization code is used to cache the second intermediate matrix in the targeted matrix operation group into the cache space.
[0127] In the process of generating the corresponding optimization code for each processing device, the first rearrangement code corresponding to the input feature data can be generated according to the data block information and the rearrangement rule designed for the input feature data. The second rearrangement code corresponding to the network parameter data is generated according to the data block information and the rearrangement rule designed for the network parameter data. Finally, based on the first rearrangement code, the second rearrangement code and the above-mentioned operation code, the optimization code corresponding to the corresponding processing device is generated.
[0128] Among them, after generating optimization codes corresponding to different processing devices and storing these optimization codes in the terminal, after receiving the inference request and executing the processes of obtaining the operation optimization code, determining the input feature data of each network layer, determining the intermediate matrix block information, segmentation processing and combination processing, the terminal can cache the first intermediate matrix in the matrix operation group to be cached into the cache space by calling the first rearrangement code corresponding to the input feature data contained in the operation optimization code; cache the second intermediate matrix in the matrix operation group to be cached into the cache space by calling the second rearrangement code corresponding to the network parameter data contained in the operation optimization code.
[0129] In the above embodiment, after the terminal combines to obtain each matrix operation group, for each matrix operation group, the first intermediate matrix and the second intermediate matrix in the matrix operation group can be cached into the cache space by calling the first rearrangement code and the second rearrangement code in the operation optimization code. This method allows the data in the cache space to be accessed continuously, thereby improving the data hit rate in the cache space and accelerating the response speed of the inference request.
[0130] In some embodiments, the first intermediate matrix in the targeted matrix operation group is cached in the cache space by using the first rearrangement code corresponding to the input feature data in the operation optimization code, including: obtaining the number of columns of the input feature data of the corresponding network layer, and when the number of columns is greater than a preset threshold, dividing the first intermediate matrix in the targeted matrix operation group into multiple first matrix blocks according to the data blocking information; caching the multiple first matrix blocks in the cache space with columns as priority, wherein the elements in each first matrix block are cached in the cache space in a column-by-column manner.
[0131] Among them, the first rearrangement code corresponding to the input feature data in the operation optimization code caches the first intermediate matrix in the matrix operation group into the cache space in the following manner: first, the number of columns of the input feature data of the corresponding network layer is obtained, and the number of columns is compared with a preset threshold. When the number of columns is greater than the preset threshold, it is determined that the first intermediate matrix in the matrix operation group needs to be rearranged, and then the first intermediate matrix in the matrix operation group is divided into multiple first matrix blocks according to the data blocking information, and the multiple first matrix blocks are cached into the cache space with columns as priority, wherein the elements in each first matrix block are cached into the cache space in a column-by-column manner.
[0132] For example, assuming that the first intermediate matrix in a matrix operation group is Figure 8 As shown, the number of rows of the first intermediate matrix is Mb, and the number of columns is 2*Kb. The first intermediate matrix can be divided into two first matrix blocks, such as Figure 8In the first matrix block 1 and the first matrix block 2, since the column where the first matrix block 1 is located is before the column where the second matrix block 2 is located, the first matrix block 1 can be cached in the cache space first, and then the first matrix block 2 can be cached in the cache space. In the process of caching the first matrix block 1 in the cache space, the elements in the first matrix block 1 can be cached in the cache space in a column-by-column manner. Similarly, in the process of caching the first matrix block 2 in the cache space, the elements in the first matrix block 2 can be cached in the cache space in a column-by-column manner.
[0133] In the above embodiment, in the process of caching the first intermediate matrix in the matrix operation group into the cache space through the first rearrangement code, the number of columns of the input feature data of the corresponding network layer is first obtained. When the number of columns is greater than the preset threshold, the first intermediate matrix in the matrix operation group is divided into multiple first matrix blocks according to the data block information, and the multiple first matrix blocks are cached into the cache space with columns as priority, wherein the elements in each first matrix block are cached into the cache space in a column-by-column cache manner. This method allows the first intermediate matrix in the cache space to be continuously accessed, improves the data hit rate in the cache space, and speeds up the response speed of the inference request.
[0134] In some embodiments, the model inference method provided by the embodiments of the present application also includes: when the number of columns is less than or equal to a preset threshold, caching the first intermediate matrix in the targeted matrix operation group into the cache space in a row-by-row caching manner.
[0135] Among them, the first rearrangement code corresponding to the input feature data in the operation optimization code caches the first intermediate matrix in the matrix operation group into the cache space in the following manner: first, the number of columns of the input feature data of the corresponding network layer is obtained, and the number of columns is compared with a preset threshold. When the number of columns is less than or equal to the preset threshold, it is determined that there is no need to rearrange the first intermediate matrix in the matrix operation group, and the first intermediate matrix in the matrix operation group is cached into the cache space in a row-by-row manner.
[0136] Among them, the preset threshold can be flexibly set according to actual conditions. For example, the preset threshold can be set to 32, but this is only an example and the embodiments of the present application do not limit this.
[0137] In the above embodiment, in the process of caching the first intermediate matrix in the matrix operation group into the cache space through the first rearrangement code, the number of columns of the input feature data of the corresponding network layer is first obtained. When the number of columns is less than or equal to the preset threshold, it is not easy to cause a cache miss. Considering that the bandwidth of the low-end processing device is relatively tight, it is determined that there is no need to rearrange the first intermediate matrix in the matrix operation group. Then, the first intermediate matrix in the matrix operation group is cached into the cache space in a row-by-row manner, thereby saving bandwidth while ensuring the cache hit rate.
[0138] In some embodiments, the second intermediate matrix in the targeted matrix operation group is cached in the cache space by operating a second rearrangement code corresponding to the network parameter data in the operation optimization code, including: dividing the second intermediate matrix in the targeted matrix operation group into multiple second matrix blocks according to data blocking information; caching the multiple second matrix blocks in the cache space with columns as priority, wherein the elements in each second matrix block are cached in the cache space in a row-by-row manner.
[0139] Among them, the second rearrangement code corresponding to the network parameter data in the operation optimization code caches the second intermediate matrix in the matrix operation group into the cache space in the following way: according to the data blocking information, the second intermediate matrix in the matrix operation group is divided into multiple second matrix blocks, and the multiple second matrix blocks are cached into the cache space with columns as priority, wherein the elements in each second matrix block are cached into the cache space in a row-by-row manner.
[0140] For example, assuming that the second intermediate matrix in a matrix operation group is Fig. 9 As shown, the number of rows of the first intermediate matrix is 2*Kb, and the number of columns is 2*Nb. The first intermediate matrix can be divided into 4 second matrix blocks, such as Fig. 9 In the second matrix block 1, the second matrix block 2, the second matrix block 3, and the second matrix block 4, since the second matrix block 1 and the second matrix block 2 are in the same column, the second matrix block 3 and the second matrix block 4 are in the same column, and the column where the second matrix block 1 and the second matrix block 2 are located is before the column where the second matrix block 3 and the second matrix block 4 are located, the second matrix block 1 can be cached in the cache space first, and then the second matrix block 2 can be cached in the cache space, and then the second matrix block 3 can be cached in the cache space, and then the second matrix block 4 can be cached in the cache space. In the process of caching the second matrix block 1 in the cache space, the elements in the second matrix block 1 can be cached in the cache space in a row-by-row manner. Similarly, in the process of caching the second matrix block 2, the second matrix block 3, and the second matrix block 4 in the cache space, the elements can be cached in the cache space in a row-by-row manner.
[0141] In the above embodiment, in the process of caching the first intermediate matrix in the matrix operation group into the cache space through the first rearrangement code, the number of columns of the input feature data of the corresponding network layer is first obtained. When the number of columns is greater than the preset threshold, the first intermediate matrix in the matrix operation group is divided into multiple first matrix blocks according to the data block information, and the multiple first matrix blocks are cached into the cache space with columns as priority, wherein the elements in each first matrix block are cached into the cache space in a column-by-column cache manner. This method allows the second intermediate matrix in the cache space to be continuously accessed, improves the data hit rate in the cache space, and speeds up the response speed of the inference request.
[0142] In some embodiments, the inference request is an image recognition request, the input data carried by the inference request is an image to be recognized, and the inference result corresponding to the inference request is an image recognition result.
[0143] Among them, the terminal responds to the image recognition request, obtains the operation optimization code matching the current processing device, determines the input feature data of each network layer in the reasoning model corresponding to the image recognition request based on the image to be recognized carried by the reasoning request; determines the intermediate matrix blocking information according to the data blocking information, the cache reference information and the cache attribute information of the cache space in the current processing device; for each network layer, according to the intermediate matrix blocking information, the input feature data and network parameter data of the corresponding network layer are segmented and processed to obtain the corresponding first intermediate matrices and second intermediate matrices; based on the first intermediate matrices and the second intermediate matrices, each matrix operation group is combined; each matrix operation group is cached in the cache space in turn, and the matrix operation group cached in the cache space each time is processed using the operation optimization code to obtain the operation results corresponding to each matrix operation group; based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the image recognition result corresponding to the image recognition request is determined.
[0144] It should be noted that the application scenarios provided above are only used to illustrate the model reasoning of this application, and the application of the model reasoning provided by this application is not limited to the application scenarios provided above. For example, it can also be applied in natural language processing scenarios, data push scenarios, etc. This embodiment of the application does not limit this.
[0145] In the above embodiment, the model inference method is applied in the image recognition scenario, which can take into account the reuse of both input feature data and network parameter data in the calculation process, reducing the possibility of cache misses. In addition, dynamically adapting the calculation optimization code to match the current processing device can improve the utilization rate of the processing device and speed up the response speed of image recognition requests.
[0146] In some embodiments, a model reasoning method is provided, which is described by taking the method executed by a terminal as an example, and the method includes:
[0147] For different processing devices, a preset feature detection tool is used to detect the architectural features of the targeted processing devices; a target processing device whose model level meets the preset conditions is determined from the different processing devices, and data block information is determined based on the statistical data of the number of registers and the number of channels corresponding to the target processing device; based on the architectural features and the data block information, an optimization code corresponding to the targeted processing device is generated, and the optimization code corresponding to each processing device is stored.
[0148] In response to the inference request, the target attribute information of the current processing device is obtained; the target attribute information is matched with the attribute information corresponding to each pre-stored optimization code, and the successfully matched optimization code is used as the operation optimization code matched with the current processing device. Based on the input data carried by the inference request, the input feature data of each network layer in the inference model corresponding to the inference request is determined.
[0149] The cache space in the current processing device includes a first cache space and a second cache space, the cache attribute information includes the space capacity of the first cache space and the space capacity of the second cache space, the cache reference information includes first reference information of row direction data and second reference information of column direction data; the access priority of the first cache space is greater than the access priority of the second cache space.
[0150] Multiply the row block information, the first reference information and the number of columns of the input feature data of the corresponding network layer in the data block information to obtain a first data volume of the input feature data that needs to be cached at least in the first cache space; multiply the first parameter to be found, the column block information in the data block information and the number of columns of the input feature data of the corresponding network layer to obtain a second data volume of the network parameter data that can be cached in the first cache space; multiply the row block information, the first reference information, the first parameter to be found and the column block information in the data block information to obtain a third data volume of the operation result that can be cached in the first cache space; add the first data volume, the second data volume and the third data volume to obtain a first summation result; establish a first equation based on the first summation result and the space capacity of the first cache space; obtain the value of the first parameter to be found according to the first equation, and use the value of the first parameter to be found as the column-wise block information.
[0151] The fourth amount of input feature data that can be cached in the second cache space is obtained by multiplying the second parameter to be determined, the row block information in the data block information, and the number of columns of the input feature data of the corresponding network layer; the fifth amount of network parameter data that needs to be cached at least in the second cache space is obtained by multiplying the second reference information, the column direction block information, the column block information in the data block information, and the number of columns of the input feature data of the corresponding network layer; the sixth amount of operation results that can be cached in the second cache space is obtained by multiplying the second parameter to be determined, the row block information in the data block information, the second reference information, the column direction block information, and the column block information in the data block information; the fourth amount of data, the fifth amount of data, and the sixth amount of data are added to obtain the second summation result; the second equation is established based on the second summation result and the space capacity of the second cache space; the value of the second parameter to be determined is obtained according to the second equation, and the value of the second parameter to be determined is used as the row direction block information. The column direction block information and the row direction block information constitute the intermediate matrix block information.
[0152] For each network layer, the input feature data and network parameter data of the corresponding network layer are segmented according to the intermediate matrix block information to obtain corresponding first intermediate matrices and second intermediate matrices.
[0153] Based on each first intermediate matrix and each second intermediate matrix, each matrix operation group is obtained by combination; for each matrix operation group, the number of columns of the input feature data of the corresponding network layer is obtained, and when the number of columns is greater than the preset threshold, the first intermediate matrix in the matrix operation group is divided into multiple first matrix blocks according to the data blocking information; the multiple first matrix blocks are cached in the cache space with columns as priority, wherein the elements in each first matrix block are cached in the cache space in a column-by-column manner. When the number of columns is less than or equal to the preset threshold, the first intermediate matrix in the matrix operation group is cached in the cache space in a row-by-row manner. According to the data blocking information, the second intermediate matrix in the matrix operation group is divided into multiple second matrix blocks; the multiple second matrix blocks are cached in the cache space with columns as priority, wherein the elements in each second matrix block are cached in the cache space in a row-by-row manner.
[0154] The operation optimization code is used to process the matrix operation groups cached in the cache space each time to obtain the operation results corresponding to each matrix operation group; based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined; based on the output data of each network layer, the inference result corresponding to the inference request is determined.
[0155] In the above embodiment, in response to an inference request, an operation optimization code matching the current processing device can be obtained, and based on the input data carried by the inference request, the input feature data of each network layer in the inference model corresponding to the inference request can be determined; in the process of calculating the output data of each network layer, the intermediate matrix blocking information can be determined according to the data blocking information, the cache reference information and the cache attribute information of the cache space in the current processing device. For each network layer, the network parameter data and the input feature data of the corresponding network layer can be segmented and processed according to the intermediate matrix blocking information to obtain the corresponding first intermediate matrices and second intermediate matrices; then, based on the first intermediate matrices and the second intermediate matrices, each matrix operation group is combined and each matrix operation group is cached in the cache space in turn. This caching method can make full use of the cache space of the current processing device. Finally, the operation optimization code is used to process the matrix operation group cached in the cache space each time to obtain the operation results corresponding to each matrix operation group; based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined. The above method can take into account the reuse of both input feature data and network parameter data during the calculation process, reducing the possibility of cache miss. In addition, dynamically adapting the calculation optimization code to match the current processing device can improve the utilization rate of the processing device and speed up the response speed of the inference request.
[0156] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0157] Based on the same inventive concept, the embodiment of the present application also provides a model reasoning device for implementing the model reasoning method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more model reasoning device embodiments provided below can refer to the limitations on the model reasoning method above, and will not be repeated here.
[0158] In one embodiment, Fig.10 As shown, a model reasoning device is provided, comprising:
[0159] An acquisition module, used to respond to an inference request, acquire an operation optimization code matching the current processing device, and determine input feature data of each network layer in the inference model corresponding to the inference request based on input data carried by the inference request;
[0160] A first determination module is used to determine the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device;
[0161] A segmentation module is used to segment the input feature data and network parameter data of the corresponding network layer according to the intermediate matrix block information for each network layer, so as to obtain corresponding first intermediate matrices and second intermediate matrices;
[0162] The operation module is used to combine and obtain each matrix operation group based on each first intermediate matrix and each second intermediate matrix; cache each matrix operation group in the cache space in sequence, and use the operation optimization code to process the matrix operation group cached in the cache space each time to obtain the operation results corresponding to each matrix operation group;
[0163] The second determination module is used to determine the output data of the corresponding network layer based on the operation results corresponding to each matrix operation group, and determine the inference result corresponding to the inference request based on the output data of each network layer.
[0164] In some embodiments, the acquisition module is specifically used to: obtain target attribute information of the current processing device; match the target attribute information with the attribute information corresponding to each pre-stored optimization code, and use the successfully matched optimization code as the operation optimization code matched with the current processing device.
[0165] In some embodiments, the acquisition module is also used to: for different processing devices, use a preset feature detection tool to detect the architectural features of the targeted processing devices; determine the target processing devices whose model levels meet preset conditions from the different processing devices, and determine the data block information based on the statistical data of the number of registers and the number of channels corresponding to the target processing devices; generate the optimization code corresponding to the targeted processing devices according to the architectural features and the data block information, and store the optimization code corresponding to each processing device.
[0166] In some embodiments, the cache space in the current processing device includes a first cache space and a second cache space, the cache attribute information includes the space capacity of the first cache space and the space capacity of the second cache space, and the cache reference information includes first reference information for row-direction data and second reference information for column-direction data; the access priority of the first cache space is greater than the access priority of the second cache space; the first determination module is specifically used to: determine the intermediate matrix blocking information according to the data blocking information, the cache reference information and the cache attribute information of the cache space in the current processing device, including: determining the column-direction blocking information according to the data blocking information, the first reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the first cache space; determining the row-direction blocking information according to the column-direction blocking information, the data blocking information, the second reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the second cache space; the column-direction blocking information and the row-direction blocking information constitute the intermediate matrix blocking information.
[0167] In some embodiments, the first determination module is specifically used to: merge the row block information, the first reference information, and the number of columns of the input feature data of the corresponding network layer in the data block information to obtain the first data volume of the input feature data that needs to be cached at least in the first cache space; merge the first parameter to be required, the column block information in the data block information, and the number of columns of the input feature data of the corresponding network layer to obtain the second data volume of the network parameter data that can be cached in the first cache space; merge the row block information, the first reference information, the first parameter to be required, and the column block information in the data block information to obtain the third data volume of the calculation result that can be cached in the first cache space; count the first data volume, the second data volume, and the third data volume to obtain a first statistical result; establish a first equation based on the first statistical result and the space capacity of the first cache space; obtain the value of the first parameter to be required according to the first equation, and use the value of the first parameter to be required as the column direction block information.
[0168] In some embodiments, the first determination module is specifically used to: merge the second parameter to be required, the row blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer to obtain a fourth data volume of the input feature data that can be cached in the second cache space; merge the second reference information, the column direction blocking information, the column blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer to obtain a fifth data volume of the network parameter data that needs to be cached at least in the second cache space; merge the second parameter to be required, the row blocking information in the data blocking information, the second reference information, the column direction blocking information, and the column blocking information in the data blocking information to obtain a sixth data volume of the calculation result that can be cached in the second cache space; count the fourth data volume, the fifth data volume, and the sixth data volume to obtain a second statistical result; establish a second equation based on the second statistical result and the space capacity of the second cache space; obtain the value of the second parameter to be required according to the second equation, and use the value of the second parameter to be required as the row direction blocking information.
[0169] In some embodiments, the second determination module is specifically used to: for each matrix operation group, cache the first intermediate matrix in the targeted matrix operation group into the cache space through the first rearrangement code corresponding to the input feature data contained in the operation optimization code; cache the second intermediate matrix in the targeted matrix operation group into the cache space through the second rearrangement code corresponding to the network parameter data contained in the operation optimization code.
[0170] In some embodiments, the second determination module is specifically used to: obtain the number of columns of the input feature data of the corresponding network layer, and when the number of columns is greater than a preset threshold, divide the first intermediate matrix in the targeted matrix operation group into multiple first matrix blocks according to the data blocking information; cache the multiple first matrix blocks into the cache space with column priority, wherein the elements in each first matrix block are cached into the cache space in a column-by-column manner.
[0171] In some embodiments, the second determination module is specifically configured to: when the number of columns is less than or equal to a preset threshold, cache the first intermediate matrix in the targeted matrix operation group into the cache space in a row-by-row cache manner.
[0172] In some embodiments, the second determination module is specifically used to: divide the second intermediate matrix in the targeted matrix operation group into multiple second matrix blocks according to the data blocking information; cache the multiple second matrix blocks into the cache space with columns as priority, wherein the elements in each second matrix block are cached into the cache space in a row-by-row manner.
[0173] In some embodiments, the inference request is an image recognition request, the input data carried by the inference request is an image to be recognized, and the inference result corresponding to the inference request is an image recognition result.
[0174] Each module in the above model reasoning device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.
[0175] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a model reasoning method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.
[0176] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0177] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0178] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0179] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0180] It should be noted that the object information (including but not limited to images to be identified, object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0182] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0183] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A model reasoning method, It is characterized in that The method comprises: In response to the inference request, obtain the operation optimization code matching the current processing device, and determine the input feature data of each network layer in the inference model corresponding to the inference request based on the input data carried by the inference request; Determine the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device; For each network layer, according to the intermediate matrix block information, the input feature data and network parameter data of the corresponding network layer are segmented to obtain corresponding first intermediate matrices and second intermediate matrices; Based on each first intermediate matrix and each second intermediate matrix, each matrix operation group is combined to obtain; each matrix operation group is sequentially cached in the cache space, and the matrix operation group cached in the cache space each time is processed using the operation optimization code to obtain the operation results corresponding to each matrix operation group; Based on the operation results corresponding to each matrix operation group, the output data of the corresponding network layer is determined, and based on the output data of each network layer, the inference result corresponding to the inference request is determined.
2. The method according to claim 1, It is characterized in that The step of obtaining the operation optimization code matching the current processing device includes: Acquire target attribute information of the current processing device; The target attribute information is matched with the attribute information corresponding to each pre-stored optimization code, and the successfully matched optimization code is used as the operation optimization code matched with the current processing device.
3. The method according to claim 1, It is characterized in that The method further comprises: For different processing devices, a preset feature detection tool is used to detect the architecture characteristics of the processing device; Determine a target processing device whose model level meets preset conditions from the different processing devices, and determine data block information based on the statistical data of the number of registers and the number of channels corresponding to the target processing device; According to the architecture characteristics and the data block information, an optimization code corresponding to the targeted processing device is generated, and the optimization code corresponding to each processing device is stored.
4. The method according to claim 1, It is characterized in that The cache space in the current processing device includes a first cache space and a second cache space, the cache attribute information includes a space capacity of the first cache space and a space capacity of the second cache space, and the cache reference information includes first reference information of row direction data and second reference information of column direction data; The access priority of the first cache space is greater than the access priority of the second cache space; The determining of the intermediate matrix block information according to the data block information, the cache reference information and the cache attribute information of the cache space in the current processing device comprises: Determine column-wise blocking information according to data blocking information, the first reference information, the number of columns of input feature data of the corresponding network layer, and the space capacity of the first cache space; The row direction blocking information is determined according to the column direction blocking information, the data blocking information, the second reference information, the number of columns of the input feature data of the corresponding network layer and the space capacity of the second cache space; the column direction blocking information and the row direction blocking information constitute the intermediate matrix blocking information.
5. The method according to claim 4, It is characterized in that The determining the column direction blocking information according to the data blocking information, the first reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the first cache space includes: Merging the row block information in the data block information, the first reference information, and the number of columns of the input feature data of the corresponding network layer to obtain a first data amount of the input feature data that needs to be cached at least in the first cache space; The first parameter to be required, the column block information in the data block information, and the number of columns of the input feature data of the corresponding network layer are merged to obtain a second data volume of the network parameter data that can be cached in the first cache space; Merging the row block information in the data block information, the first reference information, the first parameter to be requested, and the column block information in the data block information to obtain a third data volume of the operation result that can be cached in the first cache space; Counting the first data volume, the second data volume, and the third data volume to obtain a first statistical result; A first equation is established based on the first statistical result and the space capacity of the first cache space; a value of a first parameter to be determined is obtained according to the first equation, and the value of the first parameter to be determined is used as column-wise blocking information.
6. The method according to claim 4, It is characterized in that The determining the row direction blocking information according to the column direction blocking information, the data blocking information, the second reference information, the number of columns of the input feature data of the corresponding network layer, and the space capacity of the second cache space includes: The second parameter to be requested, the row block information in the data block information, and the number of columns of the input feature data of the corresponding network layer are merged to obtain a fourth data amount of the input feature data that can be cached in the second cache space; The second reference information, the column direction blocking information, the column blocking information in the data blocking information, and the number of columns of the input feature data of the corresponding network layer are merged to obtain a fifth data amount of the network parameter data that needs to be cached at least in the second cache space; The second parameter to be requested, the row block information in the data block information, the second reference information, the column direction block information, and the column block information in the data block information are merged to obtain a sixth data amount of the operation result that can be cached in the second cache space; Counting the fourth data volume, the fifth data volume, and the sixth data volume to obtain a second statistical result; A second equation is established based on the second statistical result and the space capacity of the second cache space; a value of a second parameter to be determined is obtained according to the second equation, and the value of the second parameter to be determined is used as row-wise blocking information.
7. The method according to claim 1, It is characterized in that The step of sequentially caching each matrix operation group into the cache space includes: For each matrix operation group, the first intermediate matrix in the matrix operation group is cached in the cache space by using the first rearrangement code corresponding to the input feature data contained in the operation optimization code; The second intermediate matrix in the targeted matrix operation group is cached in the cache space through the second rearrangement code corresponding to the network parameter data contained in the operation optimization code.
8. The method according to claim 7, It is characterized in that The first rearrangement code corresponding to the input feature data included in the operation optimization code caches the first intermediate matrix in the matrix operation group to the cache space, including: Obtaining the number of columns of the input feature data of the corresponding network layer, and when the number of columns is greater than a preset threshold, dividing the first intermediate matrix in the targeted matrix operation group into a plurality of first matrix blocks according to the data block information; The plurality of first matrix blocks are cached in the cache space with column priority, wherein elements in each first matrix block are cached in the cache space in a column-by-column manner.
9. The method according to claim 8, It is characterized in that The method further comprises: When the number of columns is less than or equal to a preset threshold, the first intermediate matrix in the targeted matrix operation group is cached into the cache space in a row-by-row manner.
10. The method according to claim 7, It is characterized in that The second rearrangement code corresponding to the network parameter data included in the operation optimization code caches the second intermediate matrix in the targeted matrix operation group into the cache space, including: According to the data block information, the second intermediate matrix in the targeted matrix operation group is divided into a plurality of second matrix blocks; The plurality of second matrix blocks are cached in the cache space with column priority, wherein elements in each second matrix block are cached in the cache space in a row-by-row manner.
11. The method according to any one of claims 1 to 10, It is characterized in that The inference request is an image recognition request, the input data carried by the inference request is an image to be recognized, and the inference result corresponding to the inference request is an image recognition result.
12. A model reasoning device, It is characterized in that The device comprises: An acquisition module, used to respond to an inference request, acquire an operation optimization code matching the current processing device, and determine input feature data of each network layer in the inference model corresponding to the inference request based on input data carried by the inference request; A first determination module, configured to determine intermediate matrix block information according to data block information, cache reference information, and cache attribute information of a cache space in the current processing device; A segmentation module is used to segment the input feature data and network parameter data of the corresponding network layer according to the intermediate matrix block information for each network layer, so as to obtain corresponding first intermediate matrices and second intermediate matrices; An operation module is used to combine and obtain each matrix operation group based on each first intermediate matrix and each second intermediate matrix; cache each matrix operation group in the cache space in sequence, and use the operation optimization code to process the matrix operation group cached in the cache space each time to obtain the operation results corresponding to each matrix operation group; The second determination module is used to determine the output data of the corresponding network layer based on the operation results corresponding to each matrix operation group, and determine the inference result corresponding to the inference request based on the output data of each network layer.
13. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.
14. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.
15. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.
