Methods, apparatus, equipment and storage media for accelerating network model inference

Abstract

This application relates to the field of artificial intelligence technology, and provides a method, apparatus, device, and storage medium for accelerating network model inference. The method includes: detecting the presence of a graphics processing unit (GPU) when the network model begins executing an inference task; if a GPU is present, determining the target computational structure in the network model's file; the target computational structure includes matrix multiplication; determining the target tensor associated with the target computational structure; modifying the source program format of the target tensor according to the preset unified computing device architecture parallel program format requirements when the source program format requirements of the target tensor do not meet the requirements; and performing prediction inference on the inference task based on the modified target tensor source program using the GPU. This method can transform the tensor associated with the target computational structure into a GPU-accelerated tensor. It can automatically adapt to GPUs for accelerated inference, thereby improving the convenience of accelerating network model inference.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, device and storage medium for accelerating network model inference. Background Technology

[0002] In recent years, deep learning algorithms have been widely used in various intelligent fields. Typically, deep learning network models need to be trained first. After the training phase is completed, the model will be deployed to real business scenarios. The model files, such as the network model structure and parameters, need to be deployed to persist the model files and provide prediction inference services / program loading and running for online / offline prediction.

[0003] The prediction process is a crucial step in the actual deployment of the model, generating persistent pth format model parameter files for use in the prediction process. Since deep learning network models are data-processing-centric and involve a large number of computational operations, traditional central processing units (CPUs) often struggle to meet these requirements for inference and prediction. Therefore, graphics processing units (GPUs) are sometimes needed for inference and prediction. However, GPU inference and prediction require writing services / programs specifically for each model, manually specifying which steps in the network model need to be stored and computed on the GPU. This necessitates programmers constantly adjusting the code to accelerate inference, resulting in low convenience for accelerating network model inference. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for accelerating network model inference, aiming to solve the problem of low convenience in existing methods for accelerating network model inference.

[0005] In a first aspect, embodiments of this application provide a method for accelerating network model inference, including:

[0006] When the network model begins to perform the inference task, the presence of a graphics processor is detected;

[0007] When the graphics processor is present, the target computational structure in the file of the network model is determined; the target computational structure includes matrix multiplication.

[0008] Determine the tensor associated with the target computational structure as the target tensor, the target tensor comprising a multidimensional matrix of elements of the same data type;

[0009] Detect whether the source program format of the target tensor meets the preset requirements of the parallel program format of the unified computing device architecture;

[0010] When the source program format requirements of the target tensor do not meet the preset unified computing device architecture parallel program format requirements, the source program format of the target tensor shall be modified in accordance with the preset unified computing device architecture parallel program format requirements;

[0011] The graphics processor performs predictive inference on the inference task based on the source program of the modified target tensor.

[0012] In one embodiment, determining the tensor associated with the target computational structure as the target tensor includes:

[0013] Determine the tensors associated with the target computational structure;

[0014] Determine the dimensions of the tensors associated with the target computational structure;

[0015] If the dimension of any tensor associated with the target computational structure is greater than a first preset threshold, then all tensors associated with the target computational structure are determined as the target tensor.

[0016] In one embodiment, after determining the dimension of the tensor associated with the target computational structure, the method further includes:

[0017] If any two tensors associated with the target computational structure have dimensions greater than a second preset threshold, then all tensors associated with the target computational structure are identified as the target tensor; wherein the second preset threshold is less than the first preset threshold.

[0018] In one embodiment, determining the tensor associated with the target computational structure as the target tensor includes:

[0019] Determine the tensors associated with the target computational structure;

[0020] Determine the dimensions of the tensors associated with the target computational structure;

[0021] Determine the computing power value of the graphics processor;

[0022] Based on a preset relationship mapping table, obtain the target threshold corresponding to the computing power value;

[0023] If the dimension of any tensor associated with the target computational structure is greater than the target threshold, then all tensors associated with the target computational structure are identified as the target tensor.

[0024] In one embodiment, after detecting the presence of a graphics processor while the network model is performing an inference task, the process includes:

[0025] In the absence of the graphics processor, detect whether there is a tensor corresponding to the parallel program format of the unified device architecture;

[0026] If a tensor corresponding to the unified device architecture parallel program format exists, then the source program format for calculating the tensor corresponding to the unified device architecture parallel program format is modified to the source program format for calculating the CPU computing tensor.

[0027] In one embodiment, after detecting the presence of a graphics processor when the network model performs an inference task, the method further includes:

[0028] When the graphics processor is present and its computing power is less than a second preset threshold, detect whether there is a tensor corresponding to the parallel program format of the unified device architecture.

[0029] If a tensor corresponding to the unified device architecture parallel program format exists, then the source program format for calculating the tensor corresponding to the unified device architecture parallel program format is modified to the source program format for calculating the CPU computing tensor.

[0030] In one embodiment, detecting the presence of a graphics processor when the network model begins performing an inference task includes:

[0031] When the network model starts executing the inference task, it obtains device information of the terminal device, including the processor.

[0032] If the terminal device contains device information that matches the device information of the graphics processor, then it is determined that the graphics processor exists.

[0033] If no device information matching the graphics processor's device information exists in the terminal device, then it is determined that the graphics processor does not exist.

[0034] Secondly, embodiments of this application provide an apparatus for accelerating network model inference, applied to a terminal device for deploying a network model, the apparatus comprising:

[0035] The acquisition module is used to detect the presence of a graphics processor when the network model starts performing an inference task;

[0036] The first determining module is used to determine the target computational structure in the file of the network model when the graphics processor is present; the target computational structure includes matrix multiplication.

[0037] The second determining module is used to determine the tensor associated with the target computational structure as the target tensor, the target tensor comprising a multidimensional matrix of elements of the same data type;

[0038] The first detection module is used to detect whether the source program format of the target tensor meets the requirements of the preset computing unified device architecture parallel program format;

[0039] The first modification module is used to modify the source program format of the target tensor according to the preset unified computing device architecture parallel program format requirements when the source program format requirements of the target tensor do not meet the preset unified computing device architecture parallel program format requirements.

[0040] The predictive inference module is used by the graphics processor to perform predictive inference on the inference task based on the source program of the modified target tensor.

[0041] In one embodiment, the second determining module includes:

[0042] The first determining unit is used to determine the tensors associated with the target computational structure;

[0043] The second determining unit is used to determine the dimension of the tensor associated with the target computational structure;

[0044] The third determining unit is used to determine all tensors associated with the target computing structure as the target tensor when the dimension of any tensor among all tensors associated with the target computing structure is greater than a first preset threshold.

[0045] In one embodiment, the second determining module further includes:

[0046] The fourth determining unit is configured to determine all tensors associated with the target computing structure as the target tensor when any two tensors among all tensors associated with the target computing structure have dimensions greater than a second preset threshold; wherein the second preset threshold is less than the first preset threshold.

[0047] In another embodiment, the second determining module includes:

[0048] The fifth determining unit is used to determine the tensors associated with the target computational structure;

[0049] The sixth determining unit is used to determine the dimension of the tensor associated with the target computational structure;

[0050] The seventh determining unit is used to determine the computing power value of the graphics processor;

[0051] The acquisition unit is used to acquire a target threshold corresponding to the computing power value based on a preset relationship mapping table;

[0052] The eighth determining unit is used to determine all tensors associated with the target computing structure as the target tensor when the dimension of any tensor among all tensors associated with the target computing structure is greater than the target threshold.

[0053] In one embodiment, the apparatus further includes:

[0054] The second detection module is used to detect whether there is a tensor corresponding to the parallel program format of the computing unified device architecture when the graphics processor is not present.

[0055] The second modification module is used to modify the source program format of the tensor corresponding to the parallel program format of the unified device architecture to the source program format of the CPU computing tensor if there is a tensor.

[0056] In one embodiment, the apparatus further includes:

[0057] The third detection module is used to detect whether there is a tensor corresponding to the parallel program format of the computing unified device architecture when the graphics processor is present and the computing power of the graphics processor is less than the second preset threshold.

[0058] The third modification module is used to modify the source program format of the tensor corresponding to the parallel program format of the unified device architecture to the source program format of the CPU computing tensor if there is a tensor.

[0059] In one embodiment, the acquisition module specifically includes:

[0060] The acquisition unit is used to acquire device information of the terminal device, including the processor, when the network model starts executing the inference task.

[0061] The first determination unit is used to determine that the graphics processor exists if there is device information in the terminal device that matches the device information of the graphics processor.

[0062] The second determination unit is used to determine that the graphics processor does not exist if there is no device information in the terminal device that matches the device information of the graphics processor.

[0063] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for accelerating network model inference.

[0064] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for accelerating network model inference.

[0065] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the steps of the above-described method for accelerating network model inference.

[0066] The beneficial effects of this application's embodiments compared to existing technologies are as follows: This application can detect the presence of a graphics processing unit (GPU) when the network model begins executing inference tasks; and if a GPU is present, it determines the target computational structure in the network model's file; the target computational structure includes matrix multiplication; it determines the tensor associated with the target computational structure as the target tensor, which includes a multidimensional matrix of elements of the same data type; it checks whether the source program format requirements of the target tensor meet the preset unified computing device architecture parallel program format requirements; then, if the source program format requirements of the target tensor do not meet the preset unified computing device architecture parallel program format requirements, it modifies the source program format of the target tensor according to the preset unified computing device architecture parallel program format requirements; and it performs prediction inference based on the modified target tensor source program using the GPU. Because the tensors associated with the target computational structure are transformed into GPU-accelerated tensors, they can automatically adapt to GPUs for accelerated inference, thereby improving the convenience of accelerating inference for network models.

[0067] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart illustrating the method for accelerating network model inference provided in Embodiment 1 of this application;

[0070] Figure 2 This is a flowchart illustrating the method for accelerating network model inference provided in Embodiment 2 of this application;

[0071] Figure 3 This is a flowchart illustrating the method for accelerating network model inference provided in Embodiment 3 of this application;

[0072] Figure 4 This is a schematic diagram of the structure of the network model inference acceleration device provided in Embodiment 4 of this application;

[0073] Figure 5 This is a schematic diagram of the terminal device provided in Embodiment 5 of this application. Detailed Implementation

[0074] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0075] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0076] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0077] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0078] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0079] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0080] The method for accelerating network model inference provided in this application can be applied to terminal devices, which may be personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application does not impose any restrictions on the specific type of terminal device.

[0081] To illustrate the technical solution described in this application, specific embodiments are described below.

[0082] Example 1

[0083] Please see Figure 1 This document illustrates a schematic flowchart of a method for accelerating network model inference provided in an embodiment of this application. The method for accelerating network model inference includes:

[0084] Step S101: When the network model starts performing the inference task, detect whether a graphics processor is present.

[0085] Specifically, deep learning-based network models have been widely applied in the field of machine vision. Using these models to perform tasks such as image classification and object recognition requires two phases. The first phase involves building the network model and adjusting its parameters to achieve good performance suitable for the current application scenario. This process is called model training. Training steps include data organization, feature extraction, model construction, parameter learning, and evaluation of model accuracy and generalization error. After training, the model is deployed to real-world business scenarios. This requires deploying the trained network model, persisting its structure and parameters to a model file for loading and running by prediction and inference services / programs for online / offline predictions, such as image classification or object recognition. The training and inference tasks differ in the following ways: network model parameters need continuous updating during training, but remain constant during inference; training is a one-time event, while inference is repeated. Training and testing datasets need to be prepared and labeled in advance and used repeatedly during training, while inference involves a constant influx of new test data. For example, a neural network model can be built based on the PyTorch framework. After training, optimization, and debugging, the neural network model under the PyTorch framework can generate a model file in .pth format based on the source code. Deep learning algorithms are centered on data processing, which includes a large number of computational operations. Since most of the transistors used in CPUs are used to build high-speed cache memory and control units, the parts responsible for logical operations are not abundant. Therefore, the efficiency of using CPUs to process many computational operations is relatively low. Unlike CPUs, GPUs have a large number of logical computing units in their internal structure. Therefore, GPUs can be used to accelerate the model prediction and inference process. Deep learning uses CPUs for processing by default. When the network model starts executing the inference task, it can be determined whether a GPU is present in the current execution terminal. If a GPU is present, it can be used for processing.

[0086] In one embodiment, detecting the presence of a graphics processor when the network model begins to execute an inference task includes: when the network model begins to execute an inference task, acquiring device information of the terminal device including the processor; if device information matching the graphics processor's device information exists in the terminal device, then it is determined that the graphics processor exists; if device information matching the graphics processor's device information does not exist in the terminal device, then it is determined that the graphics processor does not exist.

[0087] Specifically, when the network model starts executing the inference task, it reads the device information of the terminal device, including the processor. The device information includes, but is not limited to, the processor's model, device ID, and name. When the device information obtained from the terminal device is the same as the pre-stored GPU device information, it is determined that a GPU processor is present.

[0088] Step S102: When the graphics processor is present, determine the target computational structure in the file of the network model; the target computational structure includes matrix multiplication.

[0089] Specifically, when a GPU is available, it can be used for processing. Therefore, the target computational structure in the network model file should be determined first. For example, the target computational structure can be matrix multiplication, which is a basic computational structure in the network model. Matrix multiplication can be processed using a GPU to accelerate inference. Therefore, all matrix multiplications in the target model file should be determined first.

[0090] Step S103: Determine the tensor associated with the target computational structure as the target tensor, wherein the target tensor comprises a multidimensional matrix of elements of the same data type.

[0091] Specifically, the tensors associated with the target computational structure can be tensors used in the computations within that structure. A tensor is a multidimensional matrix containing elements of the same data type and is a core class in the Torch library. Programs need to be converted to a Compute Unified Device Architecture (CUDA) parallel format to utilize GPU acceleration. CUDA is a general-purpose parallel computing architecture that enables GPUs to solve complex computational problems. If the terminal device includes a GPU that supports CUDA modules, then the GPU can be used to accelerate network computations.

[0092] Step S104: Detect whether the source program format of the target tensor meets the preset CUDA parallel program format requirements.

[0093] Specifically, GPUs and CPUs can be interchanged during use. Deep learning uses the CPU by default, but we can transfer data computation from the CPU to the GPU when needed. The GPU format must meet the CUDA parallel program format requirements; therefore, we first check whether the source program format of the target tensor meets the preset CUDA parallel program format requirements.

[0094] Step S105: When the source program format requirements of the target tensor do not meet the preset CUDA parallel program format requirements, modify the source program format of the target tensor according to the preset CUDA parallel program format requirements.

[0095] Specifically, when the source code format of the target tensor does not meet the preset CUDA parallel program format requirements, and data computation is transferred from the CPU to the GPU, the source code format of the target tensor needs to be modified according to the preset CUDA parallel program format requirements. For example, modifying the source code format of the target tensor according to the preset CUDA parallel program format requirements can be done by simply adding ".cuda()" to the end of the code.

[0096] Step S106: Based on the graphics processor, predictive inference is performed on the inference task according to the source program of the modified target tensor.

[0097] Specifically, when we call the source code of the modified target tensor, the data migration from CPU to GPU is completed. When the source code of the modified target tensor is called, the mapping of data and the entire computation graph from CPU to GPU is completed, allowing data computation to be transferred to the GPU. This ensures the flexibility of computing resource utilization.

[0098] In one embodiment, when the network model performs an inference task, after detecting whether a graphics processor exists, the process includes: if the graphics processor does not exist, detecting whether a tensor corresponding to the CUDA parallel program format exists; if a tensor corresponding to the CUDA parallel program format exists, modifying the source program format of the tensor corresponding to the CUDA parallel program format to the source program format of the CPU computation tensor.

[0099] Specifically, when no image processor is available, the system can detect whether a tensor in the target model file exists in CUDA parallel program format. If such a tensor exists, the source code for that tensor can be modified to a CPU-computable tensor format, allowing data computation to be transferred to the CPU. For example, modifying the source code for a CUDA parallel program tensor to a CPU-computable tensor format can be achieved by simply removing the `.cuda()` directive from the end of the code, thus converting the CUDA parallel program tensor into a CPU-computable tensor.

[0100] In one application scenario, for example, there is a model definition file that needs to accelerate matrix multiplication, as follows:

[0101] import torch

[0102] a = torch.Tensor([1,2])

[0103] b = torch.Tensor([[1],[2]])

[0104] c = torch.matmul(a, b)

[0105] The target computation structure is matrix multiplication c = torch.matmul(a,b). The tensors corresponding to the target computation structure include a = torch.Tensor([1,2]) and b = torch.Tensor([[1],[2]]). After analysis, it was found that torch.Tensor([1,2]) and torch.Tensor([[1],[2]]) are two CPU-based tensors. If there is no available GPU device on the terminal device, the tensors will not be transferred to the GPU for accelerated computation during actual inference. If there is an available GPU device on the execution terminal where the current inference service is located, then a and b will be changed to torch.Tensor([1,2]).cuda() and torch.Tensor([[1],[2]]).cuda(); that is, the above code will become:

[0106] a=torch.Tensor([1,2]).cuda()

[0107] b=torch.Tensor([[1],[2]]).cuda()

[0108] c = torch.matmul(a, b)

[0109] In one embodiment, when the network model performs an inference task, after detecting whether a graphics processor exists, the method further includes: when the graphics processor exists and the computing power of the graphics processor is less than a second preset threshold, detecting whether a tensor corresponding to the CUDA parallel program format exists; if a tensor corresponding to the CUDA parallel program format exists, modifying the source program format of the tensor corresponding to the CUDA parallel program format to the source program format of the CPU computing tensor.

[0110] Specifically, when the graphics processor (GPU) is present, and its computing power is less than a second preset threshold, it indicates that although a GPU exists, its computing power is too low to handle large amounts of computational data. In this case, it is also possible to detect whether a tensor corresponding to the CUDA parallel program format exists in the target model file. If a tensor corresponding to the CUDA parallel program format exists, the source program format of the tensor corresponding to the CUDA parallel program format is modified to the source program format of the CPU computation tensor, so that the data computation can be transferred to the CPU.

[0111] This application can first detect the presence of a graphics processing unit (GPU) when the network model begins executing inference tasks; and if a GPU is present, determine the target computational structure in the network model file; the target computational structure includes matrix multiplication; determine the tensor associated with the target computational structure as the target tensor, the target tensor including a multidimensional matrix with elements of the same data type; check whether the source code format requirements of the target tensor meet the preset CUDA parallel program format requirements; then, if the source code format requirements of the target tensor do not meet the preset CUDA parallel program format requirements, modify the source code format of the target tensor according to the preset CUDA parallel program format requirements; and perform prediction inference tasks based on the modified target tensor source code using the GPU. Because the tensors associated with the target computational structure are transformed into GPU-accelerated tensors, they can automatically adapt to the GPU for accelerated inference, thereby improving the convenience of accelerating inference for the network model.

[0112] Example 2

[0113] This embodiment is a further explanation of Embodiment 1. For any parts that are the same as or similar to Embodiment 1, please refer to the relevant description in Embodiment 1; they will not be repeated here. Figure 2 As shown, steps S201 to S203 can be implemented as one way of implementing step S103 above:

[0114] Step S201: Determine the tensors associated with the target computational structure.

[0115] Specifically, the tensor associated with the target computation structure can be the tensor needed for computation in the target computation structure. A tensor is a multidimensional matrix containing elements of the same data type and is the core class of the torch library.

[0116] Step S202: Determine the dimension of the tensor associated with the target computational structure.

[0117] Specifically, the larger the dimension of the tensor in matrix multiplication, the lower the CPU's computational efficiency. Therefore, the dimension of the tensor can be determined, and then it can be considered whether to modify the tensor according to the CUDA parallel program format requirements based on the dimension of the tensor.

[0118] Step S203: When the dimension of any tensor among all tensors associated with the target computational structure is greater than a first preset threshold, all tensors associated with the target computational structure are determined as the target tensor.

[0119] Specifically, if the dimension of any tensor associated with the target computing structure is greater than a first preset threshold, it is considered that the computational load corresponding to the target computing structure will be large. All tensors associated with the target computing structure can be identified as the target tensor for processing by the GPU.

[0120] In one embodiment, after determining the dimensions of the tensors associated with the target computational structure, the method further includes: if the dimensions of any two tensors among all tensors associated with the target computational structure are greater than a second preset threshold, then all tensors associated with the target computational structure are determined as the target tensor; wherein the second preset threshold is less than the first preset threshold.

[0121] Specifically, if the dimensions of any two tensors among all tensors associated with the target computing structure are greater than the second preset threshold, it is considered that the computational load corresponding to the target computing structure will also be large. All tensors associated with the target computing structure can be identified as the target tensors for processing by the GPU.

[0122] This application embodiment can determine the dimension of tensors associated with the target computing structure; when the dimension of any tensor associated with the target computing structure is greater than a first preset threshold, all tensors associated with the target computing structure are determined as the target tensor. Whether to process via GPU can be determined based on the tensor's dimension. Network model inference can be accelerated more intelligently based on the actual environment.

[0123] Example 3

[0124] This embodiment is a further explanation of Embodiment 1. For any parts that are the same as or similar to Embodiment 1, please refer to the relevant description in Embodiment 1; they will not be repeated here. Figure 3 As shown, steps S301 to S305 can be implemented as one way of implementing step S103 above:

[0125] Step S301: Determine the tensors associated with the target computational structure.

[0126] Step S302: Determine the dimension of the tensor associated with the target computational structure.

[0127] Specifically, for the parts of steps S301 and S302 that are the same as or similar to steps S201 and S202 above, please refer to the description of steps S201 and S202 above, which will not be repeated here.

[0128] Step S303: Determine the computing power value of the graphics processor.

[0129] Specifically, the computing power of the terminal device's GPU can be determined first. For example, the computing power value of the graphics processor can be determined based on the current amount of free memory in the graphics processor.

[0130] Step S304: Based on a preset relationship mapping table, obtain the target threshold corresponding to the computing power value.

[0131] Specifically, different target thresholds corresponding to different computing capabilities are pre-stored. The smaller the threshold corresponding to the computing capability value, the better. For example, a GPU with a large computing capability can modify all tensors in matrix multiplications according to the CUDA parallel program format requirements, while a GPU with a moderate computing capability can modify only a portion of the tensors in all matrix multiplications according to the CUDA parallel program format requirements. Thus, the computing capability of the GPU can be taken into account to determine the matrix multiplications that need to be accelerated by the GPU and obtain the threshold corresponding to the computing capability, which is called the target threshold.

[0132] Step S305: When the dimension of any tensor among all tensors associated with the target computational structure is greater than the target threshold, all tensors associated with the target computational structure are determined as the target tensor.

[0133] Specifically, if the dimension of any tensor associated with the target computing structure is greater than a first preset threshold, it is considered that the computational load corresponding to the target computing structure will be large. All tensors associated with the target computing structure can be identified as the target tensor for processing by the GPU.

[0134] In this embodiment, the computing power value of the graphics processor is determined; a target threshold corresponding to the computing power value is obtained based on a preset relational mapping table; when the dimension of any tensor associated with the target computing structure is greater than the target threshold, all tensors associated with the target computing structure are determined as the target tensor. This allows for consideration of the GPU's computing power, identification of matrix multiplications requiring GPU acceleration, and further intelligent acceleration of network model inference based on the actual environment.

[0135] Example 4

[0136] Corresponding to the method for accelerating network model inference in the above embodiments, Figure 4 This diagram illustrates a structural block diagram of an apparatus for accelerating network model inference according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. The apparatus 400 for accelerating network model inference includes:

[0137] The acquisition module 401 is used to detect the presence of a graphics processor when the network model starts to perform an inference task;

[0138] The first determining module 402 is used to determine the target computational structure in the file of the network model when the graphics processor is present; the target computational structure includes matrix multiplication.

[0139] The second determining module 403 is used to determine the tensor associated with the target computation structure as the target tensor, the target tensor including a multidimensional matrix of elements of the same data type;

[0140] The first detection module 404 is used to detect whether the source program format of the target tensor meets the preset CUDA parallel program format requirements.

[0141] The first modification module 405 is used to modify the source program format of the target tensor according to the preset CUDA parallel program format requirements when the source program format requirements of the target tensor do not meet the preset CUDA parallel program format requirements.

[0142] The prediction and inference module 406 is used by the graphics processor to perform prediction and inference on the inference task based on the source program of the modified target tensor.

[0143] In one embodiment, the second determining module includes:

[0144] The first determining unit is used to determine the tensors associated with the target computational structure;

[0145] The second determining unit is used to determine the dimension of the tensor associated with the target computational structure;

[0146] The third determining unit is used to determine all tensors associated with the target computing structure as the target tensor when the dimension of any tensor among all tensors associated with the target computing structure is greater than a first preset threshold.

[0147] In one embodiment, the second determining module further includes:

[0148] The fourth determining unit is configured to determine all tensors associated with the target computing structure as the target tensor when any two tensors among all tensors associated with the target computing structure have dimensions greater than a second preset threshold; wherein the second preset threshold is less than the first preset threshold.

[0149] In another embodiment, the second determining module includes:

[0150] The fifth determining unit is used to determine the tensors associated with the target computational structure;

[0151] The sixth determining unit is used to determine the dimension of the tensor associated with the target computational structure;

[0152] The seventh determining unit is used to determine the computing power value of the graphics processor;

[0153] The acquisition unit is used to acquire a target threshold corresponding to the computing power value based on a preset relationship mapping table;

[0154] The eighth determining unit is used to determine all tensors associated with the target computing structure as the target tensor when the dimension of any tensor among all tensors associated with the target computing structure is greater than the target threshold.

[0155] In one embodiment, the apparatus further includes:

[0156] The second detection module is used to detect whether there is a tensor corresponding to the CUDA parallel program format when the graphics processor is not present.

[0157] The second modification module is used to modify the source program format of the CUDA parallel program format tensor to the source program format tensor of the CPU computation format if a tensor corresponding to the CUDA parallel program format exists.

[0158] In one embodiment, the apparatus further includes:

[0159] The third detection module is used to detect whether there is a tensor corresponding to the CUDA parallel program format when the graphics processor is present and the computing power of the graphics processor is less than a second preset threshold.

[0160] The third modification module is used to modify the source program format of the CUDA parallel program format tensor to the source program format tensor of the CPU computation format if a tensor corresponding to the CUDA parallel program format exists.

[0161] In one embodiment, the acquisition module specifically includes:

[0162] The acquisition unit is used to acquire device information of the terminal device, including the processor, when the network model starts executing the inference task.

[0163] The first determination unit is used to determine that the graphics processor exists if there is device information in the terminal device that matches the device information of the graphics processor.

[0164] The second determination unit is used to determine that the graphics processor does not exist if there is no device information in the terminal device that matches the device information of the graphics processor.

[0165] This application can detect the presence of a graphics processing unit (GPU) when the network model begins inference; if a GPU is present, it determines the target computational structure in the network model file; the target computational structure includes matrix multiplication; it identifies the tensor associated with the target computational structure as the target tensor, which includes a multidimensional matrix with elements of the same data type; it checks whether the source code format of the target tensor meets the preset CUDA parallel program format requirements; then, if the source code format of the target tensor does not meet the preset CUDA parallel program format requirements, it modifies the source code format of the target tensor according to the preset CUDA parallel program format requirements; and it performs prediction inference based on the modified source code of the target tensor using the GPU. By transforming the tensor associated with the target computational structure into a GPU-accelerated tensor, it can automatically adapt to the GPU for accelerated inference, thereby improving the convenience of accelerating inference for the network model.

[0166] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0167] Example 5

[0168] like Figure 5 As shown, one embodiment of this application also provides a terminal device 500 including: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501, such as a network model inference acceleration program. When the processor 501 executes the computer program 503, it implements the steps in the above-described methods for accelerating network model inference. When the processor 501 executes the computer program 503, it implements the functions of each module in the above-described device embodiments, for example... Figure 5 The functions of modules 401 to 406 are shown.

[0169] For example, the computer program 503 can be divided into one or more modules, which are stored in the memory 502 and executed by the processor 501 to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 503 in the terminal device 500. For example, the computer program 503 can be divided into an acquisition module, a first determination module, a second determination module, a first detection module, a first modification module, and a prediction and reasoning module, etc. The specific functions of each module have been described in the above embodiments and will not be repeated here.

[0170] The terminal device 500 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of terminal device 500 and does not constitute a limitation on terminal device 500. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computing device may also include input / output devices, network access devices, buses, etc.

[0171] The processor 501 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0172] The memory 502 can be an internal storage unit of the terminal device 500, such as a hard disk or memory of the terminal device 500. The memory 502 can also be an external storage device of the terminal device 500, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 500. Furthermore, the memory 502 can include both internal and external storage units of the terminal device 500. The memory 502 is used to store the computer program and other programs and data required by the computing device. The memory 502 can also be used to temporarily store data that has been output or will be output.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0174] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0175] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0176] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computing device and method can be implemented in other ways. For example, the apparatus / computing device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0179] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0180] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for accelerating network model inference, characterized in that, The method, applied to a terminal device for deploying a network model, includes: When the network model begins to perform the inference task, the presence of a graphics processor is detected; When the graphics processor is present, the target computational structure in the file of the network model is determined; the target computational structure includes matrix multiplication. Determine the tensor associated with the target computation structure as the target tensor, the target tensor comprising a multidimensional matrix of elements of the same data type, and the tensor associated with the target computation structure is the tensor required for computation in the target computation structure; Detect whether the source program format of the target tensor meets the preset requirements of the parallel program format of the unified computing device architecture; When the source program format requirements of the target tensor do not meet the preset unified computing device architecture parallel program format requirements, the source program format of the target tensor shall be modified in accordance with the preset unified computing device architecture parallel program format requirements; Based on the graphics processor, predictive reasoning is performed on the inference task according to the source program of the modified target tensor. When the network model performs an inference task, after detecting whether a graphics processor exists, the process includes: if the graphics processor does not exist, detecting whether a tensor corresponding to the unified device architecture parallel program format exists; if a tensor corresponding to the unified device architecture parallel program format exists, modifying the source program format of the tensor corresponding to the unified device architecture parallel program format to the source program of the CPU computing format tensor. The step of determining the tensor associated with the target computational structure as the target tensor includes: Determine the tensors associated with the target computational structure; Determine the dimensions of the tensors associated with the target computational structure; If the dimension of any tensor among all tensors associated with the target computational structure is greater than a first preset threshold, all tensors associated with the target computational structure will be determined as the target tensor. After determining the dimensions of the tensors associated with the target computational structure, the method further includes: If any two tensors associated with the target computational structure have dimensions greater than a second preset threshold, then all tensors associated with the target computational structure are identified as the target tensor; wherein the second preset threshold is less than the first preset threshold. After detecting the presence of a graphics processor when the network model performs inference tasks, the following steps are also included: When the graphics processor is present and its computing power is less than a second preset threshold, detect whether there is a tensor corresponding to the parallel program format of the unified device architecture. If a tensor corresponding to the unified device architecture parallel program format exists, then the source program format for calculating the tensor corresponding to the unified device architecture parallel program format is modified to the source program format for calculating the CPU computing tensor.

2. A method for accelerating network model inference, characterized in that, The method, applied to a terminal device for deploying a network model, includes: When the network model begins to perform the inference task, the presence of a graphics processor is detected; When the graphics processor is present, the target computational structure in the file of the network model is determined; the target computational structure includes matrix multiplication. Determine the tensor associated with the target computation structure as the target tensor, the target tensor comprising a multidimensional matrix of elements of the same data type, and the tensor associated with the target computation structure is the tensor required for computation in the target computation structure; Detect whether the source program format of the target tensor meets the preset requirements of the parallel program format of the unified computing device architecture; When the source program format requirements of the target tensor do not meet the preset unified computing device architecture parallel program format requirements, the source program format of the target tensor shall be modified in accordance with the preset unified computing device architecture parallel program format requirements; Based on the graphics processor, predictive reasoning is performed on the inference task according to the source program of the modified target tensor. When the network model performs an inference task, after detecting whether a graphics processor exists, the process includes: if the graphics processor does not exist, detecting whether a tensor corresponding to the unified device architecture parallel program format exists; if a tensor corresponding to the unified device architecture parallel program format exists, modifying the source program format of the tensor corresponding to the unified device architecture parallel program format to the source program of the CPU computing format tensor. The step of determining the tensor associated with the target computational structure as the target tensor includes: Determine the tensors associated with the target computational structure; Determine the dimensions of the tensors associated with the target computational structure; Determine the computing power value of the graphics processor; Based on a preset relationship mapping table, obtain the target threshold corresponding to the computing power value; If the dimension of any tensor associated with the target computational structure is greater than the target threshold, then all tensors associated with the target computational structure are identified as the target tensor.

3. The method according to any one of claims 1 or 2, characterized in that, When the network model begins to perform an inference task, detecting the presence of a graphics processor includes: When the network model starts executing the inference task, it obtains device information of the terminal device, including the processor. If the terminal device contains device information that matches the device information of the graphics processor, then it is determined that the graphics processor exists. If no device information matching the graphics processor's device information exists in the terminal device, then it is determined that the graphics processor does not exist.

4. A device for accelerating network model inference, characterized in that, A terminal device for deploying network models, used to execute a method for accelerating network model inference as described in any one of claims 1-3, the device comprising: The acquisition module is used to detect the presence of a graphics processor when the network model starts performing an inference task; The first determining module is used to determine the target computational structure in the file of the network model when the graphics processor is present; the target computational structure includes matrix multiplication. The second determining module is used to determine the tensor associated with the target computing structure as the target tensor. The target tensor includes a multidimensional matrix of elements of the same data type. The tensor associated with the target computing structure is the tensor used for computing in the target computing structure. The first detection module is used to detect whether the source program format of the target tensor meets the requirements of the preset computing unified device architecture parallel program format; The first modification module is used to modify the source program format of the target tensor according to the preset unified computing device architecture parallel program format requirements when the source program format requirements of the target tensor do not meet the preset unified computing device architecture parallel program format requirements. The predictive inference module is used by the graphics processor to perform predictive inference on the inference task based on the source program of the modified target tensor.

5. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

Images