Neural network reasoning and speculation acceleration method, system and device and medium
By determining the optimal number of operator layers of each hardware device and optimizing the computing power utilization of hardware devices, the problem of the calculation speed of neural network model acceleration of low-computing hardware devices is solved, and efficient neural network model inference is realized, suitable for multi-platform and multi-architecture hardware devices.
Patent Information
- Application Number
- CN202410914836.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-12
AI Technical Summary
It is difficult for the prior art to use low-computing hardware devices to complete the speculation and acceleration of neural network models, and the hardware performance requirements are high.
By obtaining the total number of operator layers of the neural network model to be inferred, we determine the optimal number of operator layers of each hardware device, and control each hardware device to infer it, accelerate the inference process of the neural network model, and use edge computing devices, The combination of cloud servers and terminal devices optimizes the computing power utilization of hardware devices and uses SIMD instruction sets and CUDA instruction sets for acceleration.
It realizes the speculation acceleration of the neural network model of low-computing hardware devices, avoids compiler and platform restrictions, reduces the cost of development and model inference, improves the speed and efficiency of inference, and is suitable for multi-platform and multi-architecture hardware devices.
Smart Images

Figure CN118940836A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of neural network inference, and in particular to a neural network reasoning and inference acceleration method, system, device and medium. Background Art
[0002] A neural network model is a mathematical model used in the field of machine learning and artificial intelligence. This model consists of multiple neurons, which are distributed in different layers to form a network. Each neuron receives input from the neurons in the previous layer and produces an output, which can be passed to the next layer of neurons. Through the training process, the neural network model adjusts its internal weights and biases so that it can learn features from the input data and make predictions or classifications on data that it has never seen. Neural network models have performed well in many fields, such as image recognition, speech recognition, natural language processing, etc. The development of neural network models includes many types, such as feedforward neural networks, recurrent neural networks, convolutional neural networks, etc., each of which has its specific application scenarios and advantages.
[0003] Inferring and accelerating neural network models is an important issue in optimizing the performance of neural networks in practical applications. Specific methods may include model pruning, quantization, mixed precision computing, hardware accelerators, model compression, and lightweight model design. However, when using the above methods to infer and accelerate neural network models, the hardware performance requirements of the device are relatively high. In view of this, how to use a central processing unit or chip device with lower computing power to complete the inference and acceleration of neural network models is an urgent problem to be solved. Summary of the invention
[0004] The embodiments of the present application solve the technical problem in the prior art that it is difficult to complete the speculation and acceleration of the neural network model with low-computing power hardware devices by providing a neural network reasoning speculation acceleration method, system, device and medium, and achieve the technical effect of being able to complete the speculation and acceleration of the neural network model with low-computing power hardware devices.
[0005] In a first aspect, the present application provides a neural network speculation acceleration method, the method comprising:
[0006] Get the total number of operator layers of the neural network model to be inferred and accelerated;
[0007] Determine the optimal number of operator layers corresponding to each hardware device respectively, wherein the number of hardware devices is several, the hardware devices are used to speculate on the speculation accelerated neural network model, the sum of the optimal number of operator layers corresponding to each hardware device is equal to the total number of operator layers, and the types of hardware devices include edge computing devices, cloud servers, and terminal devices;
[0008] Control each hardware device to infer the optimal number of operator layers corresponding to each device.
[0009] Furthermore, the optimal number of operator layers corresponding to the hardware device is determined, including:
[0010] Set the initial number of operator layers for the hardware device;
[0011] Under the rated power of the hardware device, determine the target time consumption of the hardware device to estimate the number of initial operator layers;
[0012] Adjust the number of initial operator layers;
[0013] Under the rated power of the hardware device, re-determine the target time consumption of the hardware device to infer the adjusted initial operator layers;
[0014] When the target time is the shortest and the number of initial operator layers is the largest, the initial operator layer number is used as the optimal operator layer number for the hardware device.
[0015] Furthermore, determining the optimal number of operator layers corresponding to the hardware device also includes:
[0016] Obtain the hardware indicators corresponding to the hardware device and the operator parameters corresponding to the accelerated neural network model to be inferred;
[0017] The hardware indicators and operator parameters corresponding to the hardware device are input into the preset inference neural network model to obtain the optimal number of operator layers corresponding to the hardware device.
[0018] Furthermore, after determining the optimal number of operator layers corresponding to the hardware device, the following steps are also included:
[0019] According to the bit width of the hardware device, the data of the optimal number of operator layers corresponding to the hardware device is cropped and filled;
[0020] The optimal operator layer number corresponding to the hardware device is optimized by adjusting the number of threads and the size of the data blocks of the optimal operator layer number corresponding to the hardware device.
[0021] Furthermore, the formula for calculating the neural network model is preset, including:
[0022] time=f(c,n,d,s)
[0023]
[0024] Among them, time is the estimation time, c is the CPU frequency, n is the operator type, d is the shape of the input data, s is the speed of memory exchange, L is the error between the predicted value and the training data, t p is the prediction time, t q is the actual time consumed, f() is the loss function, and i is the index of the training data.
[0025] Furthermore, an inference neural network model is constructed, including:
[0026] Obtain the hardware indicators corresponding to several hardware devices, the number of operator layers corresponding to each hardware device, the inference time of the hardware device under the number of operator layers, and the operator parameters corresponding to the accelerated neural network model to be inferred;
[0027] Input the hardware indicators corresponding to the hardware device, the number of operator layers corresponding to the hardware device, and the operator parameters corresponding to the accelerated neural network model to be inferred into the initial neural network model to obtain the simulation results corresponding to the hardware device;
[0028] When several simulation results and the corresponding inference time are less than a preset threshold, the initial neural network model is used as the inferred neural network model.
[0029] Furthermore, hardware indicators include:
[0030] Including memory, bandwidth, computing speed, power consumption, and input data of the network layer.
[0031] In a second aspect, the present application provides a neural network speculation acceleration system, the system comprising:
[0032] Client, server and application side. The application side includes edge computing devices, cloud servers and terminal devices.
[0033] The client sends a speculation request to the server;
[0034] The server receives the inference request and receives the inference result from the application;
[0035] The server returns the inference result to the client so that the client can display the inference result.
[0036] In a third aspect, the present application provides an electronic device, including:
[0037] processor;
[0038] a memory for storing processor-executable instructions;
[0039] Among them, the processor is configured to execute to implement a neural network speculation acceleration method provided in the first aspect.
[0040] In a fourth aspect, the present application provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute a neural network speculation acceleration method as provided in the first aspect.
[0041] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
[0042] The present application provides a neural network speculation acceleration method, the method comprising: obtaining the total number of operator layers of the neural network model to be speculated and accelerated; respectively determining the optimal number of operator layers corresponding to each hardware device, wherein the number of hardware devices is several, the hardware devices are used to speculate the neural network model to be speculated and accelerated, the sum of the optimal number of operator layers corresponding to each hardware device is equal to the total number of operator layers, and the types of hardware devices include edge computing devices, cloud servers, and terminal devices; and controlling each hardware device to speculate the optimal number of operator layers corresponding to each hardware device. The present application can enable low computing power devices to complete the speculation acceleration of the neural network model to be speculated and accelerated by determining the optimal number of operator layers corresponding to each hardware device.
[0043] In addition, based on itself, please provide a neural network speculation acceleration system and a neural network speculation acceleration method, both of which are implemented based on low-computing power hardware devices. The neural network speculation acceleration system uses the SIMD instruction set in the CPU hardware to accelerate model reasoning, which can avoid the limitations of compilers and platforms in the existing technology, and the speed and efficiency are higher than compiled languages such as C++ and Python.
[0044] The neural network speculation acceleration system also uses a more mature hardware device environment to reduce development costs and model inference costs, and uses the instruction set in its mature ecological environment to accelerate model inference.
[0045] The neural network speculation acceleration system also reduces the cost of environmental development by using SIMD instructions or API interface ecosystems on different hardware devices. Furthermore, the neural network speculation acceleration system environment is simple and efficient to build, integrating numerous and complex multi-platform and multi-architecture hardware devices, enabling the system to accelerate the model's reasoning speed while ensuring low development costs.
[0046] When the inference of the neural network model to be accelerated is performed, when the application uses a combination of CPU and GPU, the mature development environment based on the CPU, the integrated graphics card has low power consumption and high compatibility with the CPU, which can accelerate model reasoning more efficiently.
[0047] By using the CPU to be responsible for the reasoning of some simple modules of the model, GPU computing power can be saved and the reasoning cost can be reduced. The final model division can make the model reasoning have low memory occupancy and shorter time consumption. The backbone network structure with many calculations and complex structure is inferred by the GPU, and the simple modules are inferred by the CPU. When reasoning the model, the CPU uses the SIMD instruction set or API interface to accelerate the model reasoning, while the GPU uses the CUDA instruction set or API interface to accelerate the model reasoning, which reduces GPU energy consumption, reduces the reasoning cost, improves the model reasoning speed, and reduces memory occupancy.
[0048] When the kernel processes data, direct convolution processing is faster and more efficient than compiled language processing. Through machine learning, SIMD and CUDA are tuned and trained to obtain the fastest and most efficient SIMD instruction set for kernel feature map processing, and the division and grouping of kernels by the CUDA instruction set, so that the accelerated neural network model to be speculated can complete its reasoning process while ensuring time, higher efficiency, higher performance, and more balanced application-side load. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0050] Figure 1 A flowchart of a neural network speculation acceleration method provided in this application;
[0051] Figure 2 A topological diagram of a neural network speculation acceleration system provided for this application;
[0052] Figure 3 A data flow diagram provided for this application;
[0053] Figure 4 Another data flow diagram provided for this application;
[0054] Figure 5 A schematic diagram of neural network operator allocation provided for this application;
[0055] Figure 6 Another neural network operator allocation diagram provided for this application;
[0056] Figure 7 A schematic diagram of the structure of an electronic device provided in this application.
[0057] Reference numerals
[0058] 21-client, 22-server, 23-application. DETAILED DESCRIPTION
[0059] The embodiments of the present application provide a neural network speculation acceleration method, thereby solving the technical problem in the prior art that it is difficult to complete the speculation and acceleration of the neural network model with low computing power hardware devices.
[0060] The technical solution of the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:
[0061] A neural network speculation acceleration method, the method comprising: obtaining the total number of operator layers of a neural network model to be speculated and accelerated; respectively determining the optimal number of operator layers corresponding to each hardware device, wherein the number of hardware devices is several, the hardware devices are used to speculate the neural network model to be speculated and accelerated, the sum of the optimal number of operator layers corresponding to each hardware device is equal to the total number of operator layers, and the types of hardware devices include edge computing devices, cloud servers and terminal devices; and controlling each hardware device to speculate the optimal number of operator layers corresponding to each hardware device.
[0062] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0063] First of all, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0064] This application provides Figure 1 The neural network speculation acceleration method shown can be applied to Figure 2 A neural network speculation acceleration system is shown, the system comprising:
[0065] The client 21, the server 22 and the application 23, the application 23 includes an edge computing device, a cloud server and a terminal device.
[0066] The purpose of this application is to use low-computing power or retired hardware devices to accelerate the speculation of neural networks. Therefore, the edge computer devices, cloud servers and terminal devices used in this application are all low-computing power hardware devices or retired hardware devices. Of course, the method provided in this application is also applicable to hardware devices with higher configurations.
[0067] The model is deployed in each device in the application end 23, and the network service is started; when there is a speculation demand, the client 21 sends a speculation request to the server end 22, the server end 22 receives the speculation request, and forwards the speculation request to the application end 23. After the application end 23 receives the speculation request, it performs speculation acceleration on the speculation-accelerated neural network model. After the application end 23 completes the speculation, it returns the speculation result to the server end 22. The server end 22 receives the speculation result and forwards the speculation result to the client end 21 so that the client end 21 can display the speculation result.
[0068] In addition, the present application uses different instruction sets or APIs for model speculation acceleration according to different types of hardware devices (different architectures).
[0069] Devices such as Intel's Celeron series CPUs and AMD's Athlon 2 series CPUs use their CPU's corresponding SIMD instruction sets to accelerate model speculation.
[0070] The neural network speculation acceleration system provided by the present application is simple to build, quick to set up, and applicable to multiple platforms and different hardware architectures. The acceleration instruction sets used include WEB application technologies such as Flask and Django, SIMD, etc. In other words, the present application uses different underlying APIs to implement computing functions according to different hardware architectures (such as ARM and X86), and distinguishes and adapts different hardware devices through macro definitions during compilation, so that the hardware devices can better complete the speculation acceleration of the neural network model to be speculated.
[0071] This application provides Figure 1 A neural network speculation acceleration method is shown, comprising steps S11-S13.
[0072] Step S11, obtaining the total number of operator layers of the accelerated neural network model to be inferred.
[0073] Step S12, respectively determine the optimal number of operator layers corresponding to each hardware device, wherein the number of hardware devices is several, and the hardware devices are used to speculate on the speculated accelerated neural network model. The sum of the optimal number of operator layers corresponding to each hardware device is equal to the total number of operator layers. The types of hardware devices include edge computing devices, cloud servers, and terminal devices.
[0074] Step S13, controlling each hardware device to infer the optimal number of operator layers corresponding to each device.
[0075] Regarding step S11, the total number of operator layers of the accelerated neural network model to be inferred is obtained.
[0076] The neural network model to be accelerated is a neural network model that has a need for acceleration. There are several layers of operators in the neural network model to be accelerated.
[0077] Before obtaining the total number of operator layers of the to-be-speculated accelerated neural network model, the to-be-speculated accelerated neural network model may be classified according to its size.
[0078] When the accelerated neural network model to be speculated is small, such as a traditional target detection model, it can be speculated using edge computing devices and terminal devices; when the accelerated neural network model to be speculated is a medium-sized model, such as a super-resolution model, it can be speculated using hardware devices with an independent graphics card and CPU; when the accelerated neural network model to be speculated is a large model, such as the current large language model, the present application can split the accelerated neural network model to be speculated and execute steps S12-S13.
[0079] It should also be noted that you can first determine whether the edge computing device, cloud server, and terminal device has an independent graphics card. If so, the inference acceleration mode is to use an independent graphics card for inference. If there is no independent graphics card, the CPU and hardware acceleration module corresponding to the hardware device are used for inference acceleration.
[0080] Regarding step S12, the optimal number of operator layers corresponding to each hardware device is determined respectively, wherein the number of hardware devices is several, and the hardware devices are used to speculate on the speculated accelerated neural network model. The sum of the optimal number of operator layers corresponding to each hardware device is equal to the total number of operator layers. The types of hardware devices include edge computing devices, cloud servers and terminal devices.
[0081] In order to make full use of the maximum computing power of edge computing devices, cloud servers and terminal devices, this application splits the neural network model to be accelerated (the neural network model to be accelerated is composed of multiple layers of basic operators, and the neural network model to be accelerated can be divided into blocks), and matches the appropriate number of operator layers to each hardware device, so that each hardware device can exert its maximum computing power.
[0082] Methods for matching the optimal number of operator layers for each hardware device include [Method 1] and [Method 2]. The selection of [Method 1] or [Method 2] can be determined based on actual conditions and is not limited here.
[0083]
Method 1
[0084] Determining the optimal number of operator layers corresponding to the hardware device includes: setting an initial number of operator layers for the hardware device; determining a target time consumption for the hardware device to estimate the initial number of operator layers under the rated power of the hardware device; adjusting the initial number of operator layers; re-determining a target time consumption for the hardware device to estimate the adjusted initial number of operator layers under the rated power of the hardware device; when the target time consumption is the shortest and the estimated number of initial operator layers is the largest, using the initial number of operator layers as the optimal number of operator layers for the hardware device.
[0085] First, an initial operator layer number is set for a hardware device. Under the rated power of the hardware device, the hardware device is controlled to perform speculation acceleration on the initial operator layer number, and all speculation time (target time) of the hardware device under the initial operator layer number can be obtained. Then, the initial operator layer number is adjusted (increased or decreased), and the hardware device is continuously controlled to perform speculation acceleration on the initial operator layer number to obtain a new speculation time. The above process is repeated until the shortest speculation time and the largest number of speculated initial operator layers are obtained. The initial operator layer number in this case is used as the optimal operator layer number for the hardware device.
[0086] For example, hardware device A adjusted the initial operator layer number three times in total. When the initial operator layer number was 29, the target time consumption was 1.30s; when the initial operator layer number was 30, the target time consumption was 1.30s; when the initial operator layer number was 31, the target time consumption was 1.35s. The initial operator layer number of 30 is taken as the optimal operator layer number of hardware device A.
[0087] In addition, you can also use [Method 2] to obtain the optimal number of operator layers for the hardware device.
[0088]
Method 2
[0089] Determining the optimal number of operator layers corresponding to the hardware device also includes: obtaining hardware indicators corresponding to the hardware device and operator parameters corresponding to the accelerated neural network model to be inferred; inputting the hardware indicators and operator parameters corresponding to the hardware device into the preset inference neural network model to obtain the optimal number of operator layers corresponding to the hardware device.
[0090] Among them, the hardware indicators include: memory, bandwidth, computing speed, power consumption and input data of the network layer; the operator parameters corresponding to the accelerated neural network model to be inferred include: convolution parameter settings, pooling parameter settings or activation function parameter settings, etc.
[0091] The preset inference neural network model is a trained inference model. The preset inference neural network model is used to infer the optimal number of operator layers corresponding to the hardware device. After obtaining the hardware indicators corresponding to the hardware device and the operator parameters corresponding to the accelerated neural network model to be inferred, the optimal number of operator layers corresponding to the hardware device can be directly obtained in the preset inference neural network model.
[0092] The formula for the preset inference neural network model provided in this application includes:
[0093] time=f(c,n,d,s)
[0094]
[0095] Among them, time is the estimation time, c is the CPU frequency, n is the operator type, d is the shape of the input data, s is the speed of memory exchange, L is the error between the predicted value and the training data, t p is the prediction time, t q is the actual time consumed, f() is the loss function, and i is the index of the training data.
[0096] refer to Figure 3 and 4 The divided accelerated neural network model to be speculated is transferred to the CPU as the total data stream, and the split part that can be processed by the GPU is copied from the CPU memory to the GPU memory as data stream 1. Then, the GPU code is loaded and the corresponding part is processed. At the same time, the CPU divides the processing model into the part that the CPU needs to process, and finally the result of the GPU memory is copied to the CPU memory as data stream 2, thereby obtaining multiple hardware indicators such as device memory, bandwidth, computing speed, and power consumption network layer input data size.
[0097] In addition, the present application also provides a method for constructing an inferred neural network model, including: obtaining hardware indicators corresponding to several hardware devices, the number of operator layers corresponding to each hardware device, the speculation time of the hardware device under the operator layer number, and the operator parameters corresponding to the accelerated neural network model to be speculated; inputting the hardware indicators corresponding to the hardware devices, the number of operator layers corresponding to the hardware devices, and the operator parameters corresponding to the accelerated neural network model to be speculated into the initial neural network model to obtain the simulation results corresponding to the hardware device; when several simulation results and the corresponding speculation time are less than a preset threshold, the initial neural network model is used as the inferred neural network model.
[0098] It can be understood that the hardware indicators corresponding to the hardware device, the number of operator layers corresponding to the hardware device, the speculation time of the hardware device under the operator layer, and the operator parameters corresponding to the accelerated neural network model to be speculated correspond one to one. The above corresponding data are merged and regarded as one data. For example, hardware device A, its hardware indicator is a, and its corresponding operator layer number is 15 layers (it can be understood that hardware device A can correspond to multiple operator layers at the same time), the speculation time of 15 layers is 60s, and the operator parameter corresponding to the accelerated neural network model to be speculated is B, then the merged data is (A, a, 15, 60).
[0099] Specifically, you can set grid parameters, set the number of nodes according to the empirical formula of the number of hidden layer nodes, set the target training target error and learning rate, select the iterative algorithm to build the initial neural network model, and divide the acquired merged data into training set, test set and validation set. The division ratio can be determined according to the actual situation, for example, 7:1.5:1.5.
[0100] Input the training set into the initial neural network model. If the output simulation results and the corresponding inference time are greater than the preset threshold, it means that the accuracy of the initial neural network model does not meet the requirements. You can adjust the grid parameters, continue to input the training set, and make comparisons.
[0101] When several simulation results and the corresponding inference time are less than a preset threshold, the initial neural network model is used as the inferred neural network model.
[0102] From the above, it can be seen that according to the hardware indicators and architecture of the hardware device, the neural network model can be divided into blocks and loaded onto the corresponding hardware device for the most efficient calculation. For details, please refer to Figure 5 and Figure 6 .
[0103] Regardless of whether [Method 1] or [Method 2] is adopted, the optimal number of operator layers corresponding to the hardware device can be obtained. At this time, it is possible to make inferences according to the optimal number of operator layers corresponding to the hardware device. However, in order to further optimize the configuration and improve the inference speed of the hardware device, after determining the optimal number of operator layers corresponding to the hardware device, it is also possible to further include: obtaining the hardware indicators corresponding to the hardware device and the operator parameters corresponding to the accelerated neural network model to be inferred; inputting the hardware indicators and operator parameters corresponding to the hardware device into the preset inference neural network model to obtain the optimal number of operator layers corresponding to the hardware device.
[0104] The instructions of different hardware device architectures have different bit widths, such as SSE and NEON. Most of the inputs to the hardware devices for the inferred accelerated neural network model are high-dimensional data, and the computing power of the hardware devices is usually limited and fixed. Therefore, the input data can be cropped and padded according to the bit width of the computing core of each hardware device, and multi-core parallel processing can be enabled according to the number of computing cores of each hardware device.
[0105] By adjusting the number of threads and the size of data blocks, the data is processed and the processing results are recorded. Based on the K-means method, multivariate linear regression and other methods, evaluation is performed from two aspects: memory usage and time consumption. The final result of SIMD and CUDA division and grouping of Kernel operators with the lowest memory usage and shortest time consumption is obtained, so as to further accelerate the inference speed of hardware devices and reduce the computing power consumption of hardware devices.
[0106] Regarding step S13, each hardware device is controlled to infer the optimal number of operator layers corresponding to each hardware device.
[0107] After obtaining the optimal number of operator layers corresponding to each hardware device, each hardware device can be controlled to speculate on the optimal number of operator layers corresponding to each hardware device, so as to complete the speculation acceleration of the neural network model to be speculated and accelerated.
[0108] In summary, the present application provides a neural network speculation acceleration method, the method comprising: obtaining the total number of operator layers of the neural network model to be speculated and accelerated; respectively determining the optimal number of operator layers corresponding to each hardware device, wherein the number of hardware devices is several, the hardware devices are used to speculate the neural network model to be speculated and accelerated, the sum of the optimal number of operator layers corresponding to each hardware device is equal to the total number of operator layers, and the types of hardware devices include edge computing devices, cloud servers, and terminal devices; and controlling each hardware device to speculate on the optimal number of operator layers corresponding to each hardware device. By determining the optimal number of operator layers corresponding to each hardware device, the present application can enable low computing power devices to complete the speculation acceleration of the neural network model to be speculated and accelerated.
[0109] In addition, based on itself, please provide a neural network speculation acceleration system and a neural network speculation acceleration method, both of which are implemented based on low-computing power hardware devices. The neural network speculation acceleration system uses the SIMD instruction set in the CPU hardware to accelerate model reasoning, which can avoid the limitations of compilers and platforms in the existing technology, and the speed and efficiency are higher than compiled languages such as C++ and Python.
[0110] The neural network speculation acceleration system also uses a more mature hardware device environment to reduce development costs and model inference costs, and uses the instruction set in its mature ecological environment to accelerate model inference.
[0111] The neural network speculation acceleration system also reduces the cost of environmental development by using SIMD instructions or API interface ecosystems on different hardware devices. Furthermore, the neural network speculation acceleration system environment is simple and efficient to build, integrating numerous and complex multi-platform and multi-architecture hardware devices, enabling the system to accelerate the model's reasoning speed while ensuring low development costs.
[0112] When the inference of the neural network model to be accelerated is performed, when the application uses a combination of CPU and GPU, the mature development environment based on the CPU, the integrated graphics card has low power consumption and high compatibility with the CPU, which can accelerate model reasoning more efficiently.
[0113] By using the CPU to be responsible for the reasoning of some simple modules of the model, GPU computing power can be saved and the reasoning cost can be reduced. The final model division can make the model reasoning have low memory occupancy and shorter time consumption. The backbone network structure with many calculations and complex structure is inferred by the GPU, and the simple modules are inferred by the CPU. When reasoning the model, the CPU uses the SIMD instruction set or API interface to accelerate the model reasoning, while the GPU uses the CUDA instruction set or API interface to accelerate the model reasoning, which reduces GPU energy consumption, reduces the reasoning cost, improves the model reasoning speed, and reduces memory occupancy.
[0114] When the kernel processes data, direct convolution processing is faster and more efficient than compiled language processing. Through machine learning, SIMD and CUDA are tuned and trained to obtain the fastest and most efficient SIMD instruction set for kernel feature map processing, and the division and grouping of kernels by the CUDA instruction set, so that the accelerated neural network model to be speculated can complete its reasoning process while ensuring time, higher efficiency, higher performance, and more balanced application-side load.
[0115] Based on the same inventive concept, the present application also provides Figure 7 An electronic device as shown includes:
[0116] Processor 71;
[0117] a memory 72 for storing instructions executable by the processor 71;
[0118] Among them, the processor 71 is configured to execute to implement a neural network speculation acceleration system as provided above.
[0119] Based on the same inventive concept, the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor 71 of the electronic device, the electronic device can execute and implement a neural network speculation acceleration system as provided above.
[0120] Since the electronic device introduced in this embodiment is an electronic device used to implement the information processing method in the embodiment of the present application, based on the information processing method introduced in the embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the electronic device used by the information processing method in the embodiment of the present application, it belongs to the scope of protection of this application.
[0121] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0123] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0125] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0126] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A neural network speculation acceleration method, characterized in that: The method comprises: Get the total number of operator layers of the neural network model to be inferred and accelerated; Determine the optimal number of operator layers corresponding to each hardware device respectively, wherein the number of hardware devices is several, the hardware devices are used to speculate the accelerated neural network model to be speculated, the sum of the optimal number of operator layers corresponding to each hardware device is equal to the total number of operator layers, and the types of hardware devices include edge computing devices, cloud servers, and terminal devices; Control each hardware device to infer the optimal number of operator layers corresponding to each device.
2. A neural network speculation acceleration method as claimed in claim 1, characterized in that: Determine the optimal number of operator layers for the hardware device, including: Set the initial number of operator layers for the hardware device; Under the rated power of the hardware device, determine the target time consumption of the hardware device to estimate the number of initial operator layers; Adjust the number of initial operator layers; Under the rated power of the hardware device, re-determine the target time consumption of the hardware device to infer the adjusted initial operator layers; When the target time is the shortest and the number of initial operator layers is the largest, the initial operator layer number is used as the optimal operator layer number for the hardware device.
3. A neural network speculation acceleration method as claimed in claim 1, characterized in that: Determine the optimal number of operator layers for the hardware device, including: Obtaining hardware indicators corresponding to the hardware device and operator parameters corresponding to the accelerated neural network model to be inferred; The hardware indicators corresponding to the hardware device and the operator parameters are input into the preset inference neural network model to obtain the optimal number of operator layers corresponding to the hardware device.
4. A neural network speculation acceleration method as claimed in claim 2 or 3, characterized in that: After determining the optimal number of operator layers corresponding to the hardware device, it also includes: According to the bit width of the hardware device, the data of the optimal number of operator layers corresponding to the hardware device is cropped and filled; The optimal operator layer number corresponding to the hardware device is optimized by adjusting the number of threads and the size of the data blocks of the optimal operator layer number corresponding to the hardware device.
5. A neural network speculation acceleration method as claimed in claim 3, characterized in that: The formula for calculating the neural network model is preset, including: time=f(c,n,d,s) Among them, time is the estimation time, c is the CPU frequency, n is the operator type, d is the shape of the input data, s is the speed of memory exchange, L is the error between the predicted value and the training data, t p is the prediction time, t q is the actual time consumed, f() is the loss function, and i is the index of the training data.
6. A neural network speculation acceleration method as claimed in claim 3, characterized in that: Build an inference neural network model, including: Obtaining hardware indicators corresponding to several hardware devices, the number of operator layers corresponding to each hardware device, the inference time consumption of the hardware device under the number of operator layers, and the operator parameters corresponding to the accelerated neural network model to be inferred; Inputting the hardware indicators corresponding to the hardware device, the number of operator layers corresponding to the hardware device, and the operator parameters corresponding to the accelerated neural network model to be speculated into the initial neural network model to obtain the simulation results corresponding to the hardware device; When a number of simulation results and the corresponding inference time are both less than a preset threshold, the initial neural network model is used as the inferred neural network model.
7. A neural network speculation acceleration method as claimed in claim 1, characterized in that: The hardware indicators include: Including memory, bandwidth, computing speed, power consumption, and input data of the network layer.
8. A neural network speculation acceleration system, characterized in that: A neural network speculation acceleration method applied to any one of claims 1-7, the system comprising: Client, server and application, the application includes edge computing devices, cloud servers and terminal devices; The client sends a speculation request to the server; The server receives the inference request and receives the inference result of the application; The server returns the inference result to the client so that the client can display the inference result.
9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute to implement a neural network speculation acceleration method as described in any one of claims 1 to 7.
10. A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to implement a neural network speculation acceleration method as described in any one of claims 1 to 7.
Citation Information
Cited By
Heterogeneous sensing adaptive low-bit neural network deployment method
CN121390167A
Hetero-aware adaptive low-bit neural network deployment method
CN121390167B
