Automatic reasoning acceleration method of deep learning model and related device

Abstract

The invention discloses an automatic reasoning acceleration method for a deep learning model and a related device, and relates to the technical field of software, and the method comprises the steps: obtaining an original deep learning model, and exporting the original deep learning model as a deep learning model in an ONNX format; performing low-rank quantization processing on the deep learning model in the ONNX format to obtain an optimized deep learning model; and based on a dynamic concurrent processing mechanism, performing deployment and reasoning on the optimized deep learning model. By exporting the deep learning model into the ONNX format, it can be ensured that the model can achieve efficient reasoning on various hardware platforms, and through low-rank quantization optimization and dynamic concurrent processing, the reasoning speed can be remarkably increased and computing resource consumption can be reduced while high precision of the model is kept.

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to an automatic reasoning acceleration method and related devices for a deep learning model. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning models have become a key tool for solving complex problems. These models have shown great potential in many fields such as image recognition, natural language processing, and medical diagnosis.

[0003] However, with the rapid increase in business volume, the existing deep learning model inference framework has been unable to meet the growing performance requirements. The main problems are slow inference speed and high computing resource usage, which not only affects the user experience, but also increases the operating costs of enterprises.

[0004] Therefore, how to improve the reasoning speed of deep learning models and reduce the consumption of computing resources has become an urgent problem that needs to be solved at this stage. Summary of the invention

[0005] In view of the above problems, this application provides a method and related device for accelerating the automatic reasoning of a deep learning model to achieve the purpose of improving the reasoning speed of the deep learning model and reducing the consumption of computing resources. The specific scheme is as follows: The first aspect of the present application provides an automatic reasoning acceleration method for a deep learning model, and the automatic reasoning acceleration method for a deep learning model comprises: Obtain an original deep learning model, and export the original deep learning model as a deep learning model in ONNX format; Performing low-rank quantization processing on the deep learning model in the ONNX format to obtain an optimized deep learning model; Based on a dynamic concurrent processing mechanism, the optimized deep learning model is deployed and inferred.

[0006] In a possible implementation, exporting the original deep learning model to a deep learning model in ONNX format includes: The original deep learning model is converted into a deep learning model in the ONNX format through the torch library in Python.

[0007] In a possible implementation, performing low-rank quantization processing on the deep learning model in the ONNX format includes: Preprocessing the deep learning model in ONNX format; Dynamically quantize the preprocessed deep learning model to reduce the rank of model parameters.

[0008] In a possible implementation, the performing low-rank quantization processing on the deep learning model in the ONNX format further includes: The dynamically quantized deep learning model is tested for accuracy and speed, and after the accuracy and speed tests pass, the dynamically quantized deep learning model is used as the optimized deep learning model.

[0009] In a possible implementation, the deploying and reasoning the optimized deep learning model based on a dynamic concurrent processing mechanism includes: Through configuration management, running images and containerized deployment, the optimized deep learning model is deployed to the target hardware to achieve: The inference server generates a request queue according to the request time of different clients; generates a batch to be processed according to a preset maximum batch size and a maximum waiting time, wherein the batch includes at least one request with the earliest request time in the request queue; schedules the batch to the target hardware to infer each request in the batch through the optimized deep learning model; and returns the inference result corresponding to each request in the batch by the optimized deep learning model to the corresponding client.

[0010] A second aspect of the present application provides an automatic reasoning acceleration device for a deep learning model, the automatic reasoning acceleration device for a deep learning model comprising: A model export module, used to obtain the original deep learning model and export the original deep learning model into a deep learning model in ONNX format; A model optimization module, used to perform low-rank quantization processing on the deep learning model in the ONNX format to obtain an optimized deep learning model; The model service module is used to deploy and reason about the optimized deep learning model based on a dynamic concurrent processing mechanism.

[0011] In a possible implementation, the model export module for exporting the original deep learning model into a deep learning model in ONNX format is specifically used to: The original deep learning model is converted into a deep learning model in the ONNX format through the torch library in Python.

[0012] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the automatic reasoning acceleration method of the deep learning model of the above-mentioned first aspect or any implementation method of the first aspect.

[0013] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the automatic reasoning acceleration method of the deep learning model of the above-mentioned first aspect or any implementation method of the first aspect.

[0014] The fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the automatic reasoning acceleration method of the deep learning model of the above-mentioned first aspect or any implementation method of the first aspect.

[0015] By means of the above technical solution, the present application provides an automatic reasoning acceleration method and related device for a deep learning model, which obtains the original deep learning model and exports the original deep learning model as a deep learning model in ONNX format; performs low-rank quantization processing on the deep learning model in ONNX format to obtain an optimized deep learning model; and deploys and reasoned the optimized deep learning model based on a dynamic concurrent processing mechanism. By exporting the deep learning model to ONNX format, the present application can ensure that the model can achieve efficient reasoning on a variety of hardware platforms, and through low-rank quantization optimization and dynamic concurrent processing, it can significantly improve the reasoning speed and reduce computing resource consumption while maintaining the high accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0017] Figure 1 A flowchart of an automatic reasoning acceleration method for a deep learning model provided in an embodiment of the present application; Figure 2 A partial flow chart of a method for accelerating automatic reasoning of a deep learning model provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an automatic reasoning acceleration device for a deep learning model provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0019] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0020] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0021] To facilitate understanding of this application, the following first describes the relevant concepts involved in this application: Deep learning model: Deep learning specifically refers to machine learning based on deep neural network models and methods. It is developed based on statistical machine learning, artificial neural network and other algorithm models, combined with the development of contemporary big data and large computing power.

[0022] Low-rank quantization: Low-rank quantization is a method to reduce the complexity of a model by reducing the rank of model parameters. This method is widely used in image and video processing, pattern recognition, and computer vision. In particular, in scenarios where large-scale data transmission needs to be processed, low-rank quantization can effectively reduce hardware costs and improve data transmission speed and accuracy.

[0023] With the rapid development of artificial intelligence technology and the wide application of deep learning models in all walks of life, the slow speed of model reasoning and the high consumption of computing resources have become the main bottlenecks restricting technological progress and application expansion. In practical applications, the amount of data that deep learning models need to process is becoming increasingly large, and the existing deep learning model reasoning framework often cannot meet the growing computing needs, resulting in the reasoning process taking too long and the computing resources being too high, which not only affects the user experience, but also increases the operating costs of enterprises, limiting the further popularization and application of deep learning technology. In view of the problems of efficiency and resource management in the process of deep learning model reasoning, this application provides an automatic reasoning acceleration method for a deep learning model, which achieves the effect of significantly improving the reasoning speed and reducing resource consumption while maintaining the accuracy of the model through a series of steps such as model export, configuration and management of model file libraries, running mirrors and containerized deployment, and requesting dynamic concurrent processing. This application can meet the market's urgent demand for efficient, low-cost, and high-performance deep learning model reasoning processing, in order to promote the application and development of deep learning technology in a wider range of fields.

[0024] See also Figure 1 , Figure 1 A flowchart of an automatic reasoning acceleration method for a deep learning model provided in an embodiment of the present application. Figure 1 As shown, an automatic reasoning acceleration method for a deep learning model provided in an embodiment of the present application may include steps S10 to S30, and these steps are described in detail below.

[0025] S10, obtaining the original deep learning model, and exporting the original deep learning model to a deep learning model in ONNX format.

[0026] In an embodiment of the present application, a deep learning model to be deployed is obtained. The original deep learning model generally stores model parameters in an internally serialized state dictionary format. The keys of the state dictionary are the names of the networks in each layer of the deep learning model, and the values ​​are the weights and activation function outputs of the network layer to which it belongs.

[0027] Export the original deep learning model to a deep learning model in ONNX (Open Neural Network Exchange) format. As an open model format standard, ONNX allows deep learning models to be flexibly converted and deployed between different deep learning model reasoning frameworks, thereby improving the portability and compatibility of deep learning models. Model export can ensure that deep learning models can achieve efficient reasoning on a variety of hardware platforms, laying a solid foundation for subsequent model optimization.

[0028] In practical applications, the torch library in Python can be used to convert the original deep learning model into a deep learning model in ONNX format with one click. Specifically, you can input the original deep learning model and data samples, and export the model into an ONNX format file.

[0029] S20, performs low-rank quantization processing on the deep learning model in ONNX format to obtain an optimized deep learning model.

[0030] In the embodiment of the present application, the deep learning model in ONNX format can be subjected to low-rank quantization processing through static quantization to obtain an optimized deep learning model. For example, an 8-bit quantization algorithm can be selected to convert the floating-point values ​​of the model parameters of the deep learning model into 8-bit integer values ​​to reduce the rank of the model parameters. In this process, it is necessary to define quantization parameters (such as quantization range, step size, etc.), execute the configured quantization function, traverse each layer of the deep learning model network, and perform quantization calculations on the weights and activation function outputs of each layer of the network, and replace the original high-precision floating-point values ​​with low-bit integers, thereby reducing the model data storage size and computational complexity.

[0031] In addition, based on the above basic quantization, different levels of optimization can be performed for different hardware by defining operator libraries, so as to select the best optimization operator for different hardware and achieve hardware deep optimization. For example, the calculation method of FP8 or INT8 operators is different from that of the default FP16 or FP32. Therefore, by performing low-rank quantization on the deep learning model in ONNX format, the rank of the model parameters can be reduced, thereby reducing the complexity of the deep learning model. Through targeted hardware deep optimization, the inference speed of the deep learning model can be significantly improved while maintaining high precision, the latency can be reduced, and the size of the deep learning model can be reduced, thereby improving the inference efficiency.

[0032] See also Figure 2 , Figure 2 A partial flow chart of an automatic reasoning acceleration method for a deep learning model provided in an embodiment of the present application. Figure 2 As shown, an automatic reasoning acceleration method for a deep learning model provided in an embodiment of the present application, wherein step S20 "performs low-rank quantization processing on the deep learning model in ONNX format" may include steps S201 to S202, and these steps are described in detail below.

[0033] S201, preprocessing the deep learning model in ONNX format.

[0034] In the embodiment of the present application, preprocessing operations such as symbolic shape inference, model optimization, and ONNX shape inference can be performed on the deep learning model in ONNX format. Among them, symbolic shape inference is applicable to the transformer model; model optimization can use its own operator library to rewrite the calculation graph to improve runtime efficiency. Preprocessing can improve the quantization quality, help determine the tensor shape, and make the quantization tool run better.

[0035] S202, dynamically quantize the preprocessed deep learning model to reduce the rank of the model parameters.

[0036] In the embodiment of the present application, multiple low-rank quantization methods (such as int8, fp8, fp32, etc.) are supported for dynamic quantization. For the preprocessed deep learning model, a low-rank quantization scheme can be used for dynamic quantization. Taking fp32 as an example, the model parameter val_fp32 = scale * (val_quantized - zero_point) after rank reduction, where scale is a preset positive real number used to map floating point numbers to the quantization space; val_quantized is the original model parameter, that is, a floating point number; zero_point is the zero value in the quantization space, which can be obtained by calculating the mean of the original model parameter.

[0037] On this basis, in order to verify whether the accuracy and inference speed of the deep learning model after dynamic quantization meet the requirements, the above step S20 "performing low-rank quantization processing on the deep learning model in ONNX format" may also include the following steps: The dynamic quantized deep learning model is tested for accuracy and speed, and after passing the accuracy and speed tests, the dynamic quantized deep learning model is used as the optimized deep learning model.

[0038] In the embodiment of the present application, test samples for accuracy and speed test can be randomly generated to test the output of the deep learning model before and after dynamic quantization, and compare the cosine similarity of the two outputs. Of course, the speed will also be tested and compared, and the accuracy, time consumption and other comparison information will be output at the same time. Only after the accuracy and speed tests pass, the deep learning model after dynamic quantization is used as the optimized deep learning model.

[0039] S30, deploys and infers optimized deep learning models based on a dynamic concurrent processing mechanism.

[0040] In an embodiment of the present application, an optimized deep learning model is deployed, and dynamic concurrent model reasoning is implemented, so that the reasoning service can process multiple client requests at the same time, and optimize the processing by dynamically combining requests, thereby improving the response speed and throughput of the entire reasoning service.

[0041] In one possible implementation, the inference service can be implemented through configuration management of the model file library, running the image and containerized deployment. In this regard, the embodiment of the present application provides an automatic inference acceleration method for a deep learning model, wherein the above step S30 "deploys and infers the optimized deep learning model based on a dynamic concurrent processing mechanism" can adopt the following steps: Deploy optimized deep learning models to target hardware through configuration management, running images, and containerized deployment to achieve: The inference server generates a request queue according to the request time of different clients; generates a batch to be processed according to the preset maximum batch size and maximum waiting time, and the batch contains at least one request with the earliest request time in the request queue; schedules the batch to the target hardware to infer each request in the batch through the optimized deep learning model; and returns the inference results corresponding to each request in the batch by the optimized deep learning model to the corresponding client.

[0042] In an embodiment of the present application, after obtaining an optimized deep learning model, the input and output parameters of the model can be defined through a configuration file, and technologies such as dynamic batching, response caching, and instance groups can be used to improve the reasoning efficiency of the model. This ensures efficient management and flexible configuration of the model file library, and provides support for the rapid deployment and reasoning of model services.

[0043] Furthermore, in the embodiments of the present application, Docker container technology is used to deploy the inference server, which supports multiple deep learning model inference frameworks and can efficiently manage the deployment and inference of the model. By running the configured Docker image, the optimized deep learning model can be run on the target hardware to achieve efficient service. Among them, the target hardware can be a GPU (Graphics Processing Unit).

[0044] Finally, the embodiment of the present application writes a client code to send a request to the inference server. After receiving the client's request, the inference server uses the optimized deep learning model to perform inference and returns the inference result to the client. Among them, the inference server has the ability of dynamic concurrent processing and can process multiple requests at the same time, which greatly improves the inference efficiency of the model and the throughput of the entire service.

[0045] Specifically, after receiving the request from the client, the inference server can put the request into the request queue according to the request time of the client. For the requests in the request queue, the requests are processed in order from the head of the queue to the tail of the queue, that is, the requests are processed in order from the earliest to the latest request time. Set the maximum batch size and the maximum waiting time, add the request at the head of the queue to the existing batch or create a new batch in sequence, if the batch reaches the maximum batch size or the difference between the request time of the request and the current time reaches the maximum waiting time, the batch is treated as a batch to be processed. Schedule the batch to be processed to the target hardware to perform concurrent inference on each request in the batch through the optimized deep learning model, and record the inference results of each request in the result list and return it to the inference server, and the inference server returns the inference results of each request in the result list to their corresponding clients.

[0046] Of course, in actual applications, a recommended batch size may also be set. When the request at the head of the queue reaches an integer multiple of the recommended batch size, the request at the head of the queue may also be divided into at least one batch.

[0047] Through the above description, the automatic reasoning acceleration method of a deep learning model provided by the embodiment of the present application not only improves the reasoning speed of the deep learning model, but also reduces the consumption of computing resources, thereby achieving efficient and low-cost deep learning model reasoning processing while maintaining the accuracy of the model, and providing strong support for the application of deep learning technology in a wider range of fields. This application has the following advantages: 1) One-click model deployment function: It can simplify the model deployment process, reduce technical barriers, and enable users to deploy and use deep learning models more conveniently.

[0048] 2) Optimize resource management and reduce costs. Through model optimization and concurrent processing technology, the dependence on computing resources is reduced, thereby reducing hardware costs and energy consumption. This not only improves resource utilization, but also meets the modern requirements of environmental protection and energy saving.

[0049] 3) Technical adaptability and scalability. The flexibility and scalability of the technology are taken into consideration, and the adaptability to new model structures and algorithm updates is increased to ensure that this application can adapt to the rapid development of deep learning technology and the diversification of business needs.

[0050] This application significantly improves the efficiency of model reasoning and system throughput through low-rank quantization technology and dynamic concurrent processing mechanism. Specifically: The model's complexity is effectively reduced through model low-rank quantization technology. This technology reduces the number of model parameters while maintaining the high accuracy of the model. During the inference process, the reduction in the number of model parameters directly leads to a reduction in the amount of calculation, thereby improving the basic inference speed of the model. This optimization not only speeds up the inference speed, but also reduces the model's demand for computing resources, allowing the model to run efficiently in a resource-constrained environment.

[0051] Introduced dynamic concurrent processing capabilities. In traditional model reasoning frameworks, requests are often processed serially, which can lead to inefficiency and response delays when faced with high-concurrency requests. The dynamic concurrent processing mechanism of this application allows the system to process multiple client requests at the same time, reducing the time for request queuing and improving the system's response speed and throughput. This mechanism is particularly suitable for processing large-scale data and high-concurrency scenarios, and can significantly improve user experience and service efficiency.

[0052] The one-click model deployment system simplifies the model service launch process and lowers the technical threshold. This rapid deployment capability enables the model to respond quickly to changes in business needs and accelerates the transformation process from model development to actual application. This flexibility and convenience are essential for the development process of rapid iteration and continuous integration.

[0053] By reducing the reliance on computing resources, the present application reduces hardware costs and energy consumption, which not only improves resource utilization, but also meets the modern requirements of environmental protection and energy saving. This resource optimization management makes the present application advantageous at both economic and environmental levels.

[0054] The above introduces an automatic reasoning acceleration method for a deep learning model provided in an embodiment of the present application. The following will introduce a device for executing the automatic reasoning acceleration method for the above-mentioned deep learning model.

[0055] See also Figure 3 , Figure 3 A schematic diagram of the structure of an automatic reasoning acceleration device for a deep learning model provided in an embodiment of the present application. Figure 3 As shown, an automatic reasoning acceleration device for a deep learning model provided in an embodiment of the present application includes: The model export module 10 is used to obtain the original deep learning model and export the original deep learning model into a deep learning model in ONNX format; The model optimization module 20 is used to perform low-rank quantization processing on the deep learning model in ONNX format to obtain an optimized deep learning model; The model service module 30 is used to deploy and reason about the optimized deep learning model based on a dynamic concurrent processing mechanism.

[0056] In a possible implementation, the model export module 10 for exporting the original deep learning model into a deep learning model in ONNX format is specifically used to: The original deep learning model is converted into a deep learning model in ONNX format through the torch library in Python.

[0057] In a possible implementation, the model optimization module 20 for performing low-rank quantization processing on a deep learning model in ONNX format is specifically used to: Preprocess the deep learning model in ONNX format; dynamically quantize the preprocessed deep learning model to reduce the rank of model parameters.

[0058] In a possible implementation, the model optimization module 20 for performing low-rank quantization processing on a deep learning model in ONNX format is further used to: The dynamic quantized deep learning model is tested for accuracy and speed, and after passing the accuracy and speed tests, the dynamic quantized deep learning model is used as the optimized deep learning model.

[0059] In a possible implementation, the model service module 30 is specifically configured to: Deploy optimized deep learning models to target hardware through configuration management, running images, and containerized deployment to achieve: The inference server generates a request queue according to the request time of different clients; generates a batch to be processed according to the preset maximum batch size and maximum waiting time, and the batch contains at least one request with the earliest request time in the request queue; schedules the batch to the target hardware to infer each request in the batch through the optimized deep learning model; and returns the inference results corresponding to each request in the batch by the optimized deep learning model to the corresponding client.

[0060] It should be noted that the detailed functions of each module in the embodiment of the present application can be found in the corresponding public part of the embodiment of the automatic reasoning acceleration method of the above-mentioned deep learning model, which will not be repeated here.

[0061] The present application also provides an electronic device in an embodiment. Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0062] like Figure 4As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0063] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a memory card, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0064] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements an automatic reasoning acceleration method for any deep learning model provided in the embodiment of the present application.

[0065] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement an automatic reasoning acceleration method for any deep learning model provided in an embodiment of the present application.

[0066] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0067] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0068] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0069] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for accelerating automatic reasoning of a deep learning model, characterized in that: The automatic reasoning acceleration method of the deep learning model includes: Obtain an original deep learning model, and export the original deep learning model as a deep learning model in ONNX format; Performing low-rank quantization processing on the deep learning model in the ONNX format to obtain an optimized deep learning model; Based on a dynamic concurrent processing mechanism, the optimized deep learning model is deployed and inferred.

2. The automatic reasoning acceleration method of the deep learning model according to claim 1, characterized in that: The exporting of the original deep learning model into a deep learning model in ONNX format includes: The original deep learning model is converted into a deep learning model in the ONNX format through the torch library in Python.

3. The automatic reasoning acceleration method of the deep learning model according to claim 1, characterized in that: The low-rank quantization processing of the deep learning model in the ONNX format includes: Preprocessing the deep learning model in ONNX format; Dynamically quantize the preprocessed deep learning model to reduce the rank of model parameters.

4. The automatic reasoning acceleration method of the deep learning model according to claim 3 is characterized in that: The low-rank quantization processing of the deep learning model in the ONNX format further includes: The dynamically quantized deep learning model is tested for accuracy and speed, and after the accuracy and speed tests pass, the dynamically quantized deep learning model is used as the optimized deep learning model.

5. The automatic reasoning acceleration method of the deep learning model according to claim 1, characterized in that: The deployment and reasoning of the optimized deep learning model based on the dynamic concurrent processing mechanism includes: Through configuration management, running images and containerized deployment, the optimized deep learning model is deployed to the target hardware to achieve: The inference server generates a request queue according to the request time of different clients; generates a batch to be processed according to a preset maximum batch size and a maximum waiting time, wherein the batch includes at least one request with the earliest request time in the request queue; schedules the batch to the target hardware to infer each request in the batch through the optimized deep learning model; and returns the inference result corresponding to each request in the batch by the optimized deep learning model to the corresponding client.

6. An automatic reasoning acceleration device for a deep learning model, characterized in that: The automatic reasoning acceleration device of the deep learning model includes: A model export module, used to obtain the original deep learning model and export the original deep learning model into a deep learning model in ONNX format; A model optimization module, used to perform low-rank quantization processing on the deep learning model in the ONNX format to obtain an optimized deep learning model; The model service module is used to deploy and reason about the optimized deep learning model based on a dynamic concurrent processing mechanism.

7. The automatic reasoning acceleration device for deep learning models according to claim 6, characterized in that: The model export module for exporting the original deep learning model into a deep learning model in ONNX format is specifically used for: The original deep learning model is converted into a deep learning model in the ONNX format through the torch library in Python.

8. A computer program product, characterized in that It includes computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the automatic reasoning acceleration method for a deep learning model as described in any one of claims 1 to 5.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the automatic reasoning acceleration method of the deep learning model as described in any one of claims 1 to 5.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the automatic reasoning acceleration method for the deep learning model as described in any one of claims 1 to 5.

Images