Deep learning model online reasoning method, device, electronic equipment and storage medium

Abstract

The present invention provides a deep learning model online reasoning method, device, electronic equipment and storage medium. The method includes: receiving an online reasoning request through an RPC call interface corresponding to the deep learning framework; acquiring the depth based on the deployment of the deep learning framework A node configuration file corresponding to the learning model, the node configuration file including IP addresses and ports of multiple nodes; determining a node from the multiple nodes as an inference node, and according to the IP address and port of the inference node The online inference request is sent to the inference node, so that the inference node invokes the deep learning model to obtain an inference result; and the inference result returned by the inference node is received. The present invention realizes unified reception of online reasoning requests through the RPC call interface, and determines the reasoning node for performing online reasoning from multiple nodes deploying a deep learning model through a load balancing strategy, thereby improving online reasoning performance.

Description

technical field [0001] The present invention relates to the technical field of deep learning, in particular to an online reasoning method, device, electronic equipment and storage medium of a deep learning model. Background technique [0002] When doing deep learning algorithms, after the model training is completed, it is often necessary to deploy the model in the production environment to achieve online reasoning. The most common way is to provide an API on the server, that is, the client sends a request in a specific format to an API of the server, and the server performs calculations through the model after receiving the request data, and returns the result. [0003] Deep learning algorithms can be implemented through the deep learning framework TensorFlow model or PyTorch model during online reasoning. In the prior art, these two deep learning frameworks are deployed on a single node, which leads to poor online reasoning performance when the amount of data is large. M...

AI Technical Summary

Problems solved by technology

In the existing technology, these two deep learning frameworks are deployed on a single node, which leads to poor online reasoning performance when the amount of data is large

Moreover, there are various deep learning frameworks. If developers use different deep learning frameworks, they need to be familiar with multiple frameworks, and the learning cost is high.

Images