Deep learning model online reasoning method and device, electronic equipment and storage medium

Abstract

The invention provides a deep learning model online reasoning method and device, electronic equipment and a storage medium. The method comprises the steps: receiving an online reasoning request through an RPC calling interface corresponding to a deep learning framework; obtaining a node configuration file corresponding to a deep learning model deployed based on the deep learning framework, whereinthe node configuration file comprises IP addresses and ports of a plurality of nodes; determining one node from the plurality of nodes as an inference node, and sending the online inference request to the inference node according to an IP address and a port of the inference node, so as to enable the inference node to call the deep learning model to obtain an inference result; and receiving an inference result returned by the inference node. According to the invention, the online reasoning requests are uniformly received through the RPC calling interface, and the reasoning node for executing online reasoning is determined from the plurality of nodes of the deployment deep learning model through the load balancing strategy, so that the online reasoning performance is improved.

Description

technical field [0001] The present invention relates to the technical field of deep learning, in particular to an online reasoning method, device, electronic equipment and storage medium of a deep learning model. Background technique [0002] When doing deep learning algorithms, after the model training is completed, it is often necessary to deploy the model in the production environment to achieve online reasoning. The most common way is to provide an API on the server, that is, the client sends a request in a specific format to an API of the server, and the server performs calculations through the model after receiving the request data, and returns the result. [0003] Deep learning algorithms can be implemented through the deep learning framework TensorFlow model or PyTorch model during online reasoning. In the prior art, these two deep learning frameworks are deployed on a single node, which leads to poor online reasoning performance when the amount of data is large. M...

AI Technical Summary

Problems solved by technology

In the existing technology, these two deep learning frameworks are deployed on a single node, which leads to poor online reasoning performance when the amount of data is large

Moreover, there are various deep learning frameworks. If developers use different deep learning frameworks, they need to be familiar with multiple frameworks, and the learning cost is high.

Images