Unlock instant, AI-driven research and patent intelligence for your innovation.

Batch reasoning method and device for improving utilization rate of deep learning reasoning equipment and medium

A technology of deep learning and reasoning method, applied in the computer field, which can solve problems such as the inability to meet such needs

Inactive Publication Date: 2020-08-11
INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Unlike AI training, which has a relatively fixed computing cycle and long running time, the call of AI reasoning will fluctuate with the fluctuation of business, and there are often phenomena similar to high during the day and low at night
And in the case of large-scale and high-concurrency node requirements, the conventional deployment scheme obviously cannot meet such requirements. At this time, it is necessary to use an adaptive scheduling algorithm to complete predictive reasoning

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Batch reasoning method and device for improving utilization rate of deep learning reasoning equipment and medium
  • Batch reasoning method and device for improving utilization rate of deep learning reasoning equipment and medium
  • Batch reasoning method and device for improving utilization rate of deep learning reasoning equipment and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] Embodiments of the present invention are described below. It is to be understood, however, that the disclosed embodiments are merely examples and that other embodiments may take various alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention. As will be understood by persons of ordinary skill in the art, various features shown and described with reference to any one figure can be combined with features shown in one or more other figures to create embodiments not explicitly shown or described . Combinations of features shown provide representative embodiments for typical applications. However, various combinations and modifications of the features consiste...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a batch reasoning method and device for improving the utilization rate of deep learning reasoning equipment, and a medium, and the method comprises the following steps: receiving and analyzing a user request, and transmitting the analyzed user request data and a current thread ID to a transmission queue; sequentially reading the user request data and the corresponding threadIDs from the transmission queue and storing the user request data and the corresponding thread IDs into a data structure; in response to the fact that the user request data volume in the data structure meets the preset requirement, sending all the user request data in the data structure to an inference device for inference; and obtaining inference results and sequentially distributing the inference results to corresponding users. A large number of user requests can be combined, the reasoning frequency is reduced, the system throughput rate is increased, the average return time of single requests is shortened, and the user experience is improved.

Description

technical field [0001] The present invention relates to the field of computers, and more specifically, to a batch reasoning method, device and medium for improving the utilization rate of deep learning reasoning equipment. Background technique [0002] In AI projects, most of the time developers focus on how to train, how to tune the model, and how to achieve a satisfactory recognition rate. But for a complete project, it is usually a demand-driven project, and at the same time, the project must eventually fall into the actual business to meet the demand. [0003] For AI training and machine learning tools such as TensorFlow, it also provides the AI ​​Serving tool TensorFlow Serving. With this tool, you can simply save the trained model as a model file, then load the model in TensorFlow Serving through a script, input the data to be inferred, and get the inference result. Unlike AI training, which has a relatively fixed computing cycle and long running time, the call of AI...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06N5/04G06F9/54
CPCG06F9/546G06N5/04
Inventor 张荣国
Owner INSPUR SUZHOU INTELLIGENT TECH CO LTD