Inference board, inference method, device, medium and product based on inference board

Abstract

The present disclosure relates to the field of computer technology, and discloses an inference board, an inference method, a device, a medium and a product based on the inference board. The inference board includes a plurality of inference units representing a set of hardware resources, and the method includes: obtaining data to be inferred; determining a second number of inference models allowed to run in parallel according to the total amount of data to be inferred and a first number of inference units allowed to be used in parallel in the inference board; respectively allocating inference units to the second number of inference models, and loading each inference model into the allocated inference unit; dividing the data to be inferred into a plurality of data subsets, and allocating data to be inferred in different data subsets to each inference model loaded into the inference unit, so that the inference unit runs the inference model in parallel, and performs parallel inference on the acquired data to be inferred. The efficiency of model inference can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an inference board, an inference method, a device, a medium and a product based on the inference board. Background Art

[0002] At present, in some artificial intelligence (AI) frameworks, model reasoning usually uses the default reasoning unit in the reasoning controller. When the user inputs multiple data to be reasoned, reasoning tasks can only be issued to the same reasoning unit in a cyclic sequence, and the execution efficiency of the reasoning task is low. Summary of the invention

[0003] In view of this, the present disclosure provides an inference board, an inference method based on the inference board, an electronic device, a computer-readable storage medium and a computer program product, which can improve the execution efficiency of the inference task.

[0004] In a first aspect, the present disclosure provides an inference board, the inference board comprising:

[0005] A communication interface, used for communication connection with a host;

[0006] A plurality of inference units representing a set of hardware resources, wherein at least some of the inference units are used to perform the following operations in parallel:

[0007] Receiving, through the communication interface, the inference model sent by the host and the data to be inferred corresponding to the inference model;

[0008] Run the received inference model to infer the data to be inferred corresponding to the inference model;

[0009] The reasoning models running in the respective reasoning units are the same, and the data to be inferred corresponding to at least some of the reasoning models are different.

[0010] In a second aspect, the present disclosure provides a reasoning method based on the above-mentioned reasoning board, the method comprising:

[0011] Obtain the data to be inferred;

[0012] Determining a second number of inference models allowed to run in parallel according to the total amount of the data to be inferred and the first number of inference units allowed to be used in parallel in the inference board;

[0013] Allocating inference units to the second number of inference models respectively, and loading each of the inference models into the allocated inference units;

[0014] The data to be inferred are divided into a plurality of data subsets, and the data to be inferred in different data subsets are allocated to each inference model loaded into the inference unit, so that the inference unit runs the inference models in parallel and performs parallel inference on the acquired data to be inferred.

[0015] In a third aspect, the present disclosure provides an inference device based on an inference board, wherein the inference board includes a plurality of inference units representing a set of hardware resources, and the inference units are used to run an inference model; the device includes:

[0016] A data acquisition module is used to acquire data to be inferred;

[0017] A quantity determination module, used for determining a second number of inference models allowed to run in parallel according to the total amount of the data to be inferred and the first number of inference units allowed to be used in parallel in the inference board;

[0018] an instantiation module, configured to allocate inference units to the second number of inference models respectively, and load each of the inference models into the allocated inference units;

[0019] The inference module is used to divide the data to be inferred into multiple data subsets, and allocate the data to be inferred in different data subsets to each inference model loaded into the inference unit, so that the inference unit runs the inference models in parallel and performs parallel inference on the acquired data to be inferred.

[0020] In a fourth aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the above method by executing the computer instructions.

[0021] In a fifth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the above method.

[0022] In a sixth aspect, the present disclosure provides a computer program product, including computer instructions, and the computer instructions are used to enable a computer to execute the above method.

[0023] In the technical solutions in some embodiments of the present disclosure, after determining the number of inferences required for the data to be inferred based on the total data volume of the data to be inferred and the maximum data volume allowed to be input at a single time by the inference model, the second number of inference models allowed to run in parallel can be determined based on the number of inferences and the first number of inference units allowed to be used in parallel in the inference board, and then the instantiation operation can be performed on the second number of inference models. In this way, parallel inference can be performed on the inference data by running multiple inference models, thereby improving the inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 It is a model reasoning flow chart in the artificial intelligence reasoning framework in some technologies;

[0026] Figure 2 is a schematic diagram of an inference board provided by an embodiment of the present disclosure;

[0027] Figure 3 is a flowchart of a reasoning method provided by an embodiment of the present disclosure;

[0028] Figure 4 is a schematic diagram of a module of an inference device provided by some embodiments of the present disclosure;

[0029] Figure 5 It is a schematic diagram of the structure of an electronic device provided by some embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0031] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0032] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0033] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0034] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0035] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0036] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0037] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0038] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0039] Artificial intelligence reasoning framework, also known as AI reasoning framework, is a software architecture that optimizes models. In addition to accelerating the deployment and reasoning process of models, it can also maximize the performance of models on multiple platforms. Platforms include but are not limited to CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), etc.

[0040] In some AI frameworks, a Stream resource refers to a set of operations that are submitted to an AI processor (such as a GPU) in order for execution. Each Stream is independent, and multiple different Streams can be executed in parallel to improve computing efficiency. A Cluster refers to a group of closely coordinated processors, memory, or other computing resources in an AI processor. Each Cluster has its own corresponding Stream resource, and each Stream resource can only be computed using its corresponding Cluster. Figure 1 , which is a flowchart of model reasoning in the artificial intelligence reasoning framework in some technologies. Figure 1 In , model reasoning mainly includes the following steps:

[0041] 1) Input the model file trained by the training framework into the artificial intelligence framework for compilation. The compiled file can be saved as a so file so that it can be directly loaded and used during the next inference.

[0042] 2) Load the model to generate the mod file. In the process of generating the mod file, the input and output memory can be pre-allocated on the board according to the shape and data_type information of the input and output layers of the model.

[0043] 3) Call the set_input function of the mod file to perform the set_input operation on the input data, that is, to move the input data from the host memory to the memory of the artificial intelligence processor through the driver and the PCIe (Peripheral Component Interconnect Express) bus.

[0044] 4) Call the run function of the mod file, perform reasoning according to the compiled model structure, complete the loading of model weights and other resources, and call the Cluster of the artificial intelligence processor to perform reasoning.

[0045] 5) Call the get_output function of the mod to copy the inference results on the board to the host memory through the PCIe bus and driver, and output the results.

[0046] Figure 1 In the AI ​​processor, the model inference method is to use a default Stream resource in the AI ​​processor. The problem caused by this is that only one set of inference functions set_input / run / get_output can be executed at the same time (that is, using the same Stream resource). If the user inputs multiple data, the inference tasks can only be issued in a cyclic sequence on the same Stream resource. Data cannot be copied and calculated simultaneously between different Clusters of the AI ​​processor, and the pipeline effect cannot be achieved. In addition, when performing multi-batch inference, the data in one batch cannot be calculated in parallel using different Clusters, and the execution efficiency of the inference tasks is low.

[0047] In view of this, the present disclosure first provides an inference board. An inference board, also known as an accelerator card or a GPU accelerator card, is a hardware device specifically used for computing, and is mainly used to accelerate the inference work in artificial intelligence and deep task learning tasks. These tasks include using a trained neural network model to process actual data, that is, the process of obtaining output results from input data through model calculation. Typically, an artificial intelligence processor and other necessary hardware components, such as high-bandwidth memory, storage, etc., can be integrated on the inference board. Based on these hardware resources, the inference board can quickly and accurately complete artificial intelligence reasoning tasks through efficient computing and optimization algorithms.

[0048] For details, please refer to Figure 2 , which is a schematic diagram of an inference board provided in one embodiment of the present disclosure. Figure 2 In the inference board, the inference board includes a communication interface, a high-bandwidth memory, and multiple inference units. Each inference unit can represent a set of hardware resources on the inference board (such as a set of processors, memories, or other computing resources). Based on the set of hardware resources represented by the inference unit, the inference model can be run. The high-bandwidth memory can be used to save the inference results of the model, etc. The communication interface can be used to communicate with the host. The communication interface can include but is not limited to a PCIe interface.

[0049] At least some of the inference units are used to perform the following operations in parallel:

[0050] Receive the inference model and the to-be-inferred data corresponding to the inference model sent by the host through the communication interface;

[0051] Run the received inference model to infer the data to be inferred corresponding to the inference model;

[0052] The reasoning models running in each reasoning unit are the same, and the data to be inferred corresponding to at least some of the reasoning models are different.

[0053] In some embodiments, after the inference model completes the inference of the data to be inferred and obtains the inference result, the inference unit is further used to send the inference result to the host through the communication interface.

[0054] In simple terms, multiple inference units in the inference board disclosed in the present invention can run the same inference model in parallel. In this way, when a batch of data to be inferred is received, the data to be inferred can be divided into multiple data subsets, and different inference models can be assigned data to be inferred in different data subsets, so that the acquired data to be inferred can be inferred in parallel, thereby improving data inference efficiency.

[0055] Based on the above description, in some embodiments of the present disclosure, the host may be used to complete one or more of the following operations:

[0056] 1) Allocate an inference unit to the inference model and load the inference model into the allocated inference unit, that is, allocate hardware resources for running the inference model. In the present disclosure, this process of allocating hardware resources to the inference model is also referred to as instantiation of the inference model.

[0057] 2) Receive the data to be inferred, and send the data to be inferred to the inference board through the communication interface. Specifically, each inference model can have its own corresponding task queue. Sending the data to be inferred to the inference board mainly sends the data to be inferred to the task queue of the inference model running on the inference board, so that the inference model can infer the data to be inferred and obtain the inference result.

[0058] 3) Get the inference results from the inference board.

[0059] In some embodiments of the present disclosure, the host may include hardware resources such as a CPU. The inference board is connected to the CPU in the host through a communication interface, and program codes may be burned into the CPU. During the execution of these program codes, the operations in 1) to 3) above may be completed.

[0060] In some embodiments of the present disclosure, the host may also display a human-computer interaction interface, through which the user may control the model reasoning and other related processes.

[0061] Based on the above description, the present disclosure provides an inference method based on an inference board. The inference method can be applied to a host connected to the inference board. Figure 3 , which is a flowchart of an inference method provided by an embodiment of the present disclosure. Figure 3 In the above example, the inference method includes the following steps:

[0062] Step S301, obtaining data to be inferred.

[0063] Specifically, the data to be inferred may include, but are not limited to, image data, text data, video data, etc. For example, by performing inference on one or more images, it is determined whether each image includes a specified person. For another example, by performing inference on one or more text sentences, the expression of these text sentences is optimized.

[0064] In this embodiment, the host can obtain the data to be inferred input by the user through the human-computer interaction interface. Alternatively, the host can also communicate with other electronic devices and use the data sent by other electronic devices as the data to be inferred. The present disclosure does not limit the method of obtaining the data to be inferred.

[0065] Step S302, determining a second number of inference models allowed to run in parallel according to the total amount of data to be inferred and a first number of inference units allowed to be used in parallel in the inference board.

[0066] In this embodiment, the number of inferences required for the data to be inferred can be determined based on the total amount of data to be inferred and the maximum amount of data allowed to be input into the inference model at one time. Furthermore, the second number of inference models allowed to run in parallel can be determined based on the number of inferences and the first number of inference units allowed to be used in parallel in the inference board.

[0067] Specifically, depending on the data type of the data to be inferred, the statistical method and unit of the total data volume may be different. For example, when the data to be inferred is image data, the total data volume may be the total number of images, such as 10,000 images. For another example, when the data to be inferred is video data, the total data volume may be the total number of video files, such as 1,000 video files. For another example, when the data to be inferred is text data, the total data volume may be the number of sentences consisting of the text, such as 10 sentences, or the total data volume may be the total number of characters included in the text, such as 5,000 characters.

[0068] The statistical method and unit of the maximum amount of data allowed to be input into the inference model at one time can be the same as the statistical method and unit of the total data volume mentioned above. For example, the maximum amount of data allowed to be input into the inference model at one time is 10 pictures. It should be noted that the maximum amount of data allowed to be input into the inference model at one time refers to the maximum amount of data allowed to be received each time the inference model performs an inference task. Usually, the maximum amount of data can be related to the model structure, the performance that the model needs to achieve, etc.

[0069] The number of inferences refers to the total number of calls to a single inference model to complete the inference of all the data to be inferred based on the maximum amount of data allowed to be input at one time. For example, if the maximum amount of data allowed to be input at one time for the inference model is 10 images, and the number of inferences required for the data to be inferred is 10,000 times, the number of inferences required for the data to be inferred is 1,000.

[0070] Normally, the total amount of data to be inferred can be used as the numerator, and the maximum amount of data allowed to be input at a single time by the inference model can be used as the denominator, and the ratio determined in this way can be used as the number of inferences for the data to be inferred. However, in some cases, the ratio determined in this way has the problem of non-integer. For example, assuming that the total amount of data to be inferred is 1001, the maximum amount of data allowed to be input at a single time by the inference model is 10, and the ratio of the total amount of data to the maximum amount of data is 100.1. Since the number of inferences is usually an integer, the above-obtained ratio of 100.1 cannot be directly used as the number of inferences. In view of this, in this embodiment, the number of inferences required for the data to be inferred is determined based on the total amount of data to be inferred and the maximum amount of data allowed to be input at a single time by the inference model, which may specifically include:

[0071] Determine the ratio of the total data volume to the maximum data volume;

[0072] When the obtained ratio is not an integer, the first integer greater than the ratio is used as the value of the inference number.

[0073] For example, for the above 100.1, the first integer greater than 100.1, that is, 101, can be used as the number of inferences. Where 101 is used as the number of inferences, 10 data to be inferred can be input into the inference model each time when the inference model is called for inference for the first 100 times, and only the last data to be inferred can be input into the inference model when the inference model is called for inference for the 101st time. In this way, on the one hand, the problem of the number of inferences being a non-integer is avoided; on the other hand, it is ensured that all data to be inferred can be successfully inferred.

[0074] The first number of inference units allowed to be used in parallel in the inference board may be less than or equal to the total number of inference units included in the inference board. For example, assuming that the inference board includes a total of 6 inference units, the first number may be less than or equal to 6.

[0075] In this embodiment, the first number can be pre-set in the CPU of the host, for example, the first number can be fixedly set to 6 in the CPU. Alternatively, the host can also display a first human-computer interaction interface. The first human-computer interaction interface may include a first input area. Through the first human-computer interaction interface, the user can manually input the first number of inference units allowed to be used in parallel. The host can use the number received through the first input area as the first number of inference units allowed to be used in parallel in the inference board. The setting method of the first number can be selected according to actual conditions. Among them, pre-setting the first number in the CPU of the host can reduce the amount of user operations; setting the first number through the human-computer interaction interface allows the user to flexibly adjust the first number according to actual needs, which is more convenient.

[0076] In this embodiment, running the inference model in parallel means running multiple identical inference models in parallel in the inference board, or running multiple inference model instances of the same inference model in parallel in the inference board. In simple terms, it means that multiple instantiation operations are performed on the same inference model, and in each instantiation operation, the hardware resources allocated to the inference model are not exactly the same, so that multiple identical inference models can be run in parallel in the inference board. For example, for the inference model A used for image reasoning, after the first instantiation operation of the inference model A, the inference unit a can be assigned to the inference model A, and the instantiated inference model A1 that can be run in the inference unit a is obtained. Then, after the second instantiation operation of the inference model A, the inference unit b can be assigned to the inference model A2, and the instantiated inference model A2 that can be run in the inference unit b is obtained. In this way, multiple inference models A (i.e., inference models A1, A2) can be run in the inference board, and these multiple inference models A can be run in parallel, and the reasoning of different pictures can be performed at the same time. For example, inference model A1 performs inference on images p1-p10, and inference model A2 performs inference on images p11-p20. In this way, by running multiple identical inference models in parallel, the inference efficiency can be greatly improved.

[0077] Further, in this embodiment, each inference unit can be used to run one or more inference models respectively. Wherein, when one inference unit runs multiple inference models, the multiple inference models can be run in a time-slicing manner. For example, between time point 1 and time point 2, inference model A1 is run; between time point 2 and time point 3, inference model A2 is run; between time point 3 and time point 4, inference model A1 continues to run; between time point 4 and time point 5, inference model A2 continues to run. Running multiple inference models in a time-slicing manner can make full use of hardware resources and reduce the idle time of hardware resources. For example, assume that inference unit a is used to run inference model A1 and inference model A2. In the process of the host sending the picture to be inferred to the task queue of inference model A1, inference model A1 can perform inference operations without using the hardware resources of inference unit a. At this time, if there is a picture to be inferred in the task queue of inference model A2, then inference unit a can be used to run inference model A2 to infer the picture to be inferred in the task queue of inference model A2. After the images to be inferred in the task queue of inference model A2 are inferred, there are already images to be inferred in the task queue of inference model A1. At this time, inference unit a can be used to run inference model A1 to infer the images to be inferred in the task queue of inference model A1. In the process of inference unit a running inference model A1, the host can send images to be inferred to the task queue of inference model A2, so that after the images to be inferred in the task queue of inference model A1 are inferred, inference unit a can continue to run inference model A2. By running the inference models alternately in this way, the problem of the hardware resources of the inference unit being idle when the host sends inference images to the task queue of the inference model is avoided.

[0078] Generally, the number of inference models running in an inference unit should not be too many or too few. Specifically, when there are too many inference models running, there will be more time slices, which will cause greater performance pressure; when there are too few inference models running (for example, 1), the hardware resources of the inference unit will be idle for a long time, and the hardware resources will not be fully utilized.

[0079] In this embodiment, the number of inference models run by each inference unit is 2.

[0080] Based on the above description, in step S302, the second number of inference models allowed to run in parallel in the inference board can be determined by comprehensively considering the number of inferences, the first number of inference units allowed to be used in parallel in the inference board, the performance of the inference board, the resource utilization of the inference board, etc. For example, the second number cannot exceed twice the first number at most. In this way, it is ensured that among the inference units allowed to run, the number of inference models run by each inference unit does not exceed 2, thereby ensuring the performance of the inference board. In the subsequent embodiments of the present disclosure, a preferred embodiment for determining the second number is provided, which will not be repeated here.

[0081] Step S303: assign inference units to the second number of inference models respectively, and load each inference model into the assigned inference unit.

[0082] Specifically, allocating an inference unit to an inference model is also called an instantiation operation of the inference model. For details, please refer to the relevant description of step S302, which is not repeated here. After the instantiation operation is performed on the second number of inference models, the model files of each inference model can be loaded into the corresponding inference unit respectively to run the inference model in the inference unit.

[0083] Step S304, divide the data to be inferred into multiple data subsets, and allocate the data to be inferred in different data subsets to each inference model loaded into the inference unit, so that the inference unit runs the inference model in parallel and performs parallel inference on the acquired data to be inferred.

[0084] Specifically, in this embodiment, before running the reasoning model in parallel, the data to be inferred can be divided into multiple data subsets according to the maximum amount of data allowed to be input at a time by the reasoning model, and then a second number of data subsets are obtained from the multiple data subsets, and the obtained data subsets are respectively sent to the task queue of the corresponding reasoning model according to the one-to-one correspondence between the data subsets and the reasoning model. For example, assume that the second number is 12 (that is, the number of reasoning models running in parallel is 12). According to the maximum amount of data allowed to be input at a time by the reasoning model, the data to be inferred is divided into 50 data subsets. The amount of data in each data subset is less than or equal to the maximum amount of data allowed to be input at a time by the reasoning model. Then, 12 data subsets can be obtained from these 50 data subsets, and then these 12 data subsets can be sent to the task queue of each reasoning model according to the one-to-one correspondence between the data subsets and the reasoning model, and the reasoning model performs reasoning on the received data to be inferred. After all the data to be inferred in these 12 data subsets are inferred, 12 more data subsets can be obtained from the data subsets that have not been sent, and these 12 data subsets can be sent to the task queues of each inference model in a similar way as above. The sending operation is performed in a loop until all data subsets have been sent to the task queues of the inference models.

[0085] In the above embodiment, by dividing the data to be inferred into multiple data subsets, and according to the one-to-one correspondence between the data subsets and the inference models, the data subsets are respectively sent to the task queues of the corresponding inference models. In this way, the inference tasks can be executed in parallel in multiple inference models, thereby improving the inference efficiency.

[0086] Furthermore, in some embodiments, each data subset has a corresponding sequence number, and the sequence number is used to represent the order in which the data subsets are inferred. The above-mentioned allocating the data to be inferred in different data subsets to each inference model loaded into the inference unit may include:

[0087] For any target data subset in the data subsets, a modulo operation is performed on the sequence number of the target data subset and the second number of the inference model to obtain an operation result i;

[0088] Assign the target data subset to the i-th inference model;

[0089] The value of i is 0 or a positive integer less than the value of the second quantity.

[0090] For example, each data subset can be sorted in the order of 1, 2, ..., n. When obtaining the second number of data subsets from the data subset, the data subsets can be obtained in order from the sequence number from small to large, or from large to small. Assuming that the sequence number of the currently obtained target data subset is 5, and the second number is 12, the second number can be used as the numerator and the sequence number of the target data subset as the denominator to perform a modulo operation (i.e., find the remainder), and the result is 2. This means that the target data subset with the sequence number 5 is sent to the second inference model.

[0091] In this way, data subsets can be evenly distributed among inference models, avoiding the problem that some inference models are assigned too many data subsets and some inference models are assigned too few data subsets. Furthermore, it can prevent the problem of excessive performance pressure on some inference models and idle hardware resources corresponding to some inference models.

[0092] In other embodiments, the amount of data to be inferred included in each data subset may also be determined in combination with the number of inference models that need to be run in each inference unit. Specifically, the number of inference models run in the inference unit may be proportional to the amount of data to be inferred included in each data subset. This is because, when there are more inference models running in the inference unit, there are more time slices, and the time allocated to each inference model is relatively reduced. In this case, if the amount of data in each data subset is too much, then in one of the time slices to which the inference model is allocated, the data to be inferred in each data subset may not be fully processed, which will cause data backlog and affect the inference efficiency.

[0093] In some embodiments, after allocating the data to be inferred from different data subsets to each inference model loaded into the inference unit, the method of the present disclosure may further include:

[0094] In the host, a corresponding storage area is allocated for each data subset;

[0095] The inference results of each data subset are obtained from the inference board, and the obtained inference results are stored in the corresponding storage area.

[0096] In this way, the storage resources on the inference board can be released in time, ensuring that the subsequent data to be inferred can be inferred in time.

[0097] To sum up, in the technical solutions in some embodiments of the present disclosure, after determining the number of inferences required for the data to be inferred based on the total data amount of the data to be inferred and the maximum data amount allowed to be input at a single time by the inference model, the second number of inference models allowed to run in parallel can be determined based on the number of inferences and the first number of inference units allowed to be used in parallel in the inference board, and then the instantiation operation can be performed on the second number of inference models. In this way, parallel inference can be performed on the inference data by running multiple inference models, thereby improving the inference efficiency.

[0098] The scheme of the present disclosure is further described below.

[0099] In some embodiments, the step S302 includes:

[0100] Increase the first quantity by a preset multiple to obtain a target quantity;

[0101] If the value of the number of inferences is less than the value of the target number, determining that the value of the second number is equal to the value of the number of inferences;

[0102] If the value of the number of inferences is greater than or equal to the value of the target number, it is determined that the value of the second number is equal to the value of the target number.

[0103] Specifically, the preset multiple can be determined according to the number of inference models to be run in each inference unit. For example, if two inference models need to be run in each inference unit, the preset multiple is 2; if three inference models need to be run in each inference unit, the preset multiple is 3. And so on.

[0104] For ease of understanding, assume that the first number of inference units allowed to be used in parallel is 6, and the preset multiple is 3. By increasing the first number by 3 times, the target number 18 can be obtained. If the value of the number of inferences is less than 10, the second number of inference models allowed to run in parallel is determined to be 10; if the value of the number of inferences is 50, the second number of inference models allowed to run in parallel is determined to be 18. In this way, when the value of the number of inferences is small, all the data to be inferred can be inferred through a cycle of issuing sub-data sets, which is more efficient; when the value of the number of inferences is large, the number of inference models running in each inference unit can be controlled to prevent performance problems on the inference board.

[0105] In some embodiments, the step of increasing the first quantity by a preset multiple to obtain the target quantity includes:

[0106] Double the first quantity to get the target quantity.

[0107] Specifically, as analyzed in step S303, when the first number is doubled, the number of inference models running in each inference unit is more appropriate, which ensures the inference efficiency without causing excessive performance pressure on the inference board.

[0108] In some embodiments, allocating inference units to the second number of inference models in step S304 may include:

[0109] If the second number of inference models is greater than the first number of inference units, assigning the same inference unit to the plurality of different inference models;

[0110] If the second number of the inference models is less than or equal to the first number of the inference units, different inference units are respectively assigned to each of the inference models.

[0111] For example, assuming that the first number of inference units allowed to be used in parallel is 4, and the second number of inference models allowed to be run in parallel is 12, since the second number is greater than the first number, the number of inference units is insufficient, so one inference unit can be assigned to every three inference models. That is, three inference models can be run in the same inference unit. In this way, each inference model can be assigned to an inference unit.

[0112] For another example, assuming that the first number of inference units allowed to be used in parallel is 6, and the second number of inference models allowed to run in parallel is 5, since the second number is less than the first number, the number of inference units is sufficient. Therefore, a different inference unit can be assigned to each inference model to avoid performance pressure problems caused by running too many inference models in the same inference unit.

[0113] Specifically, as mentioned above, in order to avoid the performance pressure problem caused by running too many inference models in the same inference unit, the same inference unit can be allocated to multiple different inference models according to the correspondence between every two inference models and the same inference unit. In this way, the inference efficiency can be improved while avoiding the performance pressure problem of the inference unit. The relevant principles are described in step S303 and will not be repeated here.

[0114] In some embodiments, the inference model of the present disclosure may be converted based on a trained initial inference model;

[0115] Before obtaining the data to be inferred, the method of the present disclosure may further include:

[0116] Displaying a second human-computer interaction interface, the second human-computer interaction interface including a second input area;

[0117] The initial inference model is converted according to the amount of data that the inference model receives through the second input area and that is allowed to be input at one time.

[0118] Specifically, the amount of data that the inference model allows to be input at one time may be determined by the model structure, parameters, etc. of the inference model. Therefore, before the model is converted, the amount of data that the inference model allows to be input at one time may be specified through the human-computer interaction interface. In this way, during the model conversion process, the initial inference model may be converted based on the amount of data to obtain the inference model in the present disclosure.

[0119] By providing a human-computer interaction interface, the model properties of the inference model (such as the amount of data allowed to be input at a single time) can be specified, and then the model conversion process can be controlled, which makes it more applicable.

[0120] In summary, when performing data reasoning based on the reasoning method disclosed in the present invention, the following steps may be included:

[0121] 1) Use the artificial intelligence framework to convert the trained initial inference model file into a format that can be parsed by the model;

[0122] 2) Use the artificial intelligence framework to compile the converted reasoning model. In this step, you can optimize the reasoning model based on its operator composition and finally compile it into an object that can be executed by the underlying reasoning module;

[0123] 3) Based on the compiled inference model, perform data inference in the manner of steps S301 to S305 above.

[0124] Corresponding to the reasoning method, the present disclosure also provides a reasoning device. Figure 4 , which is a module schematic diagram of an inference device provided in some embodiments of the present disclosure. Figure 4 In the inference device, the inference device includes:

[0125] A data acquisition module 401 is used to acquire data to be inferred;

[0126] A quantity determination module 402 is used to determine a second quantity of inference models allowed to run in parallel according to the total amount of data to be inferred and the first quantity of inference units allowed to be used in parallel in the inference board;

[0127] An instantiation module 403 is used to allocate reasoning units to the second number of reasoning models respectively, and load each reasoning model into the allocated reasoning unit;

[0128] The reasoning module 404 is used to divide the data to be inferred into multiple data subsets, and allocate the data to be inferred in different data subsets to each reasoning model loaded into the reasoning unit, so that the reasoning unit runs the reasoning models in parallel and performs parallel reasoning on the acquired data to be inferred.

[0129] In some embodiments, the quantity determination module 402 is specifically configured to:

[0130] Determine the number of inferences required for the data to be inferred based on the total amount of data to be inferred and the maximum amount of data allowed to be input into the inference model at one time;

[0131] According to the number of inferences and the first number of inference units in the inference board that are allowed to be used in parallel, a second number of inference models that are allowed to run in parallel is determined.

[0132] In some embodiments, the quantity determination module 402 is specifically configured to:

[0133] Increase the first quantity by a preset multiple to obtain a target quantity;

[0134] If the value of the number of inferences is less than the value of the target number, determining that the value of the second number is equal to the value of the number of inferences;

[0135] If the value of the number of inferences is greater than or equal to the value of the target number, it is determined that the value of the second number is equal to the value of the target number.

[0136] In some embodiments, the quantity determination module 402 is specifically configured to:

[0137] Double the first quantity to get the target quantity.

[0138] In some embodiments, the quantity determination module 402 is specifically configured to:

[0139] Determine the ratio of the total data volume to the maximum data volume;

[0140] When the obtained ratio is not an integer, the first integer greater than the ratio is used as the value of the inference number.

[0141] In some embodiments, the quantity determination module 402 is further configured to:

[0142] Displaying a first human-computer interaction interface, the first human-computer interaction interface including a first input area;

[0143] The number received through the first input area is used as the first number of inference units allowed to be used in parallel in the inference board.

[0144] In some embodiments, the instantiation module 403 is specifically used to:

[0145] If the second number of inference models is greater than the first number of inference units, assigning the same inference unit to the plurality of different inference models;

[0146] If the second number of the inference models is less than or equal to the first number of the inference units, different inference units are respectively assigned to each of the inference models.

[0147] In some embodiments, the instantiation module 403 is specifically used to:

[0148] According to the corresponding relationship that every two reasoning models correspond to the same reasoning unit, the same reasoning unit is allocated to multiple different reasoning models.

[0149] In some embodiments, each data subset has a corresponding sequence number, and the sequence number is used to represent the order in which the data subsets are inferred; the instantiation module 403 is specifically used to:

[0150] For any target data subset in the data subsets, a modulo operation is performed on the sequence number of the target data subset and the second number of the inference model to obtain an operation result i;

[0151] Assign the target data subset to the i-th inference model;

[0152] The value of i is 0 or a positive integer less than the value of the second quantity.

[0153] In some embodiments, the inference model is converted based on a trained initial inference model;

[0154] Before acquiring the data to be inferred, the data acquisition module 401 is further used to:

[0155] Displaying a second human-computer interaction interface, the second human-computer interaction interface including a second input area;

[0156] The initial inference model is converted according to the amount of data that the inference model receives through the second input area and that is allowed to be input at one time.

[0157] The reasoning device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0158] The inference device disclosed in the present invention has the same beneficial effects as the above-mentioned inference method, which will not be described in detail here.

[0159] The present disclosure also provides an electronic device having the above Figure 4 The inference device shown.

[0160] Combined with reference Figure 5 , is a schematic diagram of the structure of an electronic device provided by some embodiments of the present disclosure. Figure 5As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0161] The processor 10 may be a first PCIe device, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0163] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0165] The electronic device further comprises a communication interface 30 for the electronic device to communicate with other devices or a communication network.

[0166] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0167] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0168] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An inference board, characterized in that: The inference board comprises: A communication interface, used for communication connection with a host; A plurality of inference units representing a set of hardware resources, wherein at least some of the inference units are used to perform the following operations in parallel: Receiving, through the communication interface, the inference model sent by the host and the data to be inferred corresponding to the inference model; Run the received inference model and perform inference on the to-be-inferred data corresponding to the inference model; Among them, the reasoning models running in each of the inference units are the same, and the data to be inferred corresponding to at least some of the inference models are different, and if the second number of the inference models is greater than the first number of the inference units, the same inference unit is allocated to multiple different inference models; if the second number of the inference models is less than or equal to the first number of the inference units, different inference units are allocated to each of the inference models, and each of the inference models has its own corresponding task queue, and sending the data to be inferred to the inference board card means sending the data to be inferred to the task queue of the inference model running on the inference board card.

2. The inference board according to claim 1, characterized in that: After the inference model completes the inference of the data to be inferred and obtains the inference result, the inference unit is further used to send the inference result to the host through the communication interface.

3. An inference method based on the inference board according to claim 1 or 2, characterized in that: The method comprises: Obtain the data to be inferred; Determining a second number of inference models allowed to run in parallel according to the total amount of the data to be inferred and the first number of inference units allowed to be used in parallel in the inference board; Allocating inference units to the second number of inference models respectively, and loading each of the inference models into the allocated inference units; Dividing the data to be inferred into multiple data subsets, and allocating the data to be inferred in different data subsets to each inference model loaded into the inference unit, so that the inference unit runs the inference models in parallel and performs parallel inference on the acquired data to be inferred; The allocating inference units to the second number of inference models respectively includes: If the second number of the inference models is greater than the first number of the inference units, assigning the same inference unit to a plurality of different inference models; If the second number of the inference models is less than or equal to the first number of the inference units, assigning different inference units to each of the inference models; Furthermore, each of the inference models has its own corresponding task queue, and sending the data to be inferred to the inference board card means sending the data to be inferred to the task queue of the inference model running on the inference board card.

4. The method according to claim 3, characterized in that The determining, according to the total amount of the data to be inferred and the first number of inference units in the inference board that are allowed to be used in parallel, the second number of inference models that are allowed to run in parallel comprises: Determining the number of inferences required for the data to be inferred based on the total amount of the data to be inferred and the maximum amount of data allowed to be input into the inference model at a single time; According to the number of inferences and a first number of inference units in the inference board that are allowed to be used in parallel, a second number of inference models that are allowed to run in parallel is determined.

5. The method according to claim 4, characterized in that The determining, according to the number of inferences and the first number of inference units in the inference board that are allowed to be used in parallel, the second number of inference models that are allowed to run in parallel comprises: Expanding the first quantity by a preset multiple to obtain a target quantity; If the value of the number of inferences is less than the value of the target number, determining that the value of the second number is equal to the value of the number of inferences; If the value of the number of inferences is greater than or equal to the value of the target number, it is determined that the value of the second number is equal to the value of the target number.

6. The method according to claim 5, characterized in that The step of increasing the first quantity by a preset multiple to obtain a target quantity includes: The first quantity is doubled to obtain the target quantity.

7. The method according to claim 4, characterized in that The determining the number of inference times required for the data to be inferred according to the total amount of the data to be inferred and the maximum amount of data allowed to be input at a single time by the inference model includes: Determining a ratio of the total data volume to the maximum data volume; When the ratio obtained is not an integer, the first integer greater than the ratio is used as the value of the number of inferences.

8. The method according to claim 4, characterized in that Before determining the number of inferences required for the data to be inferred, the method further includes: Displaying a first human-computer interaction interface, wherein the first human-computer interaction interface includes a first input area; The number received through the first input area is used as the first number of inference units allowed to be used in parallel in the inference board.

9. The method according to claim 3, characterized in that: The allocating the same inference unit to a plurality of different inference models comprises: According to the corresponding relationship that every two of the reasoning models correspond to the same reasoning unit, the same reasoning unit is allocated to multiple different reasoning models.

10. The method according to claim 3, characterized in that Each of the data subsets has a corresponding sequence number, and the sequence number is used to represent the order in which the data subsets are inferred; The allocating the to-be-reasoned data in different data subsets to each inference model loaded into the inference unit includes: For any target data subset among the multiple data subsets, performing a modulo operation on the sequence number of the target data subset and the second number of the inference model to obtain an operation result i; Assigning the target data subset to the i-th inference model; The value of i is 0 or a positive integer less than the value of the second number.

11. The method according to claim 3, characterized in that The inference model is obtained by converting the trained initial inference model; Before obtaining the data to be inferred, the method further includes: Displaying a second human-computer interaction interface, wherein the second human-computer interaction interface includes a second input area; The initial inference model is converted according to the amount of data allowed to be inputted at one time by the inference model received through the second input area.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the reasoning method according to any one of claims 3 to 11 by executing the computer instructions.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the reasoning method according to any one of claims 3 to 11.

14. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the reasoning method according to any one of claims 3 to 11.

Images