Method and apparatus for executing model inference task, and storage medium and computing device

Abstract

Disclosed are a method and apparatus for executing a model inference task, and a storage medium and a computing device, which relate to the technical field of computers. A processor in a host generates an instruction corresponding to a model inference task, so as to instruct an AI accelerator card to execute preprocessing / post-processing, which originally needs to be executed by the processor in the host, thereby reducing the occupancy of computing resources of the processor in the host, and increasing the utilization rate of computing power resources of the AI accelerator card. The method is applied to a processor, which is connected to an artificial intelligence (AI) accelerator card by means of a bus. The method comprises: acquiring a first task, wherein the first task is used for instructing an AI accelerator card to perform inference on first data by means of a neural network model; generating an instruction corresponding to the first task, wherein the instruction comprises a first instruction and / or a second instruction, the first instruction is used for instructing the AI accelerator card to preprocess the first data, and the second instruction is used for instructing the AI accelerator card to post-process output data of the neural network model; and sending the instruction to the AI accelerator card.

Description

Method, device, storage medium and computing device for executing model inference task

[0001] The present application claims priority to the Chinese patent application No. 202411094549.1, filed on August 9, 2024, and entitled "Method, device, storage medium and computing device for executing model inference task", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of computer, and in particular, to a method, device, storage medium and computing device for executing model inference task. BACKGROUND

[0003] In the technical field of machine learning, a model inference task refers to logical inference and prediction based on a specified data by using a neural network model. For example, the model inference task can be sensitive information detection and quality evaluation on a specified document by using a pre-trained natural language processing (NLP) model.

[0004] Generally, when a computing device executes a model inference task, the following steps are usually included: preprocessing of original data of the inference task, neural network calculation on the preprocessed data, and post-processing of data output by the neural network model. Since the neural network calculation step has a large amount of calculation, in order to accelerate the execution efficiency of the model inference task, the related art usually only assigns this step to a specific component (such as an artificial intelligence (AI) accelerator card) connected to the computing device (host computer) for execution, and the remaining steps are processed by the central processing unit (CPU) of the computing device. However, this execution mode of the model inference task occupies a large amount of computing resources of the host CPU, and even affects the user experience of other applications on the host computer. SUMMARY

[0005] The present application provides a method, device, storage medium and computing device for executing a model inference task. The processor in the computing device (host computer) generates instructions corresponding to inference data (first data) in a first task (model inference task), controls an artificial intelligence (AI) accelerator card to execute data preprocessing and post-processing, and neural network calculation and other computing tasks. That is, the host processor does not need to execute data preprocessing and post-processing, thereby significantly reducing the occupation of computing resources of the host processor, improving the utilization rate of computing resources of the AI accelerator card, and improving the execution efficiency of the model inference task.

[0006] In a first aspect, the present application provides a method for executing a model inference task, applied to a processor connected to an artificial intelligence (AI) accelerator card via a bus. The method comprises: obtaining, by the processor, a first task, the first task being used to instruct the AI accelerator card to perform inference on first data by using a neural network model; generating, by the processor, an instruction corresponding to the first task, the instruction comprising a first instruction and / or a second instruction, the first instruction being used to instruct the AI accelerator card to perform preprocessing on the first data, and the second instruction being used to instruct the AI accelerator card to perform post-processing on output data of the neural network model; and sending, by the processor, the instruction to the AI accelerator card.

[0007] It can be understood that, by generating the instruction corresponding to the first task (the model inference task) by the processor and sending the instruction to the AI accelerator card, the preprocessing and post-processing originally performed by a host processor can be offloaded to the AI accelerator card, thereby significantly reducing the occupation of the computing resources of the host processor and improving the utilization of the computing resources of the AI accelerator card.

[0008] In a possible implementation, the sending, by the processor, of the instruction to the AI accelerator card comprises: sending, by the processor, the first instruction to a first computing unit in the AI accelerator card, the first computing unit being used to perform preprocessing and / or post-processing on data; if the processor receives a preprocessing completion signal sent by the first computing unit, sending, by the processor, a third instruction to a second computing unit in the AI accelerator card, the third instruction being used to instruct the second computing unit to perform first inference on a preprocessing result obtained by the first computing unit after preprocessing the first data by using the neural network model, and the second computing unit being used to perform neural network computation on data; and if the processor receives a signal sent by the second computing unit and used to instruct completion of the first inference, sending, by the processor, the second instruction to the first computing unit.

[0009] It can be understood that, after obtaining the signal sent by the AI accelerator card and used to instruct completion of the first inference, the processor sends the corresponding instruction, which can ensure that the model inference task is performed in sequence and avoid confusion among the preprocessing, the neural network computation, and the post-processing performed on the same data (the first data), thereby ensuring that the model inference task can be performed smoothly.

[0010] In a possible implementation, the first task is further used to instruct the AI accelerator card to perform inference on second data by using the neural network model; and after the sending, by the processor, of the third instruction to the second computing unit in the AI accelerator card, the method further comprises: sending a fourth instruction to the AI accelerator card, the fourth instruction being used to instruct the first computing unit to perform preprocessing on the second data.

[0011] It can be understood that the processor can make the first computing unit and the second computing unit in the AI acceleration card perform calculation in parallel by sending the fourth instruction, that is, when the second computing unit performs inference on the preprocessing result of the first data, the first computing unit preprocesses the second data, which significantly improves the execution efficiency of the model inference task and fully utilizes the computing resources of the AI acceleration card.

[0012] In a possible implementation, after the processor sends the third instruction to the second computing unit in the AI acceleration card, the method further includes: the processor sends a fifth instruction to the AI acceleration card, the fifth instruction being used to instruct the first computing unit to post-process third data, the third data being output data obtained by the second computing unit after performing a second inference on the neural network model, the second inference being performed before the first inference.

[0013] It can be understood that the processor can make the first computing unit and the second computing unit in the AI acceleration card perform calculation in parallel by sending the fifth instruction, that is, when the second computing unit performs inference on the preprocessing result of the first data, the first computing unit post-processes the third data, which significantly improves the execution efficiency of the model inference task and fully utilizes the computing resources of the AI acceleration card.

[0014] In a possible implementation, the size difference between the first data and the second data does not exceed a data size threshold; or, the time difference between the time for the second computing unit to perform the first inference and the time for the first computing unit to preprocess the second data does not exceed a first time threshold.

[0015] It can be understood that by limiting the size difference between the first data and the second data or the time difference between the time for the second computing unit to perform the first inference and the time for the first computing unit to preprocess the second data, the efficiency of parallel calculation can be further improved, because the first computing unit is prevented from performing long-time preprocessing and the second computing unit is prevented from being in a waiting (idle) state for a long time due to a too large size difference or processing time difference between different data.

[0016] In a possible implementation, the size difference between the first data and the third data does not exceed a data size threshold; or, the time difference between the time for the second computing unit to perform the first inference and the time for the first computing unit to post-process the third data does not exceed a second time threshold.

[0017] It can be understood that by limiting the size difference between the first data and the third data, or limiting the difference between the time for the second computing unit to perform the first inference and the time for the first computing unit to post-process the third data, the size difference or processing time difference between different data is avoided, which causes the first computing unit to perform long-time post-inference, and the second computing unit is in a waiting (idle) state for a long time, thereby further improving the efficiency of parallel computing.

[0018] In a possible implementation, if the first task is a natural language processing (NLP) task, the instruction corresponding to the first task is generated, including: generating a batch instruction, the batch instruction being used to instruct the AI acceleration card to process the first data in batches, wherein the first instruction includes the batch instruction.

[0019] It can be understood that before the AI acceleration card performs neural network calculation, the processor generates a batch instruction to instruct the AI acceleration card to process the first data of the NLP task in batches, which avoids inputting too much data into the model at one time, thereby ensuring the smooth execution of the model inference task.

[0020] In a possible implementation, the batch instruction is used to instruct the AI acceleration card to process the first data in batches, including: the batch instruction is used to instruct the AI acceleration card to divide the text data corresponding to the NLP task into one or more batches according to a set length, each batch including one or more sample data, and the sample data including one sentence or paragraph in the text data.

[0021] It can be understood that dividing the text data into different batches avoids inputting too long text directly into the model, thereby ensuring the smooth execution of the model inference task.

[0022] In a possible implementation, the plurality of batches includes a first batch, a maximum value of lengths of the first sample data in the first batch is a first length, the first batch further includes second sample data with a length less than the first length, and the batch instruction is further used to instruct the AI acceleration card to splice the second sample data to obtain first spliced sample data with a length of the first length.

[0023] It can be understood that since the first spliced sample data is obtained by splicing a plurality of second sample data, the difference between the length of the first spliced sample data and the first length is smaller than that of any one of the second sample data, thereby effectively reducing the cost of length alignment in the same batch, improving the effectiveness of calculation, and further improving the inference speed.

[0024] In a second aspect, the present application provides a method for executing a model inference task, applied to an artificial intelligence (AI) accelerator card, the AI accelerator card being connected to a processor via a bus, the method comprising: receiving an instruction sent by the processor; the instruction comprising a first instruction and / or a second instruction, the first instruction being used to instruct the AI accelerator card to pre-process first data corresponding to a first task, the second instruction being used to instruct the AI accelerator card to post-process output data of a neural network model, the first task being used to instruct the AI accelerator card to perform inference on the first data via the neural network model; and generating an inference result of the first task according to the instruction.

[0025] In a third aspect, the present application provides an execution device for a model inference task, the execution device comprising modules for executing the method of any of the implementation manners of the first aspect.

[0026] For example, the execution device comprises an obtaining module, a generating module and a sending module, the obtaining module being used to obtain a first task, the first task being used to instruct an AI accelerator card to perform inference on first data via a neural network model; the generating module being used to generate an instruction corresponding to the first task, the instruction comprising a first instruction and / or a second instruction, the first instruction being used to instruct the AI accelerator card to pre-process the first data, the second instruction being used to instruct the AI accelerator card to post-process output data of the neural network model; and the sending module being used to send the instruction to the AI accelerator card.

[0027] In a fourth aspect, the present application provides an execution device for a model inference task, the execution device comprising modules for executing the method of any of the implementation manners of the second aspect.

[0028] For example, the execution device comprises a receiving module and a generating module, the receiving module being used to receive an instruction sent by a processor; the instruction comprising a first instruction and / or a second instruction, the first instruction being used to instruct an AI accelerator card to pre-process first data corresponding to a first task, the second instruction being used to instruct the AI accelerator card to post-process output data of a neural network model, the first task being used to instruct the AI accelerator card to perform inference on the first data via the neural network model; and the generating module being used to generate an inference result of the first task according to the instruction.

[0029] In a fifth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are run in a computing device, the computing device executes the method of any of the implementation manners of the first aspect and the second aspect.

[0030] In a sixth aspect, the present application provides a computer program product, when the computer program product is run in a computing device, the computing device executes the method of any of the implementation manners of the first aspect and the second aspect.

[0031] The beneficial effects of the third aspect to the sixth aspect above can refer to the description of any one of the implementation manners of the first aspect or the second aspect, and will not be described here. On the basis of the implementation manners provided in the above aspects, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0032] FIG. 1 is an execution flow diagram of a model inference task of a related technology provided by an embodiment of the present application;

[0033] FIG. 2 is a structural diagram of a computing device provided by an embodiment of the present application;

[0034] FIG. 3 is a structural diagram of an AI acceleration card related to the embodiment shown in FIG. 2;

[0035] FIG. 4 is a flowchart of an execution method of a model inference task provided by an embodiment of the present application;

[0036] FIG. 5 is a flowchart of an execution method of a model inference task provided by an embodiment of the present application;

[0037] FIG. 6 is a comparison diagram of an execution method of a model inference task related to the embodiment shown in FIG. 5;

[0038] FIG. 7 is a comparison diagram of an execution flow of a host processor related to the embodiment shown in FIG. 5;

[0039] FIG. 8 is a comparison diagram of a batch processing provided by an embodiment of the present application;

[0040] FIG. 9 is a flowchart of an execution method of a model inference task provided by an embodiment of the present application;

[0041] FIG. 10 is a structural diagram of an execution device of a model inference task provided by an embodiment of the present application;

[0042] FIG. 11 is a structural diagram of an execution device of a model inference task provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0044] “Multiple” mentioned herein refers to two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship.

[0045] Also, in the description of the embodiments of the present application, "a plurality of" means two or more than two, unless otherwise specified. "At least one of the following (one)" or the like means any combination of the items, including a single item (one) or a combination of a plurality of items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be a single item or a plurality of items.

[0046] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, "first", "second" and the like are used to distinguish the same items or similar items with basically the same function and role. The skilled in the art can understand that "first", "second" and the like do not limit the quantity and execution order, and "first", "second" and the like do not necessarily mean different. At the same time, in the embodiments of the present application, "exemplary" or "for example" and the like are used to represent as an example, illustration or explanation. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" and the like is intended to present the relevant concept in a specific manner, for understanding.

[0047] First, the application scenarios of the embodiments of the present application are exemplarily introduced.

[0048] The model inference task refers to using a neural network model to perform logical inference, prediction, etc. according to specified data. For example, the model inference task can be to perform sensitive information detection, quality evaluation, etc. on a specified document by a pre-trained natural language processing (NLP) model; for another example, the model inference task can also be to perform dangerous behavior detection, rule violation detection, etc. on a specified video by a trained behavior detection model. It should be noted that the foregoing model inference task is only an example, and the type of model inference task is not limited by the present application.

[0049] When the computing device performs the model inference task, it usually includes the following steps: 1) pre-processing the data corresponding to the inference task, 2) neural network calculation on the pre-processed data, 3) post-processing of the data output by the neural network model. Specifically:

[0050] 1) Pre-processing the data corresponding to the inference task can include format conversion of the data to meet the format requirements of the input data of the neural network model, and can also include optimization of the data corresponding to the inference task (such as reducing the noise of the data) to improve the execution efficiency of the model inference task.

[0051] 2) performing neural network calculation on the preprocessed data, which can include inputting the preprocessed data into a neural network model, and performing calculation on the preprocessed data through each network layer in the neural network model to generate output data of the neural network model.

[0052] 3) performing post-processing on the data output by the neural network model, which can improve the visual degree of the result (e.g., generating a chart according to the data output by the model), or the post-processing can also be further statistics and analysis on the data output by the neural network model, and the like.

[0053] Generally, performing neural network calculation on the preprocessed data, and the other two steps (preprocessing and post-processing of data) can be considered as two different types of computing tasks, wherein performing neural network calculation on the preprocessed data involves a large number of simple logical calculations, and is a highly parallelizable computing task, that is, neural network calculation is more suitable for being processed by a processor good at parallel computing such as a GPU or a special neural network processing unit (NPU); and the preprocessing and post-processing of data involve relatively small amount of calculation, but the operation logic is relatively complex, and there can be a certain sequence when performing, which is a less parallelizable computing task, that is, the preprocessing and post-processing of data are suitable for being processed by a processor good at processing complex tasks such as a CPU.

[0054] In order to improve the efficiency of the computing device in performing the model inference task, an artificial intelligence (AI) acceleration card good at performing neural network calculation can be accessed on the computing device. In this case, the computing device can be referred to as a host of the AI acceleration card. Since the AI acceleration card often contains a processor such as the GPU, NPU, and the like, in the related art, the neural network calculation in the model inference task is often processed by the AI acceleration card, and the remaining steps in the model inference task (such as preprocessing and post-processing of data) are processed by the host CPU. The execution flow of this way is shown in FIG. 1, which is an execution flow diagram of a model inference task of a related technology provided by an embodiment of the present application.

[0055] As shown in FIG. 1, in the execution mode of such model inference task, each computing task is specifically executed in the following order: first, the host CPU pre-processes the first data (in this process, the AI acceleration card is idle); then, the AI acceleration card performs neural network calculation on the pre-processed data; finally, the host CPU post-processes the output data of the neural network model (in this process, the AI acceleration card is idle). This way not only occupies too much computing resource of the host CPU, but also causes the AI acceleration card to be idle for a long time during the execution of the model inference task, resulting in waste of computing power resources and low utilization, and reducing the execution efficiency of the model inference task.

[0056] Therefore, the embodiments of the present application provide an execution method of a model inference task, which schedules an artificial intelligence (AI) acceleration card to perform data preprocessing and post-processing, neural network calculation and other computing tasks through a host processor, significantly reduces the occupation of computing resources of the host processor, improves the utilization of computing power resources of the AI acceleration card, and can improve the execution efficiency of the model inference task.

[0057] In some possible embodiments, the method is applied to a processor connected with an artificial intelligence (AI) acceleration card through a bus, and the method comprises: the processor acquires a first task, wherein the first task is used to instruct the AI acceleration card to perform inference on first data through a neural network model; the processor generates an instruction corresponding to the first task, wherein the instruction comprises a first instruction and / or a second instruction, the first instruction is used to instruct the AI acceleration card to pre-process the first data, and the second instruction is used to instruct the AI acceleration card to post-process output data of the neural network model; and the processor sends the instruction to the AI acceleration card. Since the processor can offload the computing tasks such as data preprocessing and post-processing, which are originally executed on the host processor, to the AI acceleration card to be executed through the first instruction and / or the second instruction, the occupation of computing resources of the host processor can be significantly reduced, and the utilization of computing power resources of the AI acceleration card can be improved.

[0058] Secondly, the system architecture of the embodiments of the present application is exemplarily introduced.

[0059] As shown in FIG. 2, FIG. 2 is a structural schematic diagram of a computing device according to an exemplary embodiment of the present application. The computing device 1000 at least comprises a memory 1010, a processor 1020, an AI acceleration card 1030 and a bus 1040.

[0060] The processor 1020 can be used to acquire a first task, generate an instruction corresponding to the first data, and send the generated instruction to the AI acceleration card 1030, wherein the first task is used to instruct the AI acceleration card to perform inference on the first data through a neural network model.

[0061] The memory 1010 can be configured to store logic code corresponding to the execution method of the model inference task provided in the embodiments of the present application, or in other words, the memory 1010 can store logic code corresponding to the execution of a certain step by the computing device 1000 described in the following embodiments.

[0062] The AI acceleration card 1030 can be configured to receive the instructions sent by the processor 1020, and generate the inference result of the first task according to the received instructions.

[0063] Specifically, the AI acceleration card 1030 includes at least two computing units, as shown in FIG. 3, which is a structural schematic diagram of an AI acceleration card according to an example embodiment of the present application. The AI acceleration card 1030 includes a first computing unit 1031 and a second computing unit 1032.

[0064] The first computing unit 1031 is configured to pre-process and / or post-process data, and the second computing unit 1032 is configured to perform neural network computation on data. In other words, the first computing unit 1031 can be a central processing unit (CPU) or other processor that is good at processing complex tasks, so the first computing unit 1031 can also be referred to as an AI CPU. The second computing unit 1032 can be a GPU, NPU or other processor that is good at parallel computation, so the second computing unit 1032 can also be referred to as an AI core. Since the AI acceleration card 1030 includes the two different computing units described above, the AI acceleration card 1030 can also be referred to as an AI acceleration card with heterogeneous computing units.

[0065] Optionally, the computing device 1000 can be a terminal device or a network device. The terminal device can include a computer, a mobile phone terminal, a tablet computer, a notebook computer, a desktop computer, an all-in-one machine, an augmented reality (AR) device, a virtual reality (VR) device, a mixed reality (MR) device, an extended reality (ER) device, a personal digital assistant (PDA), an ultra-mobile personal computer (UMPC), etc. The terminal device can also include an edge computing device (e.g., a box with a processing-capable chip) and the like. The network device can include a server and the like. The server can be one physical or logical server, or two or more physical or logical servers sharing different responsibilities and working together to implement the functions of the server. For example, the server can be a blade server, a high-density server, a rack server or a tower server, an AI server, etc.

[0066] Optionally, the memory 1010 can include random access memory (RAM), read-only memory (ROM) and the like, wherein the RAM in the memory 1010 can run the necessary operating system, and the modules such as the acquisition module, the generation module and the sending module for executing the execution method of the model inference task provided in the present application.

[0067] Optionally, the processor 1020 can be a central processing unit (CPU) or other general-purpose processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processing (DSP) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0068] Optionally, the bus 1040 can be a peripheral component interconnect (PCI) bus or a peripheral component interconnect express (PCIe) bus, etc. For the convenience of representation, only one line is used in FIG. 2, but it does not mean that there is only one bus or only one type of bus. The bus 1040 can include a path for transmitting information between various components (e.g., the memory 1010, the processor 1020, the communication interface 1030) of the computing device 1000.

[0069] It should be noted that the application scenarios and system architectures described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It can be known by those skilled in the art that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0070] For the convenience of understanding, the execution method of the model inference task provided in the present application is exemplarily introduced below in combination with the accompanying drawings, and the execution method of the model inference task is applicable to the computing device shown in FIG. 2.

[0071] It should be noted that the processor described in the following embodiments performs a certain step (such as S110 to S130 below) can be understood as: the processor 1020 performs the step, and the AI accelerator described in the following embodiments performs a certain step (such as S140 to S150 below) can be understood as: the AI accelerator 1030 performs the step.

[0072] Figure 4 shows a flowchart of a model inference task execution method provided by an embodiment of the application, which includes the following steps:

[0073] S110, the processor acquires the first task.

[0074] The first task is used to instruct the AI accelerator to infer the first data through the neural network model. For example, the first task can be used to instruct the AI accelerator to perform quality evaluation on the specified document through the trained NLP model.

[0075] In step S110, the processor can actively or passively acquire the first task, for example, the processor can actively acquire the first task from the set task library at regular intervals, or passively acquire the first task sent by the user through the terminal device.

[0076] For example, the processor acquires the first task from the set task library every 10 minutes, wherein the first task acquired by the processor instructs to perform sensitive information detection on the No. 1 document and the No. 2 document under the folder with the path: "C:\Desktop\ABCD\" through the trained NLP model.

[0077] For another example, the user uploads a video through the interactive interface displayed on the terminal device, and selects to perform violation action detection (the first task) on the video, in which case the processor can passively acquire the first task.

[0078] S120, the processor generates instructions corresponding to the first data.

[0079] In this step, the processor can generate at least two different instructions, which are respectively: the first instruction and the second instruction, wherein the first instruction is used to instruct the AI accelerator to pre-process the first data corresponding to the first task, and the second instruction is used to instruct the AI accelerator to post-process the output data of the neural network model.

[0080] In some possible embodiments, the processor can also generate a third instruction, which is used to instruct the AI accelerator to perform first inference on the preprocessing result corresponding to the aforementioned first instruction through the neural network model.

[0081] And, the three instructions generated by the processor correspond to the first task. That is, the first task can also be used to indicate the type of preprocessing performed on the first data, the type of neural network model used when performing inference on the preprocessed first data, and the type of post-processing performed by the AI accelerator on the output data of the neural network model.

[0082] The processor can obtain the aforementioned correspondence by at least the following ways, including but not limited to: in one possible implementation, if the processor obtains the target instruction identifier in the first task, the processor can generate the instructions corresponding to the first task according to the target instruction identifier.

[0083] For example, if the target instruction identifier in the first task is "NLP01", which has been pre-set to indicate that the specified document (first data) is segmented, preprocessed in batches, the preprocessed first data is quality evaluated by a trained NLP model, and the output data of the neural network model (the quality score of each document) is sorted and a corresponding chart is generated, then the processor can retrieve from the set instruction identifier library, and then generate the first instruction, the second instruction and the third instruction matching the target instruction identifier "NLP01" by the processor, wherein the first instruction instructs the AI accelerator to segment and batch process the first data corresponding to the first task, the second instruction instructs the AI accelerator to sort the output data of the neural network model and generate a corresponding chart, and the third instruction instructs the AI accelerator to quality evaluate the preprocessed first data (preprocessing result) by a trained NLP model.

[0084] In step S130, the processor sends the instructions to the AI accelerator.

[0085] In this step, the processor can send the instructions to the AI accelerator through the PCIe bus, where the AI accelerator can be the AI accelerator 1030 shown in FIG. 2.

[0086] Through the above steps S110-S130, the processor can offload the three computing tasks involved in the model inference task to the AI accelerator for execution, thereby releasing a large amount of computing resources on the processor, and improving the utilization of the AI accelerator.

[0087] In step S140, the AI accelerator receives the instructions sent by the processor.

[0088] In this step, the AI accelerator can receive the instructions sent by the processor through the PCIe bus, which can be the first instruction, the second instruction and the third instruction described above. The explanations of the instructions have been explained above and will not be repeated here.

[0089] S150, the AI acceleration card generates the inference result of the first task according to the instruction.

[0090] In this step, the AI acceleration card executes the instruction it receives, specifically, it can execute the operator indicated by the instruction on the corresponding data (for example, execute the preprocessing operator indicated by the first instruction, the post-processing operator indicated by the second instruction, and the neural network calculation operator indicated by the third instruction), where the operator can be stored in the form of software code in the memory 1010 shown in FIG. 2, and the programming language, interface specification, etc. used by the operator are not limited in the present application.

[0091] In some possible embodiments, after the AI acceleration card completes the task indicated by each instruction it receives, it can return a corresponding completion signal to the processor, for example, after the AI acceleration card completes the preprocessing of the first data corresponding to the first task indicated by the first instruction, it returns a preprocessing completion signal to the processor, which can ensure the stability of the interaction process between the processor and the AI acceleration card and the model inference task execution process.

[0092] For more detailed interaction processes between different computing units in the AI acceleration card and the processor, the following embodiments in combination with FIG. 5 and FIG. 6 will be explained and described, which will not be repeated here.

[0093] Through the above steps S110-S150, the processor generates instructions corresponding to the first task (model inference task), controls the AI acceleration card to perform data preprocessing and post-processing, and realizes the offloading of the data preprocessing and post-processing process originally performed by the host processor to the AI acceleration card, significantly reduces the occupation of the computing resources of the processor, and improves the utilization rate of the computing power resources of the AI acceleration card.

[0094] In the embodiments of the present application, the AI acceleration card includes at least two different computing units, which are a first computing unit for preprocessing and / or post-processing data, and a second computing unit for neural network calculation, which can be referred to the AI acceleration card 1030, the first computing unit 1031 and the second computing unit 1032 shown in FIG. 3.

[0095] The interaction process between the first computing unit, the second computing unit in the AI acceleration card and the host processor will be described in detail below in combination with FIG. 5 and FIG. 6, as shown in FIG. 5, which is an interaction flowchart of a model inference task execution method provided by the embodiments of the present application, including:

[0096] S210, the processor sends the first instruction to the first computing unit in the AI acceleration card.

[0097] In this step, the processor can send the first instruction to the first computing unit in the AI acceleration card through the PCIe bus.

[0098] S220, in response to the first computing unit receiving the first instruction sent by the processor, the first computing unit pre-processes the first data to obtain a pre-processing result of the first data.

[0099] For example, in response to the first computing unit receiving the first instruction sent by the processor, the first computing unit performs word segmentation processing and batch processing on the document file (first data) under the specified path to obtain a pre-processing result of the first data. Since the first data contains a large number of characters and the data volume is greater than the upper limit of the data volume allowed to be input into the neural network model at a time, the pre-processing result obtained after pre-processing includes multiple batches of data after batch processing, and the data volume in each batch of data meets the data volume threshold range of inputting the neural network model, thereby ensuring that the neural network calculation can be performed smoothly.

[0100] For another example, in response to the first computing unit receiving the first instruction sent by the processor, the first computing unit performs median filtering processing on the image file under the specified path to obtain a pre-processing result. The median filtering can remove noise (such as salt and pepper noise) in the image data. Therefore, compared with the first data before processing, since the pre-processing result includes image data after median filtering processing, the data quality is optimized, thereby improving the accuracy of neural network calculation.

[0101] S230, the first computing unit sends a pre-processing completion signal to the processor.

[0102] In this step, after the first computing unit completes the pre-processing of the first data indicated by the first instruction received in step S220, the first computing unit can send a pre-processing completion signal to the processor through the PCIe bus. In this way, the processor can timely learn the running state of the first computing unit, which helps the processor to generate new instructions in time, further improves the utilization rate of the first computing unit, and improves the execution efficiency of the model inference task.

[0103] S240, the processor sends a third instruction to the second computing unit in the AI acceleration card if the processor receives the pre-processing completion signal sent by the first computing unit.

[0104] In this step, in the case that the processor receives the preprocessing completion signal sent by the first computing unit, it can be determined that the first computing unit has completed the preprocessing calculation task of the first data, that is, the processor can determine that there is a preprocessing result of the first data to be inferred, and then the processor sends a third instruction for instructing the second computing unit to infer the preprocessing result by the neural network model to the second computing unit. The processor can send the third instruction to the second computing unit in the AI acceleration card through the PCIe bus.

[0105] S250, in response to the second computing unit receiving the third instruction sent by the processor, the second computing unit performs first inference on the preprocessing result of the first data by the neural network model, and obtains a first inference result.

[0106] For example, in response to the second computing unit receiving the third instruction sent by the processor, the second computing unit performs quality evaluation on the preprocessing result of the first data obtained in the above step S220 by the trained NLP model, and outputs the quality score (first inference result) of each batch of data processed in batches in the above step S220. The quality score can be a score in multiple dimensions (such as text fluency, information quantity, etc.).

[0107] S260, the second computing unit sends a signal for indicating the completion of the first inference to the processor.

[0108] In this step, in the case that the processor receives the preprocessing completion signal sent by the first computing unit, it can be determined that the first computing unit has completed the preprocessing calculation task of the first data, that is, the processor can determine that there is a preprocessing result of the first data to be inferred, and then the processor sends a third instruction for instructing the second computing unit to infer the preprocessing result by the neural network model to the second computing unit. The processor can send the third instruction to the second computing unit in the AI acceleration card through the PCIe bus.

[0109] S270, if the first inference completion signal sent by the second computing unit is received, a second instruction is sent to the first computing unit.

[0110] In this step, in the case that the processor receives the preprocessing completion signal sent by the first computing unit, it can be determined that the first computing unit has completed the preprocessing calculation task of the first data, that is, the processor can determine that there is a preprocessing result of the first data to be inferred, and then the processor sends a third instruction for instructing the second computing unit to infer the preprocessing result by the neural network model to the second computing unit. The processor can send the third instruction to the second computing unit in the AI acceleration card through the PCIe bus.

[0111] S280, in response to the first computing unit receiving the second instruction sent by the processor, the first computing unit post-processes the first inference result to obtain a post-processing result.

[0112] For example, in response to the first computing unit receiving the second instruction sent by the processor, the first computing unit post-processes the first inference result obtained in step S250, wherein the post-processing includes calculating a quality score of the document according to the quality scores of each batch of data, and if the quality score is a score in multiple dimensions, the first computing unit can also generate a radar chart of the quality evaluation of the document according to the quality score to more intuitively show the quality evaluation result to the user.

[0113] S290, the first computing unit sends a post-processing completion signal to the processor.

[0114] In this step, after the first computing unit completes the post-processing of the first inference result indicated by the second instruction received in step S280, the first computing unit can send a post-processing completion signal to the processor through the PCIe bus, so that the processor can learn the running state of the first computing unit in time, which helps the processor to generate new instructions in time, further improves the utilization of the first computing unit, and improves the execution efficiency of the model inference task.

[0115] Through the above steps S210-S290, the processor generates the first instruction and the second instruction corresponding to the first data corresponding to the first task (model inference task), offloads the preprocessing and post-processing processes of the data originally processed by the host processor to the first computing unit in the AI acceleration card, reduces the occupation of the computing resources of the processor, and improves the utilization rate of the computing resources of the AI acceleration card. Moreover, the third instruction is used to instruct the second computing unit in the AI acceleration card to perform neural network calculation on the preprocessed data, which ensures the smooth execution of the model inference task.

[0116] In some possible embodiments, the first task can not only be used to instruct the AI acceleration card to perform inference on the first data through the neural network model, but also be used to instruct the AI acceleration card to perform inference on at least one other data (such as second data) through the neural network model. In this case, to further improve the execution efficiency of the model inference task, the processor can send a fourth instruction and / or a fifth instruction to the first computing unit after sending the third instruction to the second computing unit in the AI acceleration card, or in the process of the second computing unit performing first inference on the preprocessed result of the first data to obtain a first inference result.

[0117] The fourth instruction is used to instruct the first computing unit to pre-process the second data, and the fifth instruction is used to instruct the first computing unit to post-process the third data, the third data being output data obtained by the second computing unit after performing a second inference on the neural network model, the second inference being performed before the first inference.

[0118] That is, in the embodiment of the present application, the processor can instruct the first computing unit to simultaneously pre-process and / or post-process other data indicated by the first task during the process in which the second computing unit performs neural network calculation, so that the first computing unit and the second computing unit can perform parallel calculation, and in some feasible embodiments, the pre-processing of data, the neural network calculation, and the post-processing of data can be performed in parallel, thereby significantly improving the execution efficiency of the model inference task.

[0119] In addition, in the embodiment of the present application, a data size threshold or a time threshold can be set to ensure that the tasks can be efficiently connected, and the first computing unit and the second computing unit can be prevented from being idle for a long time, thereby further improving the execution efficiency of the model inference task and improving the resource utilization of the AI acceleration card.

[0120] Specifically, in a possible implementation, the size of the first data and the size of the second data differ by no more than a data size threshold; or, the time for the second computing unit to perform the first inference and the time for the first computing unit to pre-process the second data do not exceed a first time threshold. The time for the second computing unit to perform the first inference can be estimated according to the size of the first data and the hardware parameters of the second computing unit, and the time for the first computing unit to pre-process the second data can be estimated according to the size of the second data and the hardware parameters of the first computing unit, which is not limited in the present application.

[0121] Correspondingly, in a possible implementation, the size of the first data and the size of the third data differ by no more than a data size threshold; or, the time for the second computing unit to perform the first inference and the time for the first computing unit to post-process the third data do not exceed a second time threshold. The second time threshold can be the same as or different from the first time threshold, and can be set according to actual needs. The explanation of this implementation is similar to the foregoing, and will not be repeated here. This implementation can ensure efficient connection between the post-processing of the third data by the first computing unit and the execution of the first inference by the second computing unit, thereby improving the execution efficiency of the model inference task.

[0122] For example, the first task is used to instruct the AI acceleration card to perform quality evaluation on the specified four document data (first document data, second document data, third document data, and fourth document data) by using the trained NLP model. After the processor instructs the second computing unit in the AI acceleration card to perform neural network calculation on the preprocessed first document data through the third instruction, since the second, third, and fourth document data are still to be processed, the processor sends a new first instruction to the first computing unit, and the first instruction is specifically used to instruct the first computing unit to preprocess the second document data. In the case that there is unprocessed data in the output data, the processor sends a second instruction to the first computing unit, and the second instruction is specifically used to instruct the first computing unit to perform post-processing on the unprocessed data in the output data of the neural network model. For details, refer to FIG. 6, which is a comparison diagram of an execution method of a model inference task according to the embodiment shown in FIG. 5. In FIG. 6, the upper part is the execution process of the related art, and the lower part is an example execution process according to the embodiment of the present application.

[0123] According to FIG. 6, it can be known that the preprocessing process of each document data in the related art can only start after the inference process of the previous document is completely finished. However, in the execution method of the model inference task provided by the embodiment of the present application, the neural network calculation process of each document can be parallel with the preprocessing and / or post-processing process of each document. For example, in FIG. 6, when the second computing unit in the AI acceleration card performs neural network calculation on the first document data (i.e., neural network calculation 1 in FIG. 6), the first computing unit in the AI acceleration card can perform preprocessing on the second document data in parallel (i.e., preprocessing 2 in FIG. 6), and then perform preprocessing on the third document data (i.e., preprocessing 3 in FIG. 6). When the second computing unit in the AI acceleration card performs neural network calculation on the second document data (i.e., neural network calculation 2 in FIG. 6), the first computing unit in the AI acceleration card can perform post-processing on the first data obtained by the neural network calculation 1 in parallel (i.e., post-processing 1 in FIG. 6), and then perform preprocessing on the fourth document data (i.e., preprocessing 4 in FIG. 6). The remaining processes are not described herein.

[0124] It can be known that, by controlling the first computing unit and the second computing unit in the AI acceleration card to perform parallel calculation through the host processor, the execution efficiency of the model inference task can be significantly improved. In some experiments, the results are shown in Table 1, which is a performance comparison table. It can be seen that, by using the execution method of the model inference task provided by the embodiment of the present application, the execution efficiency can be significantly improved.

[0125] Table 1

[0126] Moreover, from the perspective of the host processor, as shown in FIG. 7, before offloading, the host processor needs to first obtain the first task, perform preprocessing on the first data corresponding to the first task, then send the preprocessed data (preprocessing result) to the AI accelerator, then obtain the data output by the neural network model, perform post-processing on the data, and finally output the result.

[0127] However, by using the model inference task execution method (after offloading) provided in the embodiments of the present application, after the host obtains the first task, the host only needs to generate corresponding instructions to instruct the AI accelerator to perform preprocessing, neural network calculation, and post-processing, obtain the corresponding result, and finally output the result, which significantly reduces the occupation of the computing resources of the host processor and improves the utilization rate of the AI accelerator.

[0128] In the above embodiments, the processor mainly instructs the AI accelerator to execute the model inference task by generating instructions corresponding to the first task, which can maximize the release of the computing resources of the processor. However, in some feasible embodiments, the processor can also be used to perform preprocessing and / or post-processing on the data, that is, the processor and the first computing unit in the AI accelerator can each assume part of the preprocessing and post-processing computing tasks, and the specific proportion of sharing can be set according to actual needs, which is not limited in the present application.

[0129] In the embodiments of the present application, in the case where the preprocessing includes batch processing of the first data (text data), specifically, the batch processing is used to divide the first data into multiple batches, and align the lengths of the texts (such as sentences or paragraphs) included in each batch, thereby improving the execution efficiency of the model inference task. Referring to Table 2, Table 2 shows the inference time of neural network calculation in a typical NLP model inference task (sensitive information identification) when using different batch numbers and single batch lengths under the condition that the total length of the text is the same. As can be seen from Table 2, the longer the single batch length, the shorter the inference time.

[0130] Table 2

[0131] Based on the above conclusion that the longer the single batch length, the shorter the inference time, when batch processing, the single batch length is often set to the maximum single batch length that the model can process. However, since a larger single batch length contains more texts, while shortening the inference time, it also increases the possibility that the lengths of the texts in a single batch are greatly different.

[0132] In the related art, batch processing usually adopts a uniform batching method, that is, zero padding alignment is performed according to the longest text in the current batch. In the case of a large difference in the length of the text, the uniform batching method will cause a large number of zeros without actual meaning in the batched data. Since these zeros also participate in neural network calculation, a large number of invalid calculations occur in the neural network calculation step, thereby reducing the execution efficiency of the model inference task.

[0133] In the embodiments of the present application, in order to reduce invalid calculations and improve the execution efficiency of the model inference task, a possible implementation manner is provided, which can be referred to as compact batching. Specifically, the implementation manner includes: dividing text data corresponding to a natural language processing (NLP) task into one or more batches according to a set length, each batch including one or more first sample data, and the unit sample data including one sentence or paragraph in the text data. Further, the plurality of batches includes a first batch, the maximum value of the length of each first sample data in the first batch is a first length, and the first batch also includes second sample data with a length less than the first length. The second sample data is spliced to obtain first spliced sample data with a length of the first length.

[0134] Further, if the length of the spliced sample data cannot reach the first length by only splicing the second sample data, a set character can be filled into the spliced sample data to achieve the length of the spliced sample data being the first length, thereby obtaining the first spliced sample data.

[0135] That is, the compact batching takes the maximum value of the length of each first sample data in the first batch as the first length, unlike the aforementioned uniform batching method of directly filling a set character into other sample data in the first batch. Instead, the sample data (second sample data) with a length less than the first length in the batch is spliced, for example, several short sentences are spliced into a long sentence, and then the set character is filled as needed to obtain first spliced sample data with a length of the first length. In this way, the number of set characters (such as 0) that need to be filled is reduced, thereby reducing invalid calculations and improving the execution efficiency of the model inference task.

[0136] For example, as shown in FIG. 8, which is a comparison diagram of batch processing provided by the embodiments of the present application, the text data "The propagation speed of light is about 300 million kilometers per second / It is impossible to reach the speed of light / Really / Speed of light is also related to medium" is batch processed. The text data includes four first sample data, namely "The propagation speed of light is about 300 million kilometers per second", "It is impossible to reach the speed of light", "Really", and "Speed of light is also related to medium".

[0137] If the unified batch processing in the related art is used, the result is shown in the lower left part of FIG. 8, in which 24 setting characters "0" are filled. If the compact batch processing provided by the embodiment of the present application is used, the two short sentences "it is impossible to reach the speed of light" and "really?" are combined into one long sentence "it is impossible to reach the speed of light really?", and the result is shown in the lower right part of FIG. 8, in which 9 setting characters "0" are filled. The number of setting characters is significantly reduced, and the invalid calculation is reduced, and the execution efficiency of the model inference task is improved.

[0138] Next, the execution method of the model inference task and the compact batch method are described again with the sensitive information detection of the text file specified by the data management software as an example, in combination with FIG. 9. As shown in FIG. 9, which is an execution flowchart of a model inference task provided by an embodiment of the present application, the execution method of the model inference task and the compact batch method specifically include the following steps:

[0139] In S310, the computing device loads the neural network model for detection.

[0140] In this step, the computing device can be specifically the computing device 1000 in the foregoing FIG. 2. The computing device loads the neural network model corresponding to the data management software to the AI acceleration card.

[0141] For example, if the data management software is only used for sensitive information detection, the data management software can correspond to only one neural network model. When the software is started, the computing device loads the neural network model corresponding to the software to the AI acceleration card.

[0142] For another example, if the data management software has multiple data management functions, when the user determines the model inference task to be executed through the interactive interface of the software, the computing device loads the neural network model corresponding to the model inference task of the software to the AI acceleration card.

[0143] In S320, the computing device reads the text data.

[0144] For example, the user inputs the file path of the text file that needs to be detected for sensitive information through the interactive interface of the data management software. The computing device reads the corresponding text data into the memory of the computing device according to the file path, and the data in the memory can be obtained by the AI acceleration card.

[0145] In some possible embodiments, if the file corresponding to the file path input by the user through the interaction interface of the data management software is not a text file, such as an audio file, the computing device can extract text data from the file by using a pre-trained text extraction model, for example, convert the audio file into text data by using the pre-trained text extraction model, and then read the text data into the memory of the computing device, and the data in the memory can be acquired by the AI accelerator.

[0146] S330, the computing device instructs the AI accelerator to perform the model inference task.

[0147] In this step, the computing device generates instructions corresponding to sensitive information detection, such as the first instruction, the second instruction, and the third instruction described above. According to actual needs, the computing device can perform preprocessing including compact batching on the text data, neural network calculation, and corresponding post-processing.

[0148] S340, the computing device outputs the inference result.

[0149] In this step, specifically, the processor of the computing device acquires the corresponding inference result from the AI accelerator and outputs it to the terminal device (such as a display) of the computing device for display to the user.

[0150] Through the above steps S310-S340, the computing device can instruct the AI accelerator to complete the model inference task of sensitive information detection on the text data through the processor, significantly reducing the occupation of the computing resource of the processor, and improving the utilization rate of the computing resource of the AI accelerator.

[0151] The above mainly describes the scheme of the embodiments of the present application from the perspective of the method. It can be understood that the execution device of the model inference task contains at least one of the hardware structure and the software module corresponding to the execution of each function in order to realize the above functions. Those skilled in the art should easily realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present text can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application and design constraints of the technical scheme. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0152] The embodiments of the present application can divide the functional units of the model inference task execution apparatus according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be realized in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, there can be another division manner.

[0153] For example, FIG. 10 shows a structural schematic diagram of a model inference task execution apparatus 1100 provided by an example embodiment of the present application. The model inference task execution apparatus 1100 is applied to a computing device, or the model inference task execution apparatus 1100 can be a computing device. The model inference task execution apparatus 1100 includes:

[0154] The obtaining module 1110 is configured to obtain a first task, where the first task is used to instruct the AI acceleration card to perform inference on first data by using a neural network model.

[0155] The generating module 1120 is configured to generate an instruction corresponding to the first task, where the instruction includes a first instruction and / or a second instruction, the first instruction is used to instruct the AI acceleration card to perform preprocessing on the first data, and the second instruction is used to instruct the AI acceleration card to perform post-processing on output data of the neural network model.

[0156] The sending module 1130 is configured to send the instruction to the AI acceleration card.

[0157] For example, in combination with FIG. 4, the obtaining module 1110 can be configured to perform S110 shown in FIG. 4, the generating module 1120 can be configured to perform S120 shown in FIG. 4, and the sending module 1130 can be configured to perform S130 shown in FIG. 4.

[0158] In a possible implementation, the sending module 1130 is further configured to send the first instruction to a first computing unit in the AI acceleration card, the first computing unit is configured to perform preprocessing and / or post-processing on data, if a preprocessing completion signal sent by the first computing unit is received, a third instruction is sent to a second computing unit in the AI acceleration card, the third instruction is used to instruct the second computing unit to perform first inference on a preprocessing result obtained by preprocessing the first data by using the first computing unit by using the neural network model, the second computing unit is configured to perform neural network calculation on data, and if a signal indicating completion of the first inference sent by the second computing unit is received, the second instruction is sent to the first computing unit.

[0159] In a possible implementation, the first task is further used to instruct the AI acceleration card to perform inference on second data by a neural network model; after the third instruction is sent to the second computing unit in the AI acceleration card, the sending module 1130 is further used to send a fourth instruction to the AI acceleration card, where the fourth instruction is used to instruct the first computing unit to perform preprocessing on the second data.

[0160] In a possible implementation, after the third instruction is sent to the second computing unit in the AI acceleration card, the sending module 1130 is further used to send a fifth instruction to the AI acceleration card, where the fifth instruction is used to instruct the first computing unit to perform post-processing on third data, the third data being output data obtained after the second computing unit performs second inference by the neural network model, and the second inference being performed before the first inference.

[0161] In a possible implementation, a size difference between the first data and the second data does not exceed a data size threshold; or, a time difference between a time when the second computing unit performs the first inference and a time when the first computing unit performs preprocessing on the second data does not exceed a first time threshold.

[0162] In a possible implementation, a size difference between the first data and the third data does not exceed a data size threshold; or, a time difference between a time when the second computing unit performs the first inference and a time when the first computing unit performs post-processing on the third data does not exceed a second time threshold.

[0163] In a possible implementation, if the first task is a natural language processing (NLP) task, the generating module 1120 is further used to generate a batching instruction, where the batching instruction is used to instruct the AI acceleration card to perform batching processing on the first data, and the first instruction includes the batching instruction.

[0164] In a possible implementation, the batching instruction is used to instruct the AI acceleration card to perform batching processing on the first data, including: the batching instruction is used to instruct the AI acceleration card to divide text data corresponding to the NLP task into one or more batches according to a set length, and each batch includes one or more first sample data, and the first sample data includes one sentence or one paragraph in the text data.

[0165] In a possible implementation, the plurality of batches includes a first batch, a maximum value of lengths of the first sample data in the first batch is a first length, and the first batch also includes second sample data with a length less than the first length, and the batching instruction is further used to instruct the AI acceleration card to splice the second sample data to obtain first spliced sample data with the first length.

[0166] For example, FIG. 11 shows a structural schematic diagram of an execution apparatus 1200 of a model inference task according to an example embodiment of the present application. The execution apparatus 1200 of the model inference task is applied to an AI acceleration card, or the execution apparatus 1200 of the model inference task can be an AI acceleration card. The execution apparatus 1200 of the model inference task includes:

[0167] A receiving module 1210 is configured to receive an instruction sent by the processor, and the instruction includes a first instruction and / or a second instruction. The first instruction is used to instruct the AI acceleration card to pre-process first data corresponding to a first task, and the second instruction is used to instruct the AI acceleration card to post-process output data of the neural network model. The first task is used to instruct the AI acceleration card to perform inference on the first data by using the neural network model.

[0168] A generating module 1220 is configured to generate an inference result of the first task according to the instruction.

[0169] In a possible implementation, the AI acceleration card includes a first computing unit and a second computing unit, the first computing unit and the second computing unit have different computing architectures, and the receiving module 1210 is configured to:

[0170] In response to that the first computing unit receives the first instruction sent by the processor, the first computing unit pre-processes the first data to obtain a pre-processing result of the first data.

[0171] The receiving module 1210 is further configured to send a pre-processing completion signal to the processor.

[0172] In response to that the second computing unit receives a third instruction sent by the processor, the second computing unit performs first inference on the pre-processing result of the first data by using the neural network model to obtain second data. The third instruction is used to instruct the second computing unit to perform first inference on the pre-processing result of the first data by using the neural network model.

[0173] The receiving module 1210 is further configured to send a signal used to instruct completion of the first inference to the processor.

[0174] In response to the first computing unit receiving the second instruction sent by the processor, the first computing unit post-processes the second data to obtain a post-processing result of the first inference result.

[0175] The processor sends a post-processing completion signal.

[0176] In a possible implementation, the first task is further used to instruct the AI acceleration card to perform inference on second data by a neural network model, and in the process of the second computing unit performing first inference on the pre-processing result of the first data by the neural network model to obtain a first inference result in response to the second computing unit receiving a third instruction sent by the processor, the receiving module 1210 is further used to: in response to the first computing unit receiving a fourth instruction sent by the processor, the first computing unit pre-processes the second data, and the fourth instruction is used to instruct the first computing unit to pre-process the second data.

[0177] In a possible implementation, in the process of the second computing unit performing first inference on the pre-processing result of the first data by the neural network model to obtain a first inference result in response to the second computing unit receiving a third instruction sent by the processor, the receiving module 1210 is further used to: in response to the first computing unit receiving a fifth instruction sent by the processor, the first computing unit post-processes third data, and the fifth instruction is used to instruct the first computing unit to post-process the third data, the third data being output data obtained by the second computing unit after performing second inference by the neural network model, the second inference being before the first inference.

[0178] In a possible implementation, a size difference between the first data and the second data does not exceed a data size threshold; or, a time for the second computing unit to perform the first inference and a time for the first computing unit to pre-process the second data do not exceed a first time threshold.

[0179] In a possible implementation, a size difference between the first data and the second data does not exceed a data size threshold; or, a time for the second computing unit to perform the first inference and a time for the first computing unit to pre-process the second data do not exceed a second time threshold.

[0180] In a possible implementation, if the first task is a natural language processing (NLP) task, the receiving module 1210 is used to:

[0181] In response to receiving the batch instruction sent by the processor, the first data is processed in batches.

[0182] In a possible implementation, in response to receiving the batch instruction sent by the processor, the receiving module 1210 is configured to:

[0183] The text data corresponding to the NLP task is divided into one or more batches according to a set length, and each batch includes one or more first sample data, and the first sample data includes a sentence or a paragraph in the text data.

[0184] In a possible implementation, the plurality of batches includes a first batch, a maximum value of lengths of the first sample data in the first batch is a first length, and the first batch also includes second sample data with a length less than the first length, and the second sample data is spliced to obtain first spliced sample data with a length of the first length.

[0185] As an example, in combination with FIG. 2, the functions of the acquiring module 1110, the generating module 1120, and the sending module 1130 in the model inference task execution apparatus 1100 and the functions of the receiving module 1210 and part or all of the generating module 1220 in the model inference task execution apparatus 1200 can be executed by the computing device 1000 in FIG. 2. Specifically, the acquiring module 1110, the generating module 1120, and the sending module 1130 can be executed cooperatively by the processor 1020 in FIG. 2, and the receiving module 1210 and the generating module 1220 can be executed by the AI accelerator 1030 in FIG. 2. In this application, the apparatuses shown in FIGS. 10 and 11 are only examples provided by the present embodiment, and the model inference task execution apparatus 1100 or 1200 can include more or fewer units according to the execution process of the model inference task, which is not limited in the present application.

[0186] As a feasible example, the model inference task execution apparatus 1100 or 1200 provided by the present application is implemented by a software module, for example, the software module can be provided to users by a cloud service subscription mode, and users can select different subscription levels according to needs; for another example, the software module can also provide enterprise-level customized services with professional domain customization, interface personalization, and expansion functions according to the needs of users or enterprises.

[0187] In addition, the model inference task execution apparatus 1100 or 1200 provided by the present application can also be made into value-added services provided to users, and the present application does not limit this. When the model inference task execution apparatus 1100 or 1200 is implemented by a software module, the model inference task execution apparatus 1100 or 1200 can also be embedded into other model inference task execution software or data detection tool chain systems.

[0188] In an exemplary embodiment, a computer readable storage medium is also provided for storing at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement all or part of the steps of the above-mentioned model inference task execution method. For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0189] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computing device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computing device execute all or part of the steps of the method shown in any of the embodiments of FIG. 4.

[0190] In some embodiments, the method shown in the embodiments of the present application can be implemented as computer program instructions encoded in a machine-readable format on a computer readable storage medium or on other non-transitory media or articles.

[0191] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above.

[0192] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the modules or units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0193] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or a plurality of physical units, that is, can be located in one place, or can be distributed to a plurality of different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0194] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0195] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage program codes.

[0196] The above is only an optional embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for executing a model inference task, the method comprising: The method is applied to a processor connected with an artificial intelligence (AI) acceleration card through a bus, and comprises the following steps: obtaining a first task, the first task being used to instruct the AI acceleration card to perform inference on first data through a neural network model; generating instructions corresponding to the first data, the instructions comprising a first instruction and / or a second instruction, the first instruction being used to instruct the AI acceleration card to perform preprocessing on the first data, and the second instruction being used to instruct the AI acceleration card to perform postprocessing on output data of the neural network model; sending the instructions to the AI acceleration card.

2. The method of claim 1, wherein, The step of sending the instructions to the AI acceleration card comprises: sending the first instruction to a first calculation unit in the AI acceleration card, the first calculation unit being used to perform preprocessing and / or postprocessing on data; if a preprocessing completion signal sent by the first calculation unit is received, sending a third instruction to a second calculation unit in the AI acceleration card, the third instruction being used to instruct the second calculation unit to perform first inference on a preprocessing result obtained by the first calculation unit after preprocessing on the first data through the neural network model, the second calculation unit being used to perform neural network calculation on data; if a signal indicating completion of the first inference sent by the second calculation unit is received, sending the second instruction to the first calculation unit.

3. The method of claim 2, wherein, The first task is also used to instruct the AI acceleration card to perform inference on second data through the neural network model; after the step of sending the third instruction to the second calculation unit in the AI acceleration card, the method further comprises: sending a fourth instruction to the AI acceleration card, the fourth instruction being used to instruct the first calculation unit to perform preprocessing on the second data.

4. The method according to claim 2 or 3, characterized in that, After the step of sending the third instruction to the second calculation unit in the AI acceleration card, the method further comprises: sending a fifth instruction to the AI acceleration card, the fifth instruction being used to instruct the first calculation unit to perform postprocessing on third data, the third data being output data obtained by the second calculation unit after performing second inference through the neural network model, the second inference being performed before the first inference.

5. The method of claim 3, wherein, The first data and the second data differ by no more than a data size threshold; or, a time for the second calculation unit to perform the first inference and a time for the first calculation unit to perform preprocessing on the second data do not exceed a first time threshold.

6. The method of claim 4, wherein, The first data and the third data differ by no more than a data size threshold; or, a time for the second calculation unit to perform the first inference and a time for the first calculation unit to perform postprocessing on the third data do not exceed a second time threshold.

7. The method according to any one of claims 1 to 6, characterized in that, If the first task is a natural language processing (NLP) task, the step of generating the instructions corresponding to the first data comprises: generating a batching instruction, the batching instruction being used to instruct the AI acceleration card to perform batching processing on the first data, wherein the first instruction comprises the batching instruction.

8. The method of claim 7, wherein, The step of instructing the AI acceleration card to perform batching processing on the first data comprises: The batch instruction is used to instruct the AI acceleration card to divide text data corresponding to the NLP task into one or more batches according to a set length, and each batch includes one or more first sample data, and the first sample data includes a sentence or a paragraph in the text data.

9. The method of claim 8, wherein, Comprise: The plurality of batches includes a first batch, the maximum value of the length of the first sample data in the first batch is a first length, and the first batch also includes second sample data with a length less than the first length, and the batch instruction is also used to instruct the AI acceleration card to splice the second sample data to obtain first spliced sample data with a length of the first length.

10. A method for performing a model inference task, the method comprising: Applied to an artificial intelligence (AI) acceleration card, the AI acceleration card is connected with a processor through a bus, and the method comprises: Receiving an instruction sent by the processor; the instruction includes a first instruction and / or a second instruction, the first instruction is used to instruct the AI acceleration card to pre-process first data corresponding to a first task, and the second instruction is used to instruct the AI acceleration card to post-process output data of the neural network model, and the first task is used to instruct the AI acceleration card to infer the first data through the neural network model; According to the instruction, an inference result of the first task is generated.

11. The method of claim 10, wherein, The AI acceleration card includes a first computing unit and a second computing unit, the first computing unit and the second computing unit have different computing architectures, and the receiving of the instruction sent by the processor comprises: In response to the first computing unit receiving the first instruction sent by the processor, the first computing unit pre-processes the first data to obtain a pre-processing result of the first data; Sending a pre-processing completion signal to the processor; In response to the second computing unit receiving a third instruction sent by the processor, the second computing unit performs first inference on the pre-processing result of the first data through the neural network model to obtain a first inference result, and the third instruction is used to instruct the second computing unit to perform first inference on the pre-processing result of the first data through the neural network model; Sending a signal to the processor for indicating completion of first inference; In response to the first computing unit receiving the second instruction sent by the processor, the first computing unit post-processes the first inference result to obtain a post-processing result of the first inference result; Sending a post-processing completion signal to the processor.

12. The method of claim 11, wherein, The first task is also used to instruct the AI acceleration card to infer second data through the neural network model, and in the process of the second computing unit performing first inference on the pre-processing result of the first data through the neural network model to obtain a first inference result in response to the second computing unit receiving a third instruction sent by the processor, the method further comprises: In response to the first computing unit receiving the fourth instruction sent by the processor, the first computing unit pre-processes the second data, and the fourth instruction is used to instruct the first computing unit to pre-process the second data.

13. The method according to claim 11 or 12, characterized in that, In the process of the second computing unit performing the first inference on the pre-processed result of the first data through the neural network model to obtain a first inference result in response to the second computing unit receiving the third instruction sent by the processor, the method further includes: In response to the first computing unit receiving the fifth instruction sent by the processor, the first computing unit post-processes third data, and the fifth instruction is used to instruct the first computing unit to post-process the third data, and the third data is output data obtained after the second computing unit performs a second inference through the neural network model, and the second inference is before the first inference.

14. The method of claim 12, wherein, The size of the first data and the second data differs by no more than a data size threshold; or the time for the second computing unit to perform the first inference and the time for the first computing unit to pre-process the second data do not exceed a first time threshold.

15. The method of claim 13, wherein, The size of the first data and the second data differs by no more than a data size threshold; or the time for the second computing unit to perform the first inference and the time for the first computing unit to pre-process the second data do not exceed a second time threshold.

16. The method according to any one of claims 10-15, characterized in that, If the first task is a natural language processing (NLP) task, the receiving the instruction sent by the processor includes: In response to receiving the batch instruction sent by the processor, the first data is processed in batches.

17. The method of claim 16, wherein, The response to receiving the batch instruction sent by the processor to process the first data in batches includes: Dividing text data corresponding to the NLP task into one or more batches according to a set length, each batch including one or more first sample data, and the first sample data including a sentence or a paragraph in the text data.

18. The method of claim 17, wherein, The plurality of batches includes a first batch, the maximum value of the length of the first sample data in the first batch is a first length, and the first batch also includes second sample data with a length less than the first length, and the second sample data is spliced to obtain first spliced sample data with a length of the first length. The device includes:

19. An apparatus for performing a model inference task, the apparatus comprising: An acquisition module is configured to acquire a first task, and the first task is used to instruct an artificial intelligence (AI) accelerator to perform inference on first data through a neural network model. A generation module is configured to generate an instruction corresponding to the first data, and the instruction includes a first instruction and / or a second instruction, the first instruction is used to instruct the AI accelerator to pre-process the first data, and the second instruction is used to instruct the AI accelerator to post-process output data of the neural network model. A sending module is configured to send the instruction to the AI accelerator. The sending module is configured to send the instruction to the AI accelerator.

20. The execution device of claim 19, wherein, ​ sending the first instruction to a first computing unit in the AI acceleration card, the first computing unit being configured to perform pre-processing and / or post-processing on data; if a pre-processing completion signal sent by the first computing unit is received, sending a third instruction to a second computing unit in the AI acceleration card, the third instruction being configured to instruct the second computing unit to perform first inference on a pre-processing result obtained by the first computing unit after pre-processing the first data by using the neural network model, the second computing unit being configured to perform neural network calculation on data; if a signal indicating completion of the first inference sent by the second computing unit is received, sending the second instruction to the first computing unit.

21. The execution device of claim 20, wherein, The first task is also configured to instruct the AI acceleration card to perform inference on second data by using a neural network model; after the third instruction is sent to the second computing unit in the AI acceleration card, the sending module is further configured to: send a fourth instruction to the AI acceleration card, the fourth instruction being configured to instruct the first computing unit to pre-process the second data.

22. The execution device according to claim 20 or 21, characterized in that, after the third instruction is sent to the second computing unit in the AI acceleration card, the sending module is further configured to: send a fifth instruction to the AI acceleration card, the fifth instruction being configured to instruct the first computing unit to post-process third data, the third data being output data obtained by the second computing unit after performing second inference by using the neural network model, the second inference being performed before the first inference.

23. The execution device of claim 21, wherein, The size of the first data and the second data differs by no more than a data size threshold; or, the time for the second computing unit to perform the first inference and the time for the first computing unit to pre-process the second data differ by no more than a first time threshold.

24. The execution device of claim 22, wherein, The size of the first data and the third data differs by no more than a data size threshold; or, the time for the second computing unit to perform the first inference and the time for the first computing unit to post-process the third data differ by no more than a second time threshold.

25. The execution device according to any of claims 19-24, characterized by If the first task is a natural language processing (NLP) task, the generation module is configured to: generate a batch instruction, the batch instruction being configured to instruct the AI acceleration card to process the first data in batches, wherein the first instruction includes the batch instruction.

26. The execution device of claim 25, wherein, The batch instruction is configured to instruct the AI acceleration card to process the first data in batches, including: The batch instruction is configured to instruct the AI acceleration card to divide text data corresponding to the NLP task into one or more batches according to a set length, each of the batches including one or more first sample data, the first sample data including a sentence or a paragraph in the text data.

27. The execution device of claim 26, wherein, including: The plurality of batches includes a first batch, a maximum value of lengths of the respective first sample data in the first batch is a first length, and second sample data with a length less than the first length is also included in the first batch, and the batching instruction is further used to instruct the AI acceleration card to splice the second sample data to obtain first spliced sample data with the first length.

28. An apparatus for performing a model inference task, the apparatus comprising: The device comprises: The receiving module is configured to receive the instruction sent by the processor, wherein the instruction comprises a first instruction and / or a second instruction, the first instruction is used to instruct the AI acceleration card to pre-process first data corresponding to a first task, and the second instruction is used to instruct the AI acceleration card to post-process output data of the neural network model, and the first task is used to instruct the AI acceleration card to perform inference on the first data by using the neural network model. The generating module is configured to generate an inference result of the first task according to the instruction.

29. The execution device of claim 28, wherein, The AI acceleration card comprises a first computing unit and a second computing unit, the first computing unit and the second computing unit have different computing architectures, and the receiving module is configured to: In response to the first computing unit receiving the first instruction sent by the processor, the first computing unit pre-processes the first data to obtain a pre-processing result of the first data; send a pre-processing completion signal to the processor; In response to the second computing unit receiving a third instruction sent by the processor, the second computing unit performs first inference on the pre-processing result of the first data by using the neural network model to obtain a first inference result, and the third instruction is used to instruct the second computing unit to perform first inference on the pre-processing result of the first data by using the neural network model. send a signal indicating that the first inference is completed to the processor; In response to the first computing unit receiving the second instruction sent by the processor, the first computing unit post-processes the second data to obtain a post-processing result of the first inference result; send a post-processing completion signal to the processor.

30. The execution device of claim 29, wherein, The first task is also used to instruct the AI acceleration card to perform inference on second data by using the neural network model, and in the process of the second computing unit performing first inference on the pre-processing result of the first data by using the neural network model to obtain a first inference result in response to the second computing unit receiving a third instruction sent by the processor, the receiving module is further configured to: In response to the first computing unit receiving a fourth instruction sent by the processor, the first computing unit pre-processes the second data, and the fourth instruction is used to instruct the first computing unit to pre-process the second data.

31. The execution device according to claim 29 or 30, characterized in that, In the process of the second computing unit performing first inference on the pre-processing result of the first data by using the neural network model to obtain a first inference result in response to the second computing unit receiving a third instruction sent by the processor, the receiving module is further configured to: In response to the first computing unit receiving the fifth instruction sent by the processor, the first computing unit post-processes third data, the fifth instruction being used to instruct the first computing unit to post-process the third data, the third data being output data obtained by the second computing unit after performing a second inference on the neural network model, the second inference being performed before the first inference.

32. The execution device of claim 30, wherein, The first data and the second data differ in size by no more than a data size threshold; or, the second computing unit takes no more than a first time threshold to perform the first inference and the first computing unit to pre-process the second data.

33. The execution device of claim 31, wherein, The first data and the second data differ in size by no more than a data size threshold; or, the second computing unit takes no more than a second time threshold to perform the first inference and the first computing unit to pre-process the second data.

34. The execution device according to any of claims 28-33, characterized by If the first task is a natural language processing (NLP) task, the receiving the instruction sent by the processor, the receiving module is used to: In response to receiving the batching instruction sent by the processor, the first data is batch-processed.

35. The execution device of claim 34, wherein, The receiving module is used to: The text data corresponding to the NLP task is divided into one or more batches according to a set length, each batch including one or more first sample data, the first sample data including a sentence or a paragraph in the text data.

36. The execution device of claim 35, wherein, The plurality of batches includes a first batch, the maximum value of the lengths of the first sample data in the first batch being a first length, the first batch further including second sample data with a length less than the first length, the batching instruction further being used to instruct the AI accelerator to splice the second sample data to obtain first spliced sample data with a length of the first length.

37. A computing device, comprising: The computing device includes a processor and a memory; the processor is coupled with the memory; the memory is used to store computer instructions, the computer instructions being loaded and executed by the processor to enable the computing device to implement the execution method of the model inference task according to any one of claims 1 to 9.

38. A computer-readable storage medium, characterized in that, The computer readable storage medium includes computer instructions; when the computer instructions run in the computing device, the computing device executes the execution method of the model inference task according to any one of claims 1 to 9.

39. A computer program product, characterised in that, When the computer program product runs in the computing device, the computing device executes the execution method of the model inference task according to any one of claims 1 to 9.

Images