Deep learning-based model inference method, apparatus, device, and storage medium
By constructing a stage pool and configuring model parameters, assembling the inference process and performing asynchronous data transmission, the problem of fixed model inference processes and strong coupling in existing technologies is solved, achieving a flexible inference system and reducing integration costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JINGDONG TECH HLDG CO LTD
- Filing Date
- 2022-09-02
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, the model inference process is fixed, which leads to high integration costs of application programming interfaces for high-performance runtime on multiple hardware devices, and strong coupling between multiple stages, resulting in a lack of flexibility.
By acquiring a stage pool for building inference task sets, configuring model parameters according to actual needs, determining assembly stages, assembling them into an inference process, encapsulating multiple inference tasks, achieving asynchronous data transmission and batch processing, and reducing synchronous waiting time.
It improves the flexibility of the inference system, reduces the integration cost of the application programming interface, and enhances hardware resource utilization and user experience.
Smart Images
Figure CN115358403B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of deep learning technology, and in particular to a deep learning-based model inference method, apparatus, device, and storage medium. Background Technology
[0002] Models typically require a large amount of data for training, and their parameters are continuously optimized through backpropagation to enable the model to acquire certain capabilities. Training involves learning a certain capability from existing data, while inference simplifies and uses that capability, allowing it to quickly and efficiently operate on unknown data to obtain the expected results.
[0003] In related technologies, the coupling between multiple model inference stages is strong, and the model inference process is usually a fixed inference process, which cannot achieve flexible configuration between multiple stages. As a result, the integration cost of application programming interfaces is high when multiple hardware components are running at high performance. Summary of the Invention
[0004] This disclosure provides a deep learning-based model inference method, apparatus, device, and storage medium, which enables the arrangement and combination of multiple inference tasks according to actual scenario requirements, for inference processing based on the inference model. This not only greatly improves the flexibility of the inference system, but also reduces the integration cost of the application programming interface.
[0005] Firstly, this disclosure provides a deep learning-based model inference method, including:
[0006] Obtain the stage pool for constructing the inference task set;
[0007] Configure the model parameters required to construct the inference task set according to actual needs to obtain a configuration file, and determine the assembly stage according to the configuration file and multiple initial stages in the stage pool, and assemble the inference process according to the assembly stage.
[0008] Each assembly stage of the inference process is encapsulated to obtain multiple inference tasks corresponding to each assembly stage, and a set of inference tasks is constructed based on the multiple inference tasks.
[0009] The raw data to be processed is input into the inference task set for inference processing to obtain inference data.
[0010] Preferably, according to the deep learning-based model inference method provided in this disclosure, the step of determining an assembly stage based on the configuration file and multiple initial stages in the stage pool, and assembling an inference process according to the assembly stage, includes:
[0011] Based on the configuration file, multiple assembly stages are selected from multiple initial stages in the stage pool; and
[0012] The reasoning order of the multiple assembly stages is determined according to the configuration file;
[0013] The multiple assembly stages are assembled into the reasoning flow according to the reasoning sequence.
[0014] Preferably, according to the deep learning-based model inference method provided in this disclosure, the step of inputting the raw data to be processed into the inference task set for inference processing to obtain inference data includes:
[0015] The original data input to the inference task set is preprocessed to obtain batch data;
[0016] The batch data is input into the inference model for model inference processing to obtain model inference data;
[0017] The inference data of the model is post-processed to obtain the inference data.
[0018] Preferably, according to the deep learning-based model inference method provided in this disclosure, the preprocessing of the original data input to the inference task set to obtain batch data includes:
[0019] The raw data is input into a preset model queue for storage and processing, and the number of raw data in the model queue is counted in real time.
[0020] If the model queue includes at least one of the original data, a preset timer is started to perform timing processing to obtain the timing time;
[0021] The timing time is compared with a preset time threshold, and the number of data in the original data is compared with a preset number threshold.
[0022] If the timing time is less than or equal to the preset time threshold, and the number of data in the model queue reaches the preset quantity threshold, then the original data in the model queue is combined to obtain the batch data.
[0023] Preferably, according to the deep learning-based model inference method provided in this disclosure, the step of preprocessing the original data input to the inference task set to obtain batch data further includes:
[0024] If the timing time is greater than the preset time threshold, and the number of data in the model queue has not reached the preset number threshold, then the model queue is filled to obtain fill data.
[0025] The batch data is obtained by combining the filled data and at least one of the original data in the model queue.
[0026] Preferably, according to the deep learning-based model inference method provided in this disclosure, the step of inputting the batch data into the inference model for model inference processing to obtain model inference data includes:
[0027] Based on the actual needs, obtain the data transmission method corresponding to the actual needs;
[0028] Using a preset inference interface and the data transmission method, each batch of data is asynchronously transmitted to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining multiple model inference data corresponding to each batch of data.
[0029] Preferably, according to the deep learning-based model inference method provided in this disclosure, the post-processing of the model inference data to obtain the inference data includes:
[0030] Using a preset segmentation rule, each model inference data is segmented to obtain the inference data. The segmentation rule is to segment each model inference data into the same number of inference data as the preset threshold number.
[0031] Secondly, this disclosure also provides a deep learning-based model inference apparatus, comprising:
[0032] The acquisition module is used to acquire the stage pool for constructing the inference task set;
[0033] The assembly module is used to configure the model parameters required to construct the inference task set according to actual needs, obtain the configuration file, determine the assembly stage according to the configuration file and multiple initial stages in the stage pool, and assemble the inference process according to the assembly stage.
[0034] A reasoning task set construction module is used to encapsulate the multiple assembly stages of assembling the reasoning process to obtain multiple reasoning tasks corresponding to the multiple assembly stages, and to construct a reasoning task set based on the multiple reasoning tasks.
[0035] The inference module is used to input the raw data to be processed into the inference task set for inference processing to obtain inference data.
[0036] Thirdly, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the deep learning-based model inference methods described above.
[0037] Fourthly, this disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep learning-based model inference method as described above.
[0038] Fifthly, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the deep learning-based model inference method described above.
[0039] This disclosure provides a deep learning-based model inference method, apparatus, device, and storage medium. The method involves: acquiring a stage pool for constructing an inference task set; configuring model parameters required for constructing the inference task set according to actual needs to obtain a configuration file; determining assembly stages based on the configuration file and multiple initial stages in the stage pool; assembling an inference flow based on the assembly stages; encapsulating each assembly stage to obtain multiple inference tasks corresponding to each assembly stage; constructing an inference task set based on the multiple inference tasks; and inputting the raw data to be processed into the inference task set for inference processing to obtain inference data. This method enables the arrangement and combination of multiple inference tasks according to actual scenario requirements to load inference models for inference processing, greatly improving the flexibility of the inference system and reducing the integration cost of the application programming interface. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is one of the flowcharts illustrating a deep learning-based model inference method provided in this embodiment of the disclosure;
[0042] Figure 2 This is a second flowchart illustrating a deep learning-based model inference method provided in this embodiment of the disclosure;
[0043] Figure 3 This is an embodiment of the present disclosure. Figure 1 One of the flowcharts for step S200;
[0044] Figure 4 This is an embodiment of the present disclosure. Figure 1 One of the flowcharts for step S400;
[0045] Figure 5 This is an embodiment of the present disclosure. Figure 4 One of the flowcharts for step S410;
[0046] Figure 6 This is an embodiment of the present disclosure. Figure 4 The second flowchart of step S410;
[0047] Figure 7 This is one of the schematic diagrams of asynchronous inference provided in the embodiments of this disclosure;
[0048] Figure 8 This is the third flowchart illustrating a deep learning-based model inference method provided in this embodiment of the disclosure;
[0049] Figure 9 This is a schematic diagram of the structure of a deep learning-based model inference device provided in an embodiment of this disclosure;
[0050] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0052] Initial stage: The deep learning model inference process is abstracted into multiple stages such as preprocessing stage, model inference stage, and postprocessing stage.
[0053] Assembly phase: The assembly phase is selected from multiple initial phases in the phase pool to assemble the inference process. The assembly phase may include, but is not limited to, the following: preprocessing assembly phase, H2D assembly phase, model inference assembly phase, D2H assembly phase and postprocessing assembly phase. The H2D assembly phase is the assembly phase for data transmission from the host side to the device side, and the D2H assembly phase is the assembly phase for data transmission from the device side to the host side.
[0054] Reasoning tasks: The multiple assembly stages of the assembly reasoning process are encapsulated to obtain multiple reasoning tasks corresponding to each assembly stage, which are used to construct a set of reasoning tasks.
[0055] GPU (graphics processing unit), also known as display core, visual processor, or display chip, is a microprocessor that performs image and graphics-related calculations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). It is a graphics processor with inference capabilities.
[0056] An NPU (neural-network processing unit) can be considered a component (or subsystem), and is sometimes referred to as an NPU coprocessor. Network processors (NPUs) employ a "data-driven parallel computing" architecture, are particularly adept at processing massive amounts of multimedia data such as video and images, and are network processors with inference capabilities.
[0057] The Standard Inference Interface (InferDK) encapsulates the underlying inference capabilities provided by the backends. It is used to load and initialize models, control data transmission between devices and hosts or between devices, including the model driver execution interface. It ensures thread safety for model inference and provides access to basic model parameters. By encapsulating the standard inference interface, the upper-level processes are unaware of the underlying hardware when performing model inference operations. Furthermore, by extending the backend of the standard inference interface, it can support more model inference capabilities. The upper-level processes are the logical layer above the standard inference interface's logical layer, corresponding to the service and process framework.
[0058] Backends: Typically, backends are encapsulated using the native inference runtime API provided by hardware device manufacturers.
[0059] Image Processing Interface (ProcessDK): This interface unifies and encapsulates the processing capabilities of different hardware devices, including image processing units. This shields the upper layers from the various interfaces provided by each hardware processing unit. For image data input inference, it provides atomic capabilities such as typical image matting and scaling.
[0060] The following is combined with Figures 1-10This disclosure describes a deep learning-based model inference method, apparatus, device, and storage medium that can arrange and combine multiple inference tasks according to actual scenario requirements to load inference models for inference processing. This not only greatly improves the flexibility of the inference system but also significantly enhances the user experience.
[0061] The specific implementation is illustrated in the following embodiments. First, a deep learning-based model inference method is described in the embodiments of this disclosure.
[0062] like Figure 1 As shown, it is a schematic diagram of the implementation process of a deep learning-based model inference method provided in an embodiment of this disclosure. A deep learning-based model inference method may include, but is not limited to, steps S100 to S400.
[0063] S100, obtain the stage pool for constructing the inference task set;
[0064] S200: Configure the model parameters required to construct the inference task set according to actual needs to obtain a configuration file, and determine the assembly stage according to the configuration file and multiple initial stages in the stage pool, and assemble the inference process according to the assembly stage.
[0065] S300, the assembly stages of the inference process are encapsulated to obtain multiple inference tasks corresponding to the assembly stages, and a set of inference tasks is constructed based on the multiple inference tasks.
[0066] S400, the raw data to be processed is input into the inference task set for inference processing to obtain inference data.
[0067] In step S100 of some embodiments, a stage pool for constructing the inference task set is obtained. It is understood that the deep learning model inference process is abstracted into a preprocessing stage, an inference stage, and a post-processing stage.
[0068] Furthermore, in the implementation of the inference process, it will be abstracted into a transmission phase according to the data transmission process of the inference device. The transmission phase may include, but is not limited to, the following four transmission phases:
[0069] Data transfer from the host side to the device side (H2D);
[0070] Data transfer from the device side to the host side (D2H);
[0071] Device-to-device data transmission (D2D);
[0072] Host-to-host data transfer (H2H).
[0073] In this context, the device side of the inference device is typically a GPU or NPU, while the host side is typically a CPU.
[0074] The stage pool includes at least the following initial stages: preprocessing stage, model inference stage, postprocessing stage, H2D, D2H, D2D and H2H, which are used to construct the inference task set.
[0075] In step S200 of some embodiments, the model parameters required to construct the inference task set are configured according to actual needs to obtain a configuration file. Then, based on the configuration file and multiple initial stages in the stage pool, assembly stages are determined, and the inference flow is assembled according to the assembly stages. It can be understood that after completing step S100 to obtain the stage pool for constructing the inference task set, the specific execution steps can be as follows: first, configure the model parameters required to construct the inference task set according to actual needs to obtain a configuration file; based on the configuration file, select multiple assembly stages from the multiple initial stages in the stage pool; then, determine the inference order of the multiple assembly stages according to the configuration file; and finally, assemble the multiple assembly stages into the inference flow according to the inference order.
[0076] It should be noted that the parameters required for constructing the inference task set should be configured according to actual needs. These parameters include, but are not limited to: the platform backend, model path, preprocessing, postprocessing, maximum batch latency, running device number, preset thresholds, and custom fields. The configuration file is obtained by configuring these parameters.
[0077] Furthermore, the configuration file is parsed through a preset configuration parsing module to obtain the meaning of the fields corresponding to the configuration parameters. The meaning of the fields corresponding to these configuration parameters can be used in every process step of the model inference process.
[0078] Based on the configuration file and multiple initial stages in the stage pool of step S100, an assembly stage is determined, and an inference flow is assembled based on the assembly stage.
[0079] refer to Figure 2 As shown, the inference process can be: preprocessing assembly stage - H2D assembly stage - model inference assembly stage - D2H assembly stage - postprocessing assembly stage.
[0080] In step S300 of some embodiments, the multiple assembly stages of assembling the inference process are encapsulated to obtain multiple inference tasks corresponding to the multiple assembly stages, and a set of inference tasks is constructed based on the multiple inference tasks.
[0081] Reference Figure 2 As shown, after executing step S200, the model parameters required to construct the inference task set are configured according to actual needs to obtain the configuration file. Then, based on the configuration file and multiple initial stages in the stage pool, the assembly stage is determined, and the inference process is assembled according to the assembly stage. After that, the multiple assembly stages of the assembled inference process are encapsulated to obtain multiple inference tasks corresponding to the multiple assembly stages.
[0082] It should be noted that the inference tasks include at least: preprocessing inference tasks, model inference tasks, postprocessing inference tasks, H2D inference tasks, and D2H inference tasks.
[0083] A set of reasoning tasks is constructed based on the above-mentioned reasoning tasks.
[0084] In step S400 of some embodiments, the raw data to be processed is input into the inference task set for inference processing to obtain inference data. It can be understood that after step S300 is completed, the specific execution steps may be: first, preprocessing the raw data input into the inference task set to obtain batch data; then, inputting the batch data into the inference model for model inference processing to obtain model inference data; and finally, assembling the multiple assembly stages into an inference process according to the inference order.
[0085] Through the embodiments provided above, multiple inference tasks can be arranged and combined according to actual scenario requirements. The raw data to be processed is input into the inference task set for inference processing to obtain inference data. This not only greatly improves the flexibility of the inference system, but also greatly enhances the user experience.
[0086] In some embodiments, reference Figure 3 As shown, step S200 may also include, but is not limited to, steps S310 to S330.
[0087] S310, Based on the configuration file, select a plurality of assembly stages from the plurality of initial stages in the stage pool;
[0088] S320, determine the reasoning order of the multiple assembly stages according to the configuration file;
[0089] S330, the multiple assembly stages are assembled into the reasoning flow according to the reasoning order.
[0090] In step S310 of some embodiments, multiple assembly stages are selected from multiple initial stages in the stage pool based on the configuration file. It is understood that multiple assembly stages are selected from multiple initial stages in the stage pool of step S100 according to the configuration file obtained in step S200.
[0091] For example, from multiple initial stages such as preprocessing, inference, postprocessing, H2D, D2H, and H2H, the preprocessing assembly stage, H2D assembly stage, model inference assembly stage, D2H assembly stage, and postprocessing assembly stage can be selected.
[0092] In step S320 of some embodiments, the reasoning order of the multiple assembly stages is determined according to the configuration file. It can be understood that after executing step S310, which selects multiple assembly stages from the multiple initial stages in the stage pool based on the configuration file, the specific execution steps may be: determining the reasoning order of the multiple assembly stages based on the meaning of the fields corresponding to the configuration parameters in the configuration file.
[0093] For example, refer to Figure 2 As shown in the configuration file, the data transmission process first transmits the pre-processed batch data from the host side to the device side (H2D); after the inference model loaded by the inference device performs inference on the batch data, the inference model inference data is then transmitted from the device side to the host side (D2H) for post-processing of the model inference data.
[0094] In step S330 of some embodiments, the multiple assembly stages are assembled into the inference flow according to the inference order. It can be understood that after step S320, which determines the inference order of the multiple assembly stages according to the configuration file, is executed, the specific execution step may be to assemble the multiple assembly stages into the inference flow according to the inference order determined in step S320.
[0095] Preferably, the inference process can be: preprocessing assembly stage - H2D assembly stage - model inference assembly stage - D2H assembly stage - postprocessing assembly stage.
[0096] In some embodiments, reference Figure 4 As shown, step S400 may also include, but is not limited to, steps S410 to S430.
[0097] S410, preprocess the original data input to the inference task set to obtain batch data;
[0098] S420, the batch data is input into the inference model for model inference processing to obtain model inference data;
[0099] S430, the multiple assembly stages are assembled into the reasoning flow according to the reasoning order.
[0100] In step S410 of some embodiments, the raw data input to the inference task set is preprocessed to obtain batch data. It can be understood that the specific execution steps can be: first, the raw data is input into a preset model queue for storage processing, and the number of raw data in the model queue is counted in real time. If the model queue includes at least one piece of raw data, a preset timer is started to perform timing processing to obtain the timing time. The timing time is compared with a preset time threshold, and the number of raw data is compared with a preset quantity threshold. If the timing time is less than or equal to the preset time threshold, and the number of data in the model queue reaches the preset quantity threshold, the raw data in the model queue is combined to obtain the batch data. Alternatively...
[0101] If the timing time is greater than the preset time threshold and the number of data in the model queue does not reach the preset number threshold, then the model queue is filled to obtain filled data. The filled data and at least one of the original data in the model queue are combined to obtain the batch data.
[0102] In step S420 of some embodiments, the batch data is input into the inference model for model inference processing to obtain model inference data. It can be understood that after executing step S410, the specific execution steps can be: first, obtaining the data transmission method corresponding to the actual needs based on the actual needs; then, using a preset inference interface and the data transmission method, transmitting each batch of data asynchronously to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining multiple sets of model inference data corresponding to each batch of data.
[0103] In step S430 of some embodiments, the model inference data is post-processed to obtain the inference data. It can be understood that after step S420, which involves inputting the batch data into the inference model for inference processing to obtain the model inference data, the specific execution steps can be: using a preset segmentation rule, each piece of model inference data is segmented to obtain the inference data, wherein the segmentation rule is to segment each piece of model inference data into the same number of inference data as the preset threshold number.
[0104] In some embodiments, reference Figure 5As shown, step S410 may also include, but is not limited to, steps S510 to S540.
[0105] S510, the raw data is input into a preset model queue for storage processing, and the number of raw data in the model queue is counted in real time;
[0106] S520, if the model queue includes at least one of the original data, then a preset timer is started to perform timing processing to obtain the timing time;
[0107] S530, compare the timing time with a preset time threshold, and compare the number of data in the original data with a preset number threshold;
[0108] S540, if the timing time is less than or equal to the preset time threshold, and the number of data in the model queue reaches the preset quantity threshold, then the original data in the model queue is combined to obtain the batch data.
[0109] In step S510 of some embodiments, the raw data is input into a preset model queue for storage processing, and the number of raw data in the model queue is counted in real time. It can be understood that by inputting the raw data into the model queue or data stack for storage processing, and counting each piece of raw data as it is stored, the number of raw data in the model queue can be counted in real time.
[0110] In step S520 of some embodiments, if the model queue includes at least one of the original data, a preset timer is started for timing processing to obtain the timing time. It can be understood that after executing step S510, which involves inputting the original data into the preset model queue for storage processing and counting the number of original data in the model queue in real time, the specific execution step can be: if the number of original data obtained in real time is greater than 0, start the preset timer for timing processing to obtain the timing time. A timeout mechanism can be introduced based on the timing time to maintain the timeliness of inference requests.
[0111] In step S530 of some embodiments, the timing time and a preset time threshold are compared, and the number of original data is compared with a preset quantity threshold. It can be understood that after executing step S520 (if the model queue includes at least one piece of original data, a preset timer is started to obtain the timing time), the specific execution steps can be to compare the timing time obtained in step S520 with the preset time threshold to determine if a timeout has occurred. Simultaneously, the number of original data in the model queue is compared with the preset quantity threshold to determine if the number of original data in the model queue meets the preset quantity threshold requirement. Combining time comparison and quantity comparison ultimately achieves batch processing, accumulating input original data until the maximum batch size is reached before performing model inference, thus further improving resource utilization.
[0112] In step S540 of some embodiments, if the timing time is less than or equal to the preset time threshold, and the number of data in the model queue reaches the preset quantity threshold, then the original data in the model queue is combined to obtain the batch data. It can be understood that after performing step S530, which compares the timing time with the preset time threshold and the number of original data with the preset quantity threshold, the specific execution steps can be as follows: for example, if the timing time is 5 seconds, the preset time threshold is 8 seconds, the number of data in the model queue is 4, and the preset quantity threshold is 4, then the timing time of 5 seconds is less than the preset time threshold of 8 seconds, and the number of data in the model queue reaches the preset quantity threshold. Therefore, the original data in the model queue is combined, i.e., treated as the same group of data, to obtain batch data.
[0113] In some embodiments, reference Figure 6 As shown, step S410 may also include, but is not limited to, steps S610 to S620.
[0114] S610, if the timing time is greater than the preset time threshold and the number of data in the model queue has not reached the preset number threshold, then the model queue is filled to obtain filled data;
[0115] S620, the filling data and at least one of the original data in the model queue are combined and processed to obtain the batch data.
[0116] In step S610 of some embodiments, if the timing time is greater than the preset time threshold and the number of data in the model queue has not reached the preset number threshold, then the model queue is filled to obtain filled data. It can be understood that after performing step S530, which compares the timing time with the preset time threshold and compares the number of original data with the preset number threshold, the specific execution steps can be as follows: for example, if the timing time is 9 seconds, the preset time threshold is 8 seconds, the number of data in the model queue is 2, and the preset number threshold is 4, then to maintain the timeliness of the inference data, the model queue is filled to the same number as the preset number threshold; the data filled into the model queue is the filled data.
[0117] In step S620 of some embodiments, the fill data and at least one of the original data in the model queue are combined to obtain the batch data. It can be understood that after step S610, where the timing time is greater than the preset time threshold and the number of data in the model queue has not reached the preset quantity threshold, the specific execution steps can be: combining the fill data obtained in step S610 with at least one of the original data in the model queue to obtain the batch data.
[0118] In some embodiments, step S420 inputs the batch data into the inference model for model inference processing to obtain model inference data, and may also include, but is not limited to:
[0119] Based on the actual needs, obtain the data transmission method corresponding to the actual needs;
[0120] Using a preset inference interface and the data transmission method, each batch of data is asynchronously transmitted to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining multiple model inference data corresponding to each batch of data.
[0121] Understandably, based on the user's actual needs, the data transmission method corresponding to those needs should be obtained. For example, the transmission method may include, but is not limited to, the following four transmission methods:
[0122] Data transfer methods from host to device (H2D); data transfer from device to host (D2H); data transfer from device to device (D2D); and data transfer from host to host (H2H).
[0123] In this context, the device side of the inference device is typically a GPU or NPU, while the host side is typically a CPU.
[0124] Using a preset inference interface and data transmission method, each batch of processed data is transmitted asynchronously to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining multiple model inference data corresponding to each batch of processed data.
[0125] The preset inference interface may include, but is not limited to, the standard inference interface and the image processing interface.
[0126] The standard inference interface (InferDK) encapsulates the low-level inference capabilities provided by the backends. It is used to load and initialize models, control data transmission between devices and the host or between devices, including the model driver execution interface. It ensures thread safety for model inference and provides access to basic model parameters. By encapsulating the standard inference interface, the upper-level processes are unaware of the underlying hardware devices when performing model inference operations. Furthermore, by extending the backend of the standard inference interface, it can support more model inference capabilities.
[0127] Backends: Typically, backends are encapsulated using the native inference runtime API provided by hardware device manufacturers.
[0128] Image Processing Interface (ProcessDK): This interface unifies and encapsulates the processing capabilities of different hardware devices, including image processing units. This shields the upper layers from the various interfaces provided by each hardware processing unit. For image data input inference, it provides atomic capabilities such as typical image matting and scaling.
[0129] refer to Figure 7 As shown, synchronous execution is the existing execution method, while asynchronous and batch processing are among the core technical points of this disclosure. Specifically, each batch of processed data is transmitted asynchronously to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining multiple model inference data corresponding to each batch of processed data. For example, if there are two batches of processed data, A and B, these two batches are transmitted asynchronously to the inference device corresponding to the inference interface to load the inference model for inference processing, thus obtaining model inference data corresponding to each batch of processed data.
[0130] In some embodiments, step S430 may post-process the model inference data to obtain the inference data, and may include, but is not limited to:
[0131] Using a preset segmentation rule, each model inference data is segmented to obtain the inference data. The segmentation rule is to segment each model inference data into the same number of inference data as the preset threshold number.
[0132] Understandably, by using a preset segmentation rule, where each model's inference data is divided into the same number of inference data points as a preset threshold (e.g., 4 points), each model's inference data is divided into 4 inference data points. This reduces concurrency latency and further improves resource utilization.
[0133] Preferably, in some embodiments, reference is made to Figure 8 As shown, for example, a timeout mechanism is used to preprocess the four original data sets separately, the preprocessed original data sets are assembled into a batch data set, and the batch data set is transmitted to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining the model inference data corresponding to the batch data set. Then, the model inference data is segmented using a preset segmentation rule to obtain the inference data set. The segmentation rule is to segment the model inference data set into inference data sets with the same number of segments as the preset threshold.
[0134] This disclosure provides a deep learning-based model inference method. It involves obtaining a stage pool for constructing an inference task set; configuring the model parameters required for constructing the inference task set according to actual needs to obtain a configuration file; determining assembly stages based on the configuration file and multiple initial stages in the stage pool; assembling the assembly stages into an inference flow; encapsulating each assembly stage to obtain multiple inference tasks corresponding to each assembly stage; and constructing an inference task set based on these multiple inference tasks. The raw data to be processed is input into the inference task set for inference processing to obtain inference data. This method enables the arrangement and combination of multiple inference tasks according to actual scenario requirements to load inference models for inference processing, greatly improving the flexibility of the inference system and reducing the integration cost of the application programming interface.
[0135] In related technologies, ordinary computing modules or functions use a synchronous waiting method, which causes heterogeneous computing chips to need to synchronously wait for CPU processing time, resulting in long end-to-end time and low hardware resource utilization efficiency.
[0136] The embodiments provided in this disclosure utilize a combination of asynchronous execution and batch processing to eliminate the need for synchronous waiting during heterogeneous computing chip computation, thereby reducing end-to-end latency and improving hardware resource utilization efficiency.
[0137] The embodiments provided in this disclosure also introduce a timeout mechanism to maintain the timeliness of inference requests, and implement batch processing operations to accumulate the original input data and perform model inference only when the maximum batch size is reached, thereby further improving resource utilization.
[0138] The following describes a deep learning-based model inference device provided in this disclosure. The deep learning-based model inference device described below can be referred to in correspondence with the deep learning-based model inference method described above.
[0139] Reference Figure 9 As shown, a deep learning-based model inference device includes:
[0140] Module 910 is used to obtain the stage pool for constructing the inference task set;
[0141] Assembly module 920 is used to configure the model parameters required to construct the inference task set according to actual needs, obtain a configuration file, determine the assembly stage according to the configuration file and multiple initial stages in the stage pool, and assemble the inference process according to the assembly stage.
[0142] The inference task set construction module 930 is used to encapsulate the multiple assembly stages of assembling the inference process to obtain multiple inference tasks corresponding to the multiple assembly stages, and to construct an inference task set based on the multiple inference tasks.
[0143] The reasoning module 940 is used to input the raw data to be processed into the reasoning task set for reasoning processing to obtain reasoning data.
[0144] According to the deep learning-based model inference apparatus provided in this disclosure, the assembly module 920 is further configured to, based on the configuration file, select multiple assembly stages from multiple initial stages in the stage pool; and
[0145] The reasoning order of the multiple assembly stages is determined according to the configuration file;
[0146] The multiple assembly stages are assembled into the reasoning flow according to the reasoning sequence.
[0147] According to the deep learning-based model inference device provided in this disclosure, the inference module 940 is further used to preprocess the original data input to the inference task set to obtain batch data;
[0148] The batch data is input into the inference task set for model inference processing to obtain model inference data;
[0149] The inference data of the model is post-processed to obtain the inference data.
[0150] According to the deep learning-based model inference device provided in this disclosure, the inference module 940 is further used to input the original data into a preset model queue for storage processing, and to count the number of original data in the model queue in real time.
[0151] If the model queue includes at least one of the original data, a preset timer is started to perform timing processing to obtain the timing time;
[0152] The timing time is compared with a preset time threshold, and the number of data in the original data is compared with a preset number threshold.
[0153] If the timing time is less than or equal to the preset time threshold, and the number of data in the model queue reaches the preset quantity threshold, then the original data in the model queue is combined to obtain the batch data.
[0154] According to the deep learning-based model inference device provided in this disclosure, the inference module 940 is further configured to fill the model queue to obtain fill data if the timing time is greater than the preset time threshold and the number of data in the model queue has not reached the preset number threshold.
[0155] The batch data is obtained by combining the filled data and at least one of the original data in the model queue.
[0156] According to the deep learning-based model inference device provided in this disclosure, the inference module 940 is further used to obtain the data transmission method corresponding to the actual needs according to the actual needs;
[0157] Using a preset inference interface and the data transmission method, each batch of data is asynchronously transmitted to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining multiple model inference data corresponding to each batch of data.
[0158] According to the deep learning-based model inference device provided in this disclosure, the inference module 940 is further configured to perform segmentation processing on each of the model inference data using a preset segmentation rule to obtain the inference data, wherein the segmentation rule is to segment each of the model inference data into the same number of inference data as the preset quantity threshold.
[0159] This disclosure provides a deep learning-based model inference device that obtains a stage pool for constructing an inference task set; configures the model parameters required for constructing the inference task set according to actual needs to obtain a configuration file; determines an assembly stage based on the configuration file and multiple initial stages in the stage pool; assembles the assembly stage into an inference flow; encapsulates each assembly stage to obtain multiple inference tasks corresponding to each assembly stage; and constructs an inference task set based on the multiple inference tasks; inputs the raw data to be processed into the inference task set for inference processing to obtain inference data. This device enables the arrangement and combination of multiple inference tasks according to actual scenario requirements to load inference models for inference processing, greatly improving the flexibility of the inference system and reducing the integration cost of the application programming interface.
[0160] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a deep learning-based model inference method. This method includes: obtaining a stage pool for constructing an inference task set; configuring the model parameters required to construct the inference task set according to actual needs to obtain a configuration file, and determining assembly stages based on the configuration file and multiple initial stages in the stage pool, and assembling an inference flow based on the assembly stages; encapsulating the multiple assembly stages of the inference flow to obtain multiple inference tasks corresponding to the multiple assembly stages, and constructing an inference task set based on the multiple inference tasks; inputting the raw data to be processed into the inference task set for inference processing to obtain inference data.
[0161] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0162] On the other hand, this disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a deep learning-based model inference method provided by the above methods. The method includes: obtaining a stage pool for constructing an inference task set; configuring the model parameters required to construct the inference task set according to actual needs to obtain a configuration file, and determining an assembly stage according to the configuration file and multiple initial stages in the stage pool, and assembling an inference flow according to the assembly stage; encapsulating the multiple assembly stages of assembling the inference flow to obtain multiple inference tasks corresponding to the multiple assembly stages, and constructing an inference task set according to the multiple inference tasks; inputting the raw data to be processed into the inference task set for inference processing to obtain inference data.
[0163] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform a deep learning-based model inference method provided by the methods described above. This method includes: obtaining a stage pool for constructing an inference task set; configuring model parameters required for constructing the inference task set according to actual needs to obtain a configuration file; determining assembly stages based on the configuration file and multiple initial stages in the stage pool; assembling an inference flow based on the assembly stages; encapsulating the multiple assembly stages of the inference flow to obtain multiple inference tasks corresponding to the multiple assembly stages; constructing an inference task set based on the multiple inference tasks; and inputting the raw data to be processed into the inference task set for inference processing to obtain inference data.
[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.
Claims
1. A model inference method based on deep learning, characterized in that, include: Obtain the stage pool for constructing the inference task set; Configure the model parameters required to construct the inference task set according to actual needs to obtain a configuration file, and determine the assembly stage according to the configuration file and multiple initial stages in the stage pool, and assemble the inference process according to the assembly stage. Each assembly stage of the inference process is encapsulated to obtain multiple inference tasks corresponding to each assembly stage, and a set of inference tasks is constructed based on the multiple inference tasks. The raw data to be processed is input into the inference task set for inference processing to obtain inference data; The process of inputting the raw data to be processed into the inference task set for inference processing to obtain inference data includes: Preprocessing the original data input to the inference task set to obtain batch data; the preprocessing of the original data input to the inference task set to obtain batch data includes: The raw data is input into a preset model queue for storage and processing, and the number of raw data in the model queue is counted in real time. If the model queue includes at least one of the original data, a preset timer is started to perform timing processing to obtain the timing time; The timing time is compared with a preset time threshold, and the number of data in the original data is compared with a preset number threshold. If the timing time is less than or equal to the preset time threshold, and the number of data in the model queue reaches the preset number threshold, then the original data in the model queue is combined to obtain the batch data. The batch data is input into the inference model for model inference processing to obtain model inference data; The inference data of the model is post-processed to obtain the inference data.
2. The deep learning-based model inference method according to claim 1, characterized in that, The step of determining the assembly stage based on the configuration file and multiple initial stages in the stage pool, and assembling the inference process based on the assembly stage, includes: Based on the configuration file, multiple assembly stages are selected from multiple initial stages in the stage pool; and The reasoning order of the multiple assembly stages is determined according to the configuration file; The multiple assembly stages are assembled into the reasoning flow according to the reasoning sequence.
3. The deep learning-based model inference method according to claim 1, characterized in that, The step of preprocessing the original data input to the inference task set to obtain batch data further includes: If the timing time is greater than the preset time threshold, and the number of data in the model queue has not reached the preset number threshold, then the model queue is filled to obtain fill data. The batch data is obtained by combining the filled data and at least one of the original data in the model queue.
4. The deep learning-based model inference method according to claim 1, characterized in that, The step of inputting the batch data into the inference model for model inference processing to obtain model inference data includes: Based on the actual needs, obtain the data transmission method corresponding to the actual needs; Using a preset inference interface and the data transmission method, each batch of data is asynchronously transmitted to the inference device corresponding to the inference interface to load the inference model for inference processing, thereby obtaining multiple model inference data corresponding to each batch of data.
5. The deep learning-based model inference method according to claim 1, characterized in that, The post-processing of the model inference data to obtain the inference data includes: Using a preset segmentation rule, each model inference data is segmented to obtain the inference data. The segmentation rule is to segment each model inference data into the same number of inference data as the preset quantity threshold.
6. A model inference device based on deep learning, characterized in that, include: The acquisition module is used to acquire the stage pool for constructing the inference task set; The assembly module is used to configure the model parameters required to construct the inference task set according to actual needs, obtain the configuration file, determine the assembly stage according to the configuration file and multiple initial stages in the stage pool, and assemble the inference process according to the assembly stage. A reasoning task set construction module is used to encapsulate the multiple assembly stages of assembling the reasoning process to obtain multiple reasoning tasks corresponding to the multiple assembly stages, and to construct a reasoning task set based on the multiple reasoning tasks. The reasoning module is used to input the raw data to be processed into the reasoning task set for reasoning processing to obtain reasoning data; The step of inputting the raw data to be processed into the inference task set for inference processing to obtain inference data includes: preprocessing the raw data input into the inference task set to obtain batch data; the preprocessing of the raw data input into the inference task set to obtain batch data includes: inputting the raw data into a preset model queue for storage processing, and counting the number of raw data in the model queue in real time; if the model queue includes at least one piece of raw data, then starting a preset timer for timing processing to obtain a timing time; comparing the timing time with a preset time threshold, and comparing the number of raw data with a preset number threshold; if the timing time is less than or equal to the preset time threshold, and the number of data in the model queue reaches the preset number threshold, then combining the raw data in the model queue to obtain the batch data; inputting the batch data into the inference model for model inference processing to obtain model inference data; and postprocessing the model inference data to obtain the inference data.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the deep learning-based model inference method as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the deep learning-based model inference method as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the deep learning-based model inference method as described in any one of claims 1 to 5.