Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a data processing method and device, electronic equipment and storage medium. The method comprises: for a plurality of trained neural network models, creating an inference engine corresponding to each neural network model; creating a target execution context corresponding to each inference engine to obtain a plurality of target execution contexts; the target execution context does not allocate running video memory; determining the running video memory occupation of each inference engine, and selecting the maximum running video memory occupation as the target running video memory occupation; determining the target video memory based on the target running video memory occupation, and taking the target video memory as the running video memory of the plurality of target execution contexts. In the present disclosure, the plurality of target execution contexts can share the target video memory, realizing the reuse of the target video memory by the plurality of inference engines corresponding to the plurality of neural network models, saving video memory occupation while ensuring high flexibility and no delay effect.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to data processing methods, apparatus, electronic devices and storage media. Background Technology

[0002] With the rapid development of computer technology, neural network models have been gradually applied to various fields such as computer vision and natural language processing, achieving good results. Currently, by running deep learning inference frameworks on graphics processing unit (GPU) hardware platforms, deep learning models can be deployed and inferred on GPUs with low latency and high throughput.

[0003] In related technologies, to save GPU memory, a multi-model single-engine technique is usually adopted for the deployment and inference of multiple neural network models. This technique puts multiple models into one engine, and multiple models need to run simultaneously, resulting in poor flexibility and high latency. As a result, the deployment and inference of multiple neural network models in related technologies cannot ensure high flexibility and low latency while saving GPU memory. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, electronic device, and storage medium to at least solve the problem in related technologies where the deployment and inference of multiple neural network models cannot ensure high flexibility and low latency while saving video memory. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0006] For multiple trained neural network models, create an inference engine corresponding to each neural network model;

[0007] A target execution context is created for each inference engine, resulting in multiple target execution contexts; no runtime memory is allocated for each target execution context.

[0008] Determine the memory usage of each inference engine and select the largest memory usage as the target memory usage.

[0009] The target video memory is determined based on the target video memory usage, and the target video memory is used as the video memory for the execution context of the plurality of targets.

[0010] In some possible implementations, using the target video memory as the runtime video memory of the plurality of target execution contexts includes:

[0011] Obtain the pointer information of the target video memory;

[0012] For each target execution context, the pointer information is used as the input parameter of the video memory setting function, and the target video memory is designated as the running video memory of the target execution context by calling the video memory setting function.

[0013] In some possible implementations, after using the target video memory as the runtime video memory of the plurality of target execution contexts, the method further includes:

[0014] In response to a model addition request, create the inference engine corresponding to the added neural network model;

[0015] Create the target execution context corresponding to the added inference engine to obtain the added target execution context;

[0016] Obtain the memory usage corresponding to the added inference engine to get the first memory usage.

[0017] If the first running video memory usage is greater than the target running video memory usage, the target video memory is updated based on the first running video memory usage to obtain an updated first target video memory; the updated first target video memory is greater than the target video memory.

[0018] The updated first target memory is used as the runtime memory for the execution context of the plurality of targets and the added target execution context.

[0019] In some possible implementations, the method further includes:

[0020] If the first running video memory usage is less than or equal to the target running video memory usage, the target video memory is used as the running video memory for the plurality of target execution contexts and the added target execution context.

[0021] In some possible implementations, after using the target video memory as the runtime video memory of the plurality of target execution contexts, the method further includes:

[0022] In response to the model reduction request, the largest memory usage among the remaining inference engines is determined to obtain the remaining target memory usage; the remaining inference engines are the inference engines remaining after the model reduction process.

[0023] If the remaining target memory usage is less than the target memory usage, the target memory is updated based on the remaining target memory usage to obtain an updated second target memory; the updated second target memory is less than the target memory.

[0024] The updated second target memory is used as the runtime memory of the target execution context corresponding to the remaining inference engine.

[0025] In some possible implementations, after using the target video memory as the runtime video memory of the plurality of target execution contexts, the method further includes:

[0026] In response to a first running request for a first neural network model, a first inference engine is obtained by determining the inference engine corresponding to the first neural network model; the first neural network model is any one of the plurality of neural network models.

[0027] Based on the memory usage of the first inference engine, the first memory region is locked in the target memory.

[0028] The first running request is processed based on the first video memory region using the target execution context corresponding to the first inference engine.

[0029] In some possible implementations, the method further includes:

[0030] During the processing of the first running request, if a second running request for the second neural network model is obtained, the remaining target video memory capacity is determined; the remaining target video memory is the video memory area other than the first video memory area, and the second neural network model is any neural network model other than the first neural network model among the plurality of neural network models.

[0031] The second inference engine is obtained by determining the inference engine corresponding to the second neural network model.

[0032] When the memory usage of the second inference engine is less than or equal to the memory capacity of the remaining target memory, the second running request is processed based on the remaining target memory using the target execution context of the second inference engine.

[0033] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0034] The engine creation unit is configured to perform the following: for multiple trained neural network models, create an inference engine corresponding to each neural network model.

[0035] The execution context creation unit is configured to create a target execution context corresponding to each inference engine, resulting in multiple target execution contexts; the target execution contexts are not allocated runtime memory.

[0036] The video memory usage determination unit is configured to determine the running video memory usage corresponding to each of the inference engines, and select the largest running video memory usage as the target running video memory usage.

[0037] The first runtime video memory determination unit is configured to determine the target video memory based on the target runtime video memory usage, and use the target video memory as the runtime video memory of the plurality of target execution contexts.

[0038] In some possible implementations, the first runtime memory determination unit includes:

[0039] The pointer information acquisition unit is configured to acquire the pointer information of the target video memory;

[0040] The function call unit is configured to execute, for each target execution context, using the pointer information as input parameters to a video memory setting function, and to specify the target video memory as the running video memory of the target execution context by calling the video memory setting function.

[0041] In some possible implementations, the device further includes:

[0042] The first response unit is configured to respond to the model addition request by creating the addition inference engine corresponding to the added neural network model.

[0043] The first creation unit is configured to create the target execution context corresponding to the added inference engine, thereby obtaining the added target execution context;

[0044] The first acquisition unit is configured to acquire the running memory usage corresponding to the added inference engine, and obtain the first running memory usage.

[0045] The first update unit is configured to update the target video memory based on the first video memory usage when the first running video memory usage is greater than the target video memory usage, to obtain an updated first target video memory; the updated first target video memory is greater than the target video memory.

[0046] The second runtime memory determination unit is configured to use the updated first target runtime memory as the runtime memory for the plurality of target execution contexts and the added target execution context.

[0047] In some possible implementations, the device further includes:

[0048] The third runtime memory determination unit is configured to, when the first runtime memory usage is less than or equal to the target runtime memory usage, use the target runtime memory as the runtime memory of the plurality of target execution contexts and the added target execution context.

[0049] In some possible implementations, the device further includes:

[0050] The second response unit is configured to respond to the model reduction request by determining the largest runtime memory usage among the remaining inference engines, and obtaining the remaining target runtime memory usage; the remaining inference engines are the inference engines remaining after the model reduction process.

[0051] The second update unit is configured to update the target video memory based on the remaining target video memory usage when the remaining target video memory usage is less than the target video memory usage, to obtain an updated second target video memory; the updated second target video memory is less than the target video memory.

[0052] The third runtime memory determination unit is configured to use the updated second target runtime memory as the runtime memory of the target execution context corresponding to the remaining inference engine.

[0053] In some possible implementations, the device further includes:

[0054] The third response unit is configured to execute a response to a first running request for the first neural network model, determine the inference engine corresponding to the first neural network model, and obtain the first inference engine; the first neural network model is any one of the plurality of neural network models.

[0055] The video memory locking unit is configured to perform a first video memory region locking in the target video memory based on the running video memory usage corresponding to the first inference engine.

[0056] The first request processing unit is configured to process the first running request based on the first video memory region using the target execution context corresponding to the first inference engine.

[0057] In some possible implementations, the device further includes:

[0058] The remaining target video memory capacity determination unit is configured to determine the remaining target video memory capacity if a second running request for a second neural network model is obtained during the processing of the first running request; the remaining target video memory is the video memory region other than the first video memory region, and the second neural network model is any neural network model other than the first neural network model among the plurality of neural network models.

[0059] The engine determination unit is configured to determine the inference engine corresponding to the second neural network model to obtain the second inference engine.

[0060] The second request processing unit is configured to process the second running request based on the remaining target memory when the running memory usage corresponding to the second inference engine is less than or equal to the memory capacity of the remaining target memory.

[0061] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0062] processor;

[0063] Memory used to store the processor's executable instructions;

[0064] The processor is configured to execute the instructions to implement the data processing method of the first aspect described above.

[0065] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the data processing method of the first aspect described above.

[0066] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method of the first aspect described above.

[0067] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0068] For multiple trained neural network models, an inference engine is created for each neural network model, and a target execution context without memory allocation is created for each inference engine. Then, the largest memory usage among the memory usage of the multiple inference engines is selected as the target memory usage, and the target memory is determined based on this target memory usage. This target memory is then used as the execution memory for the multiple target execution contexts, allowing the multiple target execution contexts to share the target memory. This enables multiple inference engines corresponding to multiple neural network models to reuse the target memory, saving memory usage while ensuring high flexibility and zero latency.

[0069] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0071] Figure 1 This is a schematic diagram illustrating an application environment of a data processing method according to an exemplary embodiment;

[0072] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment;

[0073] Figure 3 This is an example of a video memory layout illustrated according to an exemplary embodiment;

[0074] Figure 4 This is a flowchart illustrating another data processing method according to an exemplary embodiment;

[0075] Figure 5 This is a flowchart illustrating another data processing method according to an exemplary embodiment;

[0076] Figure 6 This is a flowchart illustrating another data processing method according to an exemplary embodiment;

[0077] Figure 7 This is a flowchart illustrating another data processing method according to an exemplary embodiment;

[0078] Figure 8 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment;

[0079] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0080] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0081] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0082] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0083] In practical applications, neural network models typically involve two phases: the training phase and the inference phase. The training phase is the process by which the network continuously learns from the training data, including forward propagation and backpropagation. Forward propagation is used to predict labels, and then backpropagation is performed to continuously modify the network weights based on the difference between the predicted labels and the true labels. The network weights are constantly changing throughout the training process. The purpose of the inference phase is to output the predicted labels. It only includes forward propagation, and the network weights remain unchanged. In short, the inference phase is to use the trained neural network model to make predictions, and speed is a very important factor during prediction.

[0084] Deep learning inference frameworks can optimize trained neural network models, accelerating inference. Specifically, deep learning inference frameworks enable high-performance inference acceleration of neural network models on GPUs. In other words, a deep learning inference framework only involves forward propagation. This framework can parse neural network models trained on various deep learning frameworks, then map them one-to-one with the corresponding layers in the deep learning inference framework, unifying models from other frameworks into the deep learning inference framework. Then, optimization strategies can be implemented for GPUs within the deep learning inference framework, and deployment acceleration can be achieved.

[0085] In related technologies, to save on GPU memory usage during inference, a multi-model single-engine technique is used, which involves placing multiple neural network models into a single inference engine. These models need to run simultaneously, resulting in at least the following drawbacks: 1) Inflexible construction: It requires prior knowledge of which neural network models to combine, making dynamic adjustments at runtime impossible. Construction is time-consuming, and removing models necessitates rebuilding all models, increasing debugging complexity. 2) Inflexible operation: Multiple neural network models can only be invoked simultaneously, not at different frequencies. Furthermore, business logic requires placing inputs and outputs from different contexts (different services) into the same context for inference processing. 3) High latency: The output of each neural network model can only be obtained after the entire inference engine has finished running. For example, if the individual execution times of four neural network models are 3ms, 3ms, 4ms, and 4ms respectively, ideally, the results would be output at 3ms, 6ms, 10ms, and 14ms respectively. However, in related technologies, the results of all models are only obtained at 14ms, resulting in high latency.

[0086] Given the limitations of related technologies in deploying and inferring multiple neural network models while ensuring high flexibility and low latency while saving GPU memory, this disclosure provides a data processing method. This method creates an inference engine for each of the multiple trained neural network models and a target execution context without allocated GPU memory for each inference engine. Then, it selects the largest GPU memory usage from the multiple inference engines as the target GPU memory usage and determines the target GPU memory based on this target GPU memory usage. This target GPU memory is then used as the execution memory for the multiple target execution contexts, allowing them to share the target GPU memory. This enables multiple inference engines corresponding to multiple neural network models to reuse the target GPU memory, significantly saving GPU memory usage. Furthermore, each neural network model has its own inference engine, ensuring both construction and operational flexibility. Additionally, each inference engine can output the corresponding model results, achieving a latency-free inference effect.

[0087] Please see Figure 1 The diagram illustrates an application environment for a data processing method according to an exemplary embodiment. The application environment may include a terminal 110 and a server 120, which can be connected via a wired network or a wireless network.

[0088] Terminal 110 can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Terminal 110 may have client software, such as an application (App), installed to provide data processing functions. This application can be a standalone application or a subroutine within an application. For example, the application may include a video application, a live streaming application, etc. Users of terminal 110 can log in to the application using pre-registered user information, which may include an account and password.

[0089] Specifically, the application in terminal 110 is a deep learning inference framework-based application. When processing data, it can call a neural network model deployed based on the deep learning inference framework to perform inference (i.e., prediction processing) to obtain the data processing results. The neural network model can correspond to the actual scenario of the application; for example, the neural network model can include deep learning models based on computer vision (such as image recognition models) and deep learning models based on natural language processing (such as translation models).

[0090] Server 120 can be a server providing background services for applications in terminal 110. Specifically, server 120 uses a deep learning inference framework to optimize a trained neural network model, and when the application calls the neural network model for data processing, the optimized model can be run on a GPU to accelerate inference. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0091] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment, such as... Figure 2 As shown, data processing methods are applied to Figure 1 Taking the server as an example, the following steps are included:

[0092] In step S201, for multiple trained neural network models, an inference engine is created for each neural network model.

[0093] Among them, the multiple trained neural network models can be neural network models trained using any deep learning framework.

[0094] Each neural network model corresponds to an inference engine, which is a forward computation instance.

[0095] In specific implementation, since different deep learning frameworks can be used for training during the training phase of the neural network model, for each trained neural network model, a parser corresponding to the format of the training framework can be used to convert the trained neural network model into the internal network of the deep learning inference framework. After successful parsing, an optimized inference engine is built according to the network optimization configuration (i.e., BuilderConfig). The network optimization configuration (i.e., BuilderConfig) specifies the attribute information used to create the inference engine, including batch size, workspace size, mixed precision, and dynamic shape boundaries, etc.

[0096] In practical applications, for each trained neural network model, the inference engine creation function can be called to create the corresponding inference engine.

[0097] In step S203, a target execution context is created for each inference engine, resulting in multiple target execution contexts.

[0098] The target execution context refers to the Execution Context, which is the execution environment required by the runtime inference engine.

[0099] It should be noted that in this embodiment of the present disclosure, the target execution context created in step S203 does not allocate runtime memory. That is, the target execution context at this time does not have workspace. Runtime memory, or workspace, is used as temporary storage space when each layer of network is executed.

[0100] In practice, for each inference engine, the target execution context creation function can be called to create a target execution context without a workspace.

[0101] In step S205, the memory usage of each inference engine is determined, and the largest memory usage is selected as the target memory usage.

[0102] The amount of video memory used by the inference engine is the workspace size.

[0103] In practice, for each inference engine, the device memory size acquisition function can be called to obtain the workspace size of that inference engine and thus its memory usage.

[0104] Assuming there are n trained neural network models, step S205 can yield n one-to-one corresponding inference engines, as well as the memory usage of each of the n inference engines. Then, based on the memory usage of these n inference engines, the largest memory usage is selected as the target memory usage.

[0105] In step S207, the target video memory is determined based on the target running video memory usage, and this target video memory is used as the running video memory for multiple target execution contexts.

[0106] Specifically, the target video memory capacity can be equal to or greater than the target runtime video memory usage. When the target video memory capacity is greater than the target runtime video memory usage, a preset offset capacity can be obtained, the sum of the target runtime video memory usage and the preset offset capacity can be calculated, and then this sum can be used as the target video memory capacity for target video memory allocation.

[0107] In specific implementation, when the target video memory capacity is equal to the target running video memory usage, the target running video memory usage can be used as a parameter to call the dynamic video memory allocation function cudaMalloc() to allocate video memory and obtain a target video memory with a capacity equal to the target running video memory usage; when the target video memory capacity is the sum of the target running video memory usage and the preset offset capacity, this sum can be used as a parameter to call the dynamic video memory allocation function cudaMalloc() to allocate video memory and obtain a target video memory with a capacity equal to this sum.

[0108] Using the target memory as the runtime memory for multiple target execution contexts allows multiple target execution contexts to share the target memory. In other words, when each target execution context performs inference based on its corresponding inference engine, it will use the target memory as temporary storage space for each layer of the network during execution.

[0109] like Figure 3 The illustration shows an example of a memory layout provided in an embodiment of this disclosure. This example uses two trained neural network models as an example, creating inference engines and corresponding target execution contexts for each. Runtime memory is created based on the largest workspace size among the workspace sizes corresponding to the inference engines, and this runtime memory is designated as a shared workspace for the target execution contexts corresponding to the two inference engines. For example, if the workspace size of the inference engine corresponding to neural network model 1 is 1GB and the workspace size of the inference engine corresponding to neural network model 2 is 2GB, then 2GB of runtime memory is created and designated as a shared workspace for the target execution contexts corresponding to the two inference engines.

[0110] In some possible implementations, step S207 above, when using the target video memory as the running video memory of multiple target execution contexts, may include: obtaining pointer information of the target video memory; for each target execution context, using the pointer information as the input parameter of the video memory setting function, and using the video memory setting function to designate the target video memory as the running video memory of the target execution context.

[0111] The pointer information of the target video memory represents the address of the target video memory. The above implementation uses the pointer information of the target video memory as a parameter to call the video memory setting function to assign the target video memory as the running video memory (workspace) of each of the multiple target execution contexts, thereby enabling multiple target execution contexts to share the target video memory.

[0112] This embodiment creates an inference engine for each neural network model and a target execution context without runtime memory (i.e., workspace) for each inference engine. It then determines the runtime memory usage for each inference engine, selects the largest runtime memory usage as the target runtime memory usage, determines the target runtime memory based on this target runtime memory usage, and designates this target runtime memory as the runtime memory for multiple target execution contexts. This achieves shared runtime memory (i.e., workspace) for multiple target execution contexts, enabling multiple inference engines corresponding to multiple neural network models to reuse this runtime memory, significantly saving runtime memory usage. Simultaneously, each neural network model has its own inference engine, ensuring flexibility in construction and operation. Furthermore, each inference engine can output the corresponding model results, thus achieving zero-latency inference.

[0113] In some possible implementations, such as Figure 4 A flowchart of another data processing method is provided. After step S207 above, the method may further include:

[0114] In step S401, in response to the model addition request, an addition inference engine corresponding to the added neural network model is created.

[0115] The model addition request is used to request the addition of a trained neural network model. The neural network model to be added can be a neural network model trained using any deep learning framework.

[0116] To create the inference engine corresponding to the added neural network model (i.e., the trained neural network model that is requested to be added) (hereinafter referred to as the added inference engine), you can call the inference engine creation function to create the added inference engine.

[0117] In step S403, a target execution context corresponding to the added inference engine is created, and the added target execution context is obtained.

[0118] In this case, the added target context has not been allocated runtime memory, meaning that the added target execution context does not include a workspace.

[0119] In practice, for the above-mentioned addition of an inference engine, the target execution context creation function can be called to create a target execution context without a workspace.

[0120] In step S405, the memory usage corresponding to the added inference engine is obtained to obtain the first memory usage.

[0121] In practice, the device memory size acquisition function can be called to obtain the workspace size of the added inference engine and thus its memory usage.

[0122] In step S407, it is determined whether the first running video memory usage is greater than the target running video memory usage.

[0123] Specifically, if the first running video memory usage is greater than the target running video memory usage, then steps S409 to S411 can be executed; conversely, if the first running video memory usage is less than or equal to the target running video memory usage, then step S413 can be executed.

[0124] In step S409, the target video memory is updated based on the first running video memory usage to obtain the updated first target video memory.

[0125] The updated first target video memory is larger than the target video memory. Specifically, the capacity of the updated first target video memory can be equal to or greater than the first running video memory usage.

[0126] In practice, updating the target video memory can be done by first deleting the previously allocated target video memory, and then using the first running video memory usage as a parameter, calling the dynamic video memory allocation function cudaMalloc() to reallocate video memory to obtain the updated first target video memory with a capacity equal to the first running video memory usage.

[0127] In step S411, the updated first target memory is used as the running memory for multiple target execution contexts and the added target execution context.

[0128] Specifically, when the first running memory usage is greater than the target running memory usage, it indicates that the current running memory capacity cannot meet the running requirements of the added neural network model. At this time, the target memory is updated based on the first running memory usage to obtain an updated first target memory with a capacity equal to or greater than the first running memory usage. This updated first target memory is designated as the running memory shared by the original multiple target execution contexts and the added target execution context, which can ensure that the shared running memory meets the running requirements of each neural network model.

[0129] In specific implementation, the updated pointer information of the first target video memory can be obtained first. Then, for each target execution context, the updated pointer information of the first target video memory is used as a parameter to call the video memory setting function to assign the updated first target video memory as the running video memory (workspace) of each of the multiple target execution contexts. Similarly, for the added target execution context, the updated pointer information of the first target video memory is used as a parameter to call the video memory setting function to assign the updated first target video memory as the running video memory (workspace) of the added target execution context, thereby enabling multiple target execution contexts and the added target execution context to share the running video memory.

[0130] For example, suppose initially, the system can support n neural network models for inference, and the maximum workspace size (i.e., the amount of GPU memory used) is 3GB. A 3GB target GPU memory block is allocated as the shared workspace for the n target execution contexts. If an (n+1)th neural network model is added, the workspace size for its corresponding inference engine is 4GB, which is larger than the target GPU memory capacity of 3GB. In this case, the previously allocated 3GB target GPU memory can be deleted, and a new 4GB GPU memory block can be allocated as the updated first target GPU memory. Then, the GPU memory setting function is called for each of the n+1 target execution contexts to specify the newly allocated 4GB GPU memory.

[0131] The above implementation method enables the expansion of the workspace by adding a neural network model midway, which saves memory usage and further improves operational flexibility.

[0132] In step S413, the target video memory is used as the running video memory for multiple target execution contexts and the target execution context is added.

[0133] Specifically, when the first running video memory usage is less than or equal to the target running video memory usage, it indicates that the current running video memory capacity can meet the running requirements of the added neural network model. At this time, the pointer information of the target video memory can be used as a parameter to call the video memory setting function to specify the added target execution context to the target video memory so that the target video memory can also be used as the running video memory of the added target execution context, thereby realizing that the added target execution context and the original multiple target execution contexts share the workspace.

[0134] Taking the previous example, suppose that an (n+1)th neural network model is added midway through the process. The workspace size of the inference engine corresponding to the (n+1)th neural network model is 2GB, and the target video memory capacity is 3GB. At this time, it is only necessary to call the video memory setting function for the (n+1)th target execution context (i.e., the target execution context of the inference engine corresponding to the (n+1)th neural network model) to specify it to the 3GB target video memory. This allows the (n+1)th target execution context to share the workspace, which not only meets the running requirements of each neural network model, but also further improves the running flexibility.

[0135] In some possible implementations, such as Figure 5 A flowchart of another data processing method is provided. After step S207, the method may further include:

[0136] In step S501, in response to the model reduction request, the largest amount of running memory usage among the remaining inference engines is determined to obtain the remaining target running memory usage.

[0137] The remaining inference engines are those that remain after model reduction processing.

[0138] The model pruning request is used to request the pruning of a target neural network model, which can be any one of multiple neural network models. The model pruning process is used to remove the inference engine corresponding to the target neural network model and the target execution context corresponding to the inference engine.

[0139] For example, suppose initially, n neural network models can be supported for inference, and the maximum workspace size (i.e., the amount of GPU memory used) is 3GB. At this point, a 3GB block of target GPU memory is allocated as the shared GPU memory (workspace) for the execution context of the n targets. If the nth neural network model is requested to be deleted midway through the process, then the remaining n-1 neural network models will have any one of the remaining inference engines.

[0140] Therefore, determining the largest memory usage among the remaining inference engines is equivalent to determining the largest memory usage among the remaining n-1 inference engines.

[0141] In step S503, it is determined whether the remaining target running video memory usage is less than the target running video memory usage.

[0142] Specifically, if the remaining target running video memory usage is less than the target running video memory usage, then steps S505 to S507 can be executed; otherwise, if the remaining target running video memory usage is equal to the target running video memory usage, then the operation ends, that is, no action is performed.

[0143] In step S505, the target video memory is updated based on the remaining target video memory usage to obtain the updated second target video memory.

[0144] The updated second target video memory is smaller than the target video memory. Specifically, the capacity of the updated second target video memory can be equal to or greater than the remaining target running video memory usage.

[0145] In practice, updating the target video memory can be done by first deleting the previously allocated target video memory, and then using the remaining target running video memory usage as a parameter, calling the dynamic video memory allocation function cudaMalloc() to reallocate video memory to obtain an updated second target video memory with a capacity equal to the remaining target running video memory usage.

[0146] In step S507, the updated second target memory is used as the running memory of the target execution context corresponding to the remaining inference engine.

[0147] Specifically, when the remaining target running memory usage is less than the target running memory usage, it indicates that the current running memory capacity is too large. At this time, the target memory is updated based on the remaining target running memory usage to obtain a second target memory with a capacity equal to or greater than the remaining target running memory usage. This updated second target memory is then designated as the running memory shared by the target execution context corresponding to the remaining inference engine. This can further reduce the memory usage while meeting the running requirements of each neural network model.

[0148] In the specific implementation, the pointer information of the updated second target memory can be obtained first. Then, for the target execution context corresponding to each remaining inference engine, the pointer information of the updated second target memory is used as a parameter to call the memory setting function to assign the updated second target memory as the running memory, i.e., workspace, of the target execution context corresponding to each remaining inference engine.

[0149] Taking the example above, suppose the maximum workspace size of the inference engine corresponding to the remaining n-1 neural network models is 2GB, which is less than the target video memory capacity of 3GB. In this case, the previously allocated 3GB target video memory can be deleted, and a new 2GB video memory can be allocated as the updated second target video memory. Then, the video memory setting function is called for each of the n-1 target execution contexts to specify the newly allocated 2GB video memory.

[0150] The above implementation method reduces the workspace size by removing neural network models midway through the process, further saving video memory usage and improving operational flexibility.

[0151] In some possible implementations, such as Figure 6 A flowchart of another data processing method is provided. After step S207, the method may further include:

[0152] In step S601, in response to the first running request for the first neural network model, the inference engine corresponding to the first neural network model is determined to obtain the first inference engine.

[0153] The first neural network model is any one of the multiple neural network models.

[0154] The first run request is used to request the invocation of the first neural network model for data processing. The first run request may be sent by the terminal in response to the user's data processing instruction, and the first run request may carry the data to be processed.

[0155] In specific implementation, when the server receives the first running request for the first neural network model, it responds to the first running request by searching for the inference engine corresponding to the first neural network model from among the multiple inference engines that have been created to obtain the first inference engine.

[0156] In step S603, based on the memory usage of the first inference engine, the first memory region is locked in the target memory.

[0157] The capacity of the first video memory region is the amount of video memory occupied by the first inference engine.

[0158] Specifically, locking the first video memory area prevents other inference engines besides the first inference engine from using the first video memory area.

[0159] In step S605, the first running request is processed based on the first video memory region using the target execution context corresponding to the first inference engine.

[0160] In specific implementation, the target execution context corresponding to the first inference engine runs, performs inference (i.e. prediction processing) based on the data to be processed in the first run request, and uses the first video memory area as a temporary storage space for the execution of each network layer during the inference process.

[0161] Understandably, after the first running request is processed based on the first video memory region, other inference engines that can unlock the first video memory region and share the target video memory can use the first video memory region.

[0162] The above implementation method executes the running request for any neural network model based on the shared target video memory, realizing the reuse of target video memory by multiple neural network models, which greatly saves video memory usage.

[0163] In some possible implementations, in order to improve data processing efficiency, such as Figure 7 A flowchart illustrating another improved data processing method, which may further include:

[0164] In step S701, if a second running request for the second neural network model is obtained during the processing of the first running request, the remaining target video memory capacity is determined.

[0165] The remaining target memory refers to the target memory area excluding the first memory area.

[0166] The second neural network model is any neural network model other than the first neural network model mentioned above among the plurality of neural network models.

[0167] In step S703, the inference engine corresponding to the second neural network model is determined to obtain the second inference engine.

[0168] In step S705, it is determined whether the memory usage of the second inference engine is greater than the memory capacity of the remaining target memory.

[0169] Specifically, if the memory usage of the second inference engine is less than or equal to the memory capacity of the remaining target memory, step S707 can be executed; otherwise, if the memory usage of the second inference engine is greater than the memory capacity of the remaining target memory, it is necessary to wait and process the second running request based on the target memory after the first running request has been processed.

[0170] In step S707, the second run request is processed based on the remaining target memory using the target execution context of the second inference engine.

[0171] For example, suppose there are three neural network models, each with a corresponding inference engine. Their respective GPU memory usage is 3GB, 1GB, and 1GB. The target GPU memory is allocated based on the largest GPU memory usage (3GB) as the shared GPU memory for the target execution contexts of the three inference engines. When the second inference engine's target execution context performs inference, it locks the 0.0–1.0GB region of target GPU memory. If the third inference engine's target execution context needs to perform inference simultaneously, since the 1.0–3.0GB region is not locked, it can select the 1.0–2.0GB region of target GPU memory from that region to perform the inference process. Therefore, the simultaneous execution of the target execution contexts of these two inference engines will not cause data conflicts.

[0172] The above implementation achieves dynamic concurrent scheduling. Each inference engine locks a region in the shared target memory when making a call, while the unlocked region can be called concurrently by other inference engines, thereby improving data processing efficiency.

[0173] Figure 8 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The data processing device 800 includes:

[0174] Engine creation unit 810 is configured to perform the following: for multiple trained neural network models, create an inference engine corresponding to each neural network model.

[0175] The execution context creation unit 820 is configured to create a target execution context corresponding to each inference engine, resulting in multiple target execution contexts; the target execution contexts are not allocated runtime memory.

[0176] The video memory usage determination unit 830 is configured to determine the running video memory usage corresponding to each of the inference engines and select the largest running video memory usage as the target running video memory usage.

[0177] The first running video memory determination unit 840 is configured to determine the target video memory based on the target running video memory usage, and use the target video memory as the running video memory of the plurality of target execution contexts.

[0178] In some possible implementations, the first runtime memory determination unit 840 includes:

[0179] The pointer information acquisition unit is configured to acquire the pointer information of the target video memory;

[0180] The function call unit is configured to execute, for each target execution context, using the pointer information as input parameters to a video memory setting function, and to specify the target video memory as the running video memory of the target execution context by calling the video memory setting function.

[0181] In some possible implementations, the device further includes:

[0182] The first response unit is configured to respond to the model addition request by creating the addition inference engine corresponding to the added neural network model.

[0183] The first creation unit is configured to create the target execution context corresponding to the added inference engine, thereby obtaining the added target execution context;

[0184] The first acquisition unit is configured to acquire the running memory usage corresponding to the added inference engine, and obtain the first running memory usage.

[0185] The first update unit is configured to update the target video memory based on the first video memory usage when the first running video memory usage is greater than the target video memory usage, to obtain an updated first target video memory; the updated first target video memory is greater than the target video memory.

[0186] The second runtime memory determination unit is configured to use the updated first target runtime memory as the runtime memory for the plurality of target execution contexts and the added target execution context.

[0187] In some possible implementations, the device further includes:

[0188] The third runtime memory determination unit is configured to, when the first runtime memory usage is less than or equal to the target runtime memory usage, use the target runtime memory as the runtime memory of the plurality of target execution contexts and the added target execution context.

[0189] In some possible implementations, the device further includes:

[0190] The second response unit is configured to respond to the model reduction request by determining the largest runtime memory usage among the remaining inference engines, and obtaining the remaining target runtime memory usage; the remaining inference engines are the inference engines remaining after the model reduction process.

[0191] The second update unit is configured to update the target video memory based on the remaining target video memory usage when the remaining target video memory usage is less than the target video memory usage, to obtain an updated second target video memory; the updated second target video memory is less than the target video memory.

[0192] The third runtime memory determination unit is configured to use the updated second target runtime memory as the runtime memory of the target execution context corresponding to the remaining inference engine.

[0193] In some possible implementations, the device further includes:

[0194] The third response unit is configured to execute a response to a first running request for the first neural network model, determine the inference engine corresponding to the first neural network model, and obtain the first inference engine; the first neural network model is any one of the plurality of neural network models.

[0195] The video memory locking unit is configured to perform a first video memory region locking in the target video memory based on the running video memory usage corresponding to the first inference engine.

[0196] The first request processing unit is configured to process the first running request based on the first video memory region using the target execution context corresponding to the first inference engine.

[0197] In some possible implementations, the device further includes:

[0198] The remaining target video memory capacity determination unit is configured to determine the remaining target video memory capacity if a second running request for a second neural network model is obtained during the processing of the first running request; the remaining target video memory is the video memory region other than the first video memory region, and the second neural network model is any neural network model other than the first neural network model among the plurality of neural network models.

[0199] The engine determination unit is configured to determine the inference engine corresponding to the second neural network model to obtain the second inference engine.

[0200] The second request processing unit is configured to process the second running request based on the remaining target memory when the running memory usage corresponding to the second inference engine is less than or equal to the memory capacity of the remaining target memory.

[0201] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0202] In some possible implementations, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the data processing method provided in any of the above implementations.

[0203] The electronic device can be a terminal, a server, or a similar computing device. Taking a server as an example... Figure 9 This is a block diagram illustrating an electronic device for data processing according to an exemplary embodiment, such as... Figure 9 As shown, the server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (CPUs 910 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 930 for storing data, and one or more storage media 920 (e.g., one or more mass storage devices) for storing application programs 923 or data 922. The memory 930 and storage media 920 may be temporary or persistent storage. The program stored in the storage media 920 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 910 may be configured to communicate with the storage media 920 and execute the series of instruction operations stored in the storage media 920 on the server 900. Server 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input / output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0204] The input / output interface 940 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 900. In one example, the input / output interface 940 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 940 may be a radio frequency (RF) module for wireless communication with the Internet.

[0205] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 900 may also include... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0206] In some possible implementations, a computer-readable storage medium including instructions is also provided, such as a memory 930 including instructions, which can be executed by a processor 910 of the device 900 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0207] In some possible implementations, a computer program product is also provided, including a computer program that, when executed by a processor, implements the data processing method provided in any of the above implementations.

[0208] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0209] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: For multiple trained neural network models, create an inference engine corresponding to each neural network model; Create a target execution context corresponding to each inference engine to obtain multiple target execution contexts; The target execution context has not been allocated runtime memory; Determine the memory usage of each inference engine and select the largest memory usage as the target memory usage. The target video memory is determined based on the target video memory usage, and the target video memory is used as the video memory of the execution context of the plurality of targets; In response to a model addition request, create the inference engine corresponding to the added neural network model; Create the target execution context corresponding to the added inference engine to obtain the added target execution context; Obtain the memory usage corresponding to the added inference engine to get the first memory usage. If the first running video memory usage is greater than the target running video memory usage, the target video memory is updated based on the first running video memory usage to obtain the updated first target video memory. The updated first target video memory is larger than the target video memory; The updated first target video memory is used as the runtime video memory for the multiple target execution contexts and the added target execution context; When the first running video memory usage is less than or equal to the target running video memory usage, the target video memory is used as the running video memory of the plurality of target execution contexts and the added target execution context; In response to the model reduction request, determine the largest amount of running memory usage among the remaining inference engines to obtain the remaining target running memory usage; The remaining inference engines are the inference engines remaining after model reduction processing; If the remaining target memory usage is less than the target memory usage, the target memory is updated based on the remaining target memory usage to obtain an updated second target memory; the updated second target memory is less than the target memory. The updated second target memory is used as the running memory of the target execution context corresponding to the remaining inference engine.

2. The method according to claim 1, characterized in that, The step of using the target video memory as the runtime video memory of the plurality of target execution contexts includes: Obtain the pointer information of the target video memory; For each target execution context, the pointer information is used as the input parameter of the video memory setting function, and the target video memory is designated as the running video memory of the target execution context by calling the video memory setting function.

3. The method according to claim 1, characterized in that, After using the target video memory as the runtime video memory of the plurality of target execution contexts, the method further includes: In response to a first running request for a first neural network model, the inference engine corresponding to the first neural network model is determined to obtain a first inference engine; the first neural network model is any one of the plurality of trained neural network models; Based on the memory usage of the first inference engine, the first memory region is locked in the target memory. The first running request is processed based on the first video memory region using the target execution context corresponding to the first inference engine.

4. The method according to claim 3, characterized in that, The method further includes: During the processing of the first running request, if a second running request for the second neural network model is obtained, the remaining target video memory capacity is determined; the remaining target video memory is the video memory area other than the first video memory area, and the second neural network model is any neural network model other than the first neural network model among the plurality of trained neural network models. The second inference engine is obtained by determining the inference engine corresponding to the second neural network model. When the memory usage of the second inference engine is less than or equal to the memory capacity of the remaining target memory, the second running request is processed based on the remaining target memory using the target execution context of the second inference engine.

5. A data processing apparatus, characterized in that, include: The engine creation unit is configured to perform the following: for multiple trained neural network models, create an inference engine corresponding to each neural network model. The execution context creation unit is configured to create a target execution context corresponding to each inference engine, thereby obtaining multiple target execution contexts; The target execution context has not been allocated runtime memory; The video memory usage determination unit is configured to determine the running video memory usage corresponding to each of the inference engines, and select the largest running video memory usage as the target running video memory usage. The first running video memory determination unit is configured to determine the target video memory based on the target running video memory usage, and use the target video memory as the running video memory of the plurality of target execution contexts; In response to a model addition request, create the inference engine corresponding to the added neural network model; Create the target execution context corresponding to the added inference engine to obtain the added target execution context; Obtain the memory usage corresponding to the added inference engine to get the first memory usage. If the first running video memory usage is greater than the target running video memory usage, the target video memory is updated based on the first running video memory usage to obtain the updated first target video memory. The updated first target video memory is larger than the target video memory; The updated first target video memory is used as the runtime video memory for the multiple target execution contexts and the added target execution context; When the first running video memory usage is less than or equal to the target running video memory usage, the target video memory is used as the running video memory of the plurality of target execution contexts and the added target execution context; In response to the model reduction request, determine the largest amount of running memory usage among the remaining inference engines to obtain the remaining target running memory usage; The remaining inference engines are the inference engines remaining after model reduction processing; If the remaining target memory usage is less than the target memory usage, the target memory is updated based on the remaining target memory usage to obtain an updated second target memory; the updated second target memory is less than the target memory. The updated second target memory is used as the running memory of the target execution context corresponding to the remaining inference engine.

6. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data processing method as described in any one of claims 1 to 4.

Images