Model migration method and related equipment

Abstract

The present application discloses a model migration method and related equipment to solve the problem in the prior art that when the deep learning model is quantized so that the GPU has resources to perform reasoning tasks, the prediction ability and accuracy of the model are reduced, resulting in the problem that it is not applicable in application scenarios with high accuracy and precision requirements. The method includes: determining the call frequency of the algorithm model in the graphics processing unit being called to perform reasoning tasks; if the call frequency is less than the preset call frequency threshold, the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose call frequency is less than the preset call frequency threshold are migrated to the CPU and memory, and the thread of the idle algorithm model is put into a dormant state; when a call request of the idle algorithm model is received, the dormant thread is awakened, and the hierarchical structure information and model parameter information of the idle algorithm model are migrated from the CPU and memory to the graphics processing unit.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model migration method and related equipment. Background Art

[0002] In the training and reasoning process of deep learning models, the Graphics Processing Unit (GPU) has become the core hardware device supporting large-scale computing due to its powerful parallel computing capabilities.

[0003] At present, with the continuous expansion of the parameter scale of deep learning models, the GPU computing resources and memory resources occupied by a single deep learning model have increased dramatically; at the same time, as the application scenarios of artificial intelligence are becoming more and more extensive, involving natural language processing, image recognition, speech processing, intelligent recommendation and other fields, and the deep learning models in these application scenarios each have unique functional requirements, it is necessary to deploy and manage multiple different models in parallel on the GPU. These reasons ultimately lead to the limited GPU being unable to carry the reasoning tasks that exceed the load.

[0004] In the prior art, in order to address the problem that the GPU cannot handle inference tasks that exceed the load, it is proposed to quantize the deep learning model deployed in the GPU. For example, by reducing the accuracy of model parameters, reducing the model hidden layers, simplifying the neural network structure, etc., the deep learning model can be "slimmed down" so that the GPU can have remaining resources for inference tasks.

[0005] However, the existing technology of quantizing deep learning models will lead to a decrease in the predictive ability of deep learning models, which is not suitable for application scenarios with high accuracy and precision requirements.

[0006] Based on this, how to manage the deep learning model deployed in the GPU to ensure that the GPU has remaining resources for reasoning tasks is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0007] An embodiment of the present application provides a model migration method to solve the problem in the prior art that when the deep learning model is quantized so that the GPU has resources to perform reasoning tasks, the prediction ability and accuracy of the model are reduced, resulting in it being unsuitable for application scenarios requiring high accuracy and precision.

[0008] Embodiments of the present application also provide a model migration device, an electronic device, a computer-readable storage medium, and a computer program product.

[0009] The present application embodiment adopts the following technical solutions:

[0010] In a first aspect, the present application provides a model migration method, comprising:

[0011] Determine the calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task;

[0012] If the calling frequency is less than a preset calling frequency threshold, the hierarchical structure information of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold and the model parameter information related to the reasoning task are migrated to the CPU and the memory, and the thread of the idle algorithm model is put into a dormant state;

[0013] When a call request of the idle algorithm model is received, the thread in the dormant state is awakened, and the hierarchical structure information and the model parameter information of the idle algorithm model are migrated from the CPU and the memory to the graphics processing unit.

[0014] Optionally, the hierarchical structure information of the idle algorithm model whose calling frequency is less than a preset calling frequency threshold and the model parameter information related to the reasoning task are migrated to the CPU and memory, and the thread of the idle algorithm model is put into a dormant state, including:

[0015] The current running state of the idle algorithm model is set to a pending transfer state, and the thread of the idle algorithm model enters a dormant state;

[0016] Migrate the idle algorithm model whose calling frequency is less than the preset calling frequency threshold to the CPU and memory.

[0017] Optionally, the method further includes:

[0018] When the idle algorithm model is migrated to the CPU and memory, the current running state of the idle algorithm model is reset to the migrated state.

[0019] Optionally, when a call request of the idle algorithm model is received, the thread in the dormant state is awakened, and the hierarchical structure information and model parameter information of the idle algorithm model are migrated from the CPU and the memory to the graphics processing unit, including:

[0020] When receiving a call request of the idle algorithm model, the thread in the dormant state is awakened, and the current running state of the idle algorithm model is reset to being executed;

[0021] The idle algorithm model is then migrated from the CPU and memory to the graphics processing unit.

[0022] Optionally, before waking up the dormant thread and migrating the hierarchical structure information and model parameter information of the idle algorithm model from the CPU and the memory to the graphics processing unit, the method further includes:

[0023] Determine whether the last running state of the idle algorithm model is a transferred state, wherein the last running state refers to the running state of the idle algorithm model at the historical moment closest to the current moment;

[0024] If it is in the transferred state, determining whether the current resources of the graphics processing unit meet the resource requirement conditions for the idle algorithm model to execute the reasoning task;

[0025] If the current resources do not meet the resource requirement conditions, an abnormal prompt message for prompting that the graphics processing unit resources are insufficient is sent.

[0026] Optionally, the method further includes:

[0027] When it is detected that the main thread registers the algorithm model to the model registration machine, the algorithm model identifier of the algorithm model is sent to the main thread;

[0028] Initialize the running state of the algorithm model to executing.

[0029] In a second aspect, the present application provides a model migration device, including a determination module, a first migration module and a second migration module, wherein:

[0030] A determination module, used to determine the calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task;

[0031] A first migration module is used to migrate the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold to the CPU and memory, and put the thread of the idle algorithm model into a dormant state if the calling frequency is less than the preset calling frequency threshold;

[0032] The second migration module, when receiving a call request of the idle algorithm model, wakes up the thread in the dormant state, and migrates the hierarchical structure information and model parameter information of the idle algorithm model from the CPU and the memory to the graphics processing unit.

[0033] Optionally, the first migration module is used to:

[0034] The current running state of the idle algorithm model is set to a pending transfer state, and the thread of the idle algorithm model enters a dormant state;

[0035] Migrate the idle algorithm model whose calling frequency is less than the preset calling frequency threshold to the CPU and memory.

[0036] Optionally, the device is also used for:

[0037] When the idle algorithm model is migrated to the CPU and memory, the current running state of the idle algorithm model is reset to the migrated state.

[0038] Optionally, a second migration module is used to:

[0039] When receiving a call request of the idle algorithm model, the thread in the dormant state is awakened, and the current running state of the idle algorithm model is reset to being executed;

[0040] The idle algorithm model is then migrated from the CPU and memory to the graphics processing unit.

[0041] Optionally, the device is also used for:

[0042] Determine whether the last running state of the idle algorithm model is a transferred state, wherein the last running state refers to the running state of the idle algorithm model at the historical moment closest to the current moment;

[0043] If it is in the transferred state, determining whether the current resources of the graphics processing unit meet the resource requirement conditions for the idle algorithm model to execute the reasoning task;

[0044] If the current resources do not meet the resource requirement conditions, an abnormal prompt message for prompting that the graphics processing unit resources are insufficient is sent.

[0045] Optionally, the device is also used for:

[0046] When it is detected that the main thread registers the algorithm model to the model registration machine, the algorithm model identifier of the algorithm model is sent to the main thread;

[0047] Initialize the running state of the algorithm model to executing.

[0048] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the model migration method as described above.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the model migration method as described above are implemented.

[0050] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the model migration method as described above when executed by a processor.

[0051] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0052] By adopting the method provided in the embodiment of the present application, the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold can be migrated to the CPU and memory, and the thread of the idle algorithm model can be put into a dormant state until the idle algorithm model is requested to be called again to perform the reasoning task, and then the dormant thread is awakened, and the hierarchical structure information and model parameter information of the idle algorithm model are re-migrated to the graphics processing unit. In this way, the resources of the graphics processing unit occupied by the idle algorithm model can be released in time so that the graphics processing unit can have remaining resources to perform the reasoning task. In addition, by adopting the method provided in the embodiment of the present application, since it is not necessary to "slim down" the deep learning model, while ensuring that the graphics processing unit can have remaining resources to perform the reasoning task, the prediction ability of the deep learning model can also be guaranteed, which can be suitable for application scenarios with high accuracy and precision requirements.

[0053] In addition, when migrating the hierarchical information and model parameter information of the idle algorithm model in the present application, the thread of the idle algorithm model can enter a dormant state. In this way, when the idle algorithm model is subsequently requested to be called again to perform an inference task, it can immediately respond to the call request, thereby ensuring the processing efficiency of the inference task. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0055] Figure 1 A schematic diagram of the implementation flow of a model migration method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of the specific structure of a migration device of a model is provided for the embodiment of the present application;

[0057] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0059] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0060] Example 1

[0061] In order to solve the problem in the prior art that when the deep learning model is quantized so that the GPU has resources to perform inference tasks, the prediction ability and accuracy of the model are reduced, resulting in it being unsuitable for application scenarios requiring high accuracy and precision, an embodiment of the present application provides a model migration method.

[0062] The execution subject of the method may be various types of computing devices, or an application or application (Application, APP) installed on the computing device. The computing device may be, for example, a user terminal such as a mobile phone, a tablet computer, a smart wearable device, or a server.

[0063] For ease of description, the present application embodiment takes the execution subject of the method as an example of a server to introduce the method. Those skilled in the art can understand that the present application embodiment takes the server as an example to introduce the method, which is only an exemplary description and does not limit the scope of protection of the claims corresponding to the present solution.

[0064] Specifically, the implementation process of the method provided in the embodiment of the present application is as follows: Figure 1 As shown, the following steps are included:

[0065] Step 11, determining the calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task.

[0066] An algorithmic model refers to a deep learning model or machine learning model used to perform specific tasks. For example, in the insurance business, the algorithmic model may include: a deep learning model or machine learning model used to assess customer risk, a deep learning model or machine learning model used to analyze patterns and abnormal behaviors in insurance claims data and predict whether a claim application is suspected of fraud, or a deep learning model or machine learning model used to perform regression analysis of large-scale historical data and predict the probability of payment under specific insurance terms, etc.

[0067] It should be noted that the algorithm model cited above is only an exemplary description of the embodiment of the present application and does not impose any limitation on the embodiment of the present application.

[0068] The call frequency refers to the frequency at which the algorithm model is triggered in the graphics processing unit (GPU) to perform reasoning tasks, or it can also be understood as the number of times the algorithm model is triggered and executed per unit time.

[0069] For example, in the insurance business scenario, suppose there is a GPU-based deep learning model, and the deep learning model is integrated into the insurance company's customer relationship management system. When a customer submits an insurance application, the customer relationship management system will automatically trigger the deep learning model to perform risk assessment and provide personalized insurance quotes based on the assessment results. Based on this, if the deep learning model is triggered to process 1,000 new insurance applications every day, and only one prediction analysis is performed for each new insurance application, then it can be considered that the deep learning model is called 1,000 times per day.

[0070] Considering that in the related art, when determining the calling frequency of the algorithm model being called to perform reasoning tasks, it is usually directly observed that the algorithm model is called within a preset time period, and then the calling frequency is determined based on the number of calls and the preset time period. However, the calling frequency determined in this way is prone to deviation. For example, suppose that two deep learning algorithm models A and B are deployed in a background system for automatically reviewing insurance policies and claims applications. The actual calling frequency of model A is once every 5 minutes, while the actual calling frequency of model B is once every 10 minutes. At this time, if the monitoring time period is only 6 minutes, the monitoring results will show that model A has been called once, while model B has not been called, resulting in the calling frequency of model B being underestimated. In order to avoid this problem, when determining the calling frequency of the algorithm model in the graphics processing unit being called to perform reasoning tasks, it is crucial to reasonably set the monitoring time period to ensure accurate evaluation of the calling frequency of all algorithm models and avoid statistical deviations caused by too short a time period.

[0071] Based on the above problems, in an optional implementation, when determining the calling frequency of the algorithm model in the graphics processing unit to perform reasoning tasks, the specified monitoring period can be determined first; then, the number of times the algorithm model in the graphics processing unit is called to perform reasoning tasks within the specified monitoring period can be determined; finally, the calling frequency of the algorithm model in the graphics processing unit to perform reasoning tasks is determined based on the number of calls and the specified monitoring period.

[0072] Optionally, when determining the specified monitoring period, for example, the following methods may be considered:

[0073] (1) Based on the least common multiple: that is, the least common multiple of all model call intervals is selected as the monitoring period. This ensures that each model is called at least once during the monitoring period. For example, model A is called once every 5 minutes and model B is called once every 10 minutes. Their least common multiple is 10 minutes. Therefore, selecting 10 minutes as the monitoring period ensures that the call frequency of both models is accurately recorded.

[0074] (2) Based on business needs: If the business has specific real-time requirements, the monitoring time period can be determined based on the response time requirements for inference tasks in the business scenario. For example, if the system requires statistical results to be provided every 10 minutes, then choosing 10 minutes as the monitoring time period also meets business needs and can take into account the calling frequency of multiple models.

[0075] (3) Dynamically adjust the monitoring time period: In some complex application scenarios, the monitoring time period can be dynamically adjusted. That is, the monitoring time period is adjusted in real time according to the current business load and the calling frequency of each model. For example, when the system detects that some models are called frequently, the monitoring time period is automatically shortened to allow more frequent statistics on the model calls; when the calling frequency is low, the monitoring time period is appropriately extended to obtain more accurate statistical data.

[0076] (4) Based on historical data analysis: By analyzing historical call data, the model call pattern can be predicted, thereby determining a reasonable monitoring period. For example, by analyzing the call frequency of each model over a period of time, a monitoring period that is neither too long nor too short can be selected to balance real-time performance and accuracy.

[0077] In an optional implementation, when determining the calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task, the following method may also be used:

[0078] (1) Obtain information such as the historical call frequency of the algorithm model in the graphics processing unit, the call load fluctuation rate, the priority of the processed tasks, the complexity of the algorithm model, the diversity of the data input dimensions of the algorithm model, the task asynchrony coefficient of the algorithm model, the uneven distribution of the inference time of the algorithm model, as well as the business real-time requirements and data loading frequency.

[0079] Among them, the historical call frequency of the algorithm model, that is, the average number of calls of the algorithm model in a specific time period;

[0080] Business real-time requirements and data loading frequency can be set according to actual needs.

[0081] The call load volatility of the algorithm model is used to characterize the fluctuation of the call frequency of the algorithm model;

[0082] The complexity of the algorithm model, such as the number of layers, number of parameters or computational complexity in the neural network, can be determined according to actual needs.

[0083] The diversity of data input dimensions is used to reflect the types and complexity of data processed by the algorithm model.

[0084] (2) Based on the historical call frequency of the algorithm model in the graphics processing unit obtained in the above step (1), the call load fluctuation rate, the priority of the processed task, the complexity of the algorithm model, the diversity of the data input dimensions of the algorithm model, the task asynchrony coefficient of the algorithm model, the uneven distribution of the inference time of the algorithm model, as well as the business real-time requirements and data loading frequency, etc., the designated monitoring period is determined according to the following calculation formula:

[0085]

[0086] in, Indicates the specified monitoring period; Indicates the historical call frequency of the algorithm model, that is, the average number of calls of the algorithm model in a specific time period; Indicates the real-time requirements of the business; Indicates the call load volatility of the algorithm model, which is used to characterize the fluctuation of the call frequency of the algorithm model; Indicates the priority of the task processed by the algorithm model.

[0087] The complexity of the algorithm model, such as the number of layers, parameters or computational complexity in a neural network, can be determined based on actual needs.

[0088] Indicates the diversity of data input dimensions, which is used to reflect the types and complexity of data processed by the algorithm model. Generally, multi-dimensional data input may cause the model to call unstable frequencies in different time periods, so a longer time period is required for monitoring.

[0089] Represents the task asynchrony coefficient of the algorithm model. Since the inference delay in the asynchronous tasks of the algorithm model will affect the real-time call frequency, the asynchrony coefficient is introduced into the formula. The stronger the asynchrony, the longer the monitoring period should be.

[0090] Indicates the uneven distribution of the algorithm model's reasoning time. Generally, if the algorithm model's reasoning tasks are highly concentrated in certain time periods, the monitoring time period should be adjusted accordingly to capture more task execution situations.

[0091] Indicates the frequency of data loading; usually, frequent data loading may affect the frequency of algorithm model calls. If data loading is too frequent, the monitoring period should be shortened to capture the impact of data loading on model calls. , , , , . , , and These are all preset weight coefficients.

[0092] (3) determining the number of times the algorithm model in the graphics processing unit is called to perform the inference task within a specified monitoring period;

[0093] (4) Determine the frequency with which the algorithm model in the graphics processing unit is called to perform the inference task based on the number of calls and the specified monitoring period.

[0094] In an optional implementation, before determining the calling frequency of the algorithm model in the graphics processing unit to be called to perform the inference task, each algorithm model may be registered in the model registration machine. Specifically, the "model registration machine" may be started, and then the model registration requests of each algorithm model from different threads may be received, so that each algorithm model may be registered in the model registration machine based on the model registration requests of each algorithm model.

[0095] Optionally, in order to achieve the independence of the registration process and the scheduling process of the algorithm model, in the embodiment of the present application, an asynchronous thread can be used to start the model registration machine. Among them, the model registration machine is responsible for receiving model registration requests and model scheduling requests from different threads, so as to achieve the independent operation of the model scheduling management work and the main thread. In this way, the model registration process and the scheduling process can be carried out in parallel without affecting the main thread, ensuring that the system runs more efficiently and flexibly.

[0096] Optionally, in an embodiment of the present application, when it is monitored that the main thread registers the algorithm model to the model registration machine, the algorithm model identifier of the algorithm model can be sent to the main thread; and the running state of the algorithm model can be initialized to executing.

[0097] Step 12, if the call frequency is less than the preset call frequency threshold, the hierarchical structure information of the idle algorithm model whose call frequency is less than the preset call frequency threshold and the model parameter information related to the reasoning task are migrated to the CPU and memory, and the thread of the idle algorithm model is put into a dormant state.

[0098] The model parameter information related to the reasoning task includes the weight of the idle algorithm model, which is also the connection parameter between neurons in the idle algorithm model.

[0099] The hierarchical structure information of the idle algorithm model includes the overall structure layout information (or also called network topology structure information) of different layers (including input layer, hidden layer and output layer) in the idle algorithm model.

[0100] In the embodiments of the present application, the CPU and memory mentioned refer to different hardware components in the same device as the above-mentioned GPU.

[0101] Among them, the thread of the idle algorithm model is put into a dormant state, that is, the reasoning task processing thread of the idle algorithm model is temporarily suspended until there is a new reasoning request related to the idle algorithm model again, and then the reasoning task processing thread of the idle algorithm model is reactivated.

[0102] In an embodiment of the present application, the current running state of the idle algorithm model can be set to a pending transfer state, and the thread of the idle algorithm model can be put into a sleep state; then the idle algorithm model whose call frequency is less than a preset call frequency threshold is migrated to the CPU and memory.

[0103] For example, assuming that the preset call frequency threshold is 100 times / minute, and the actual call frequency of an algorithm model A is 10 times / minute, or the actual call frequency is 0 (that is, it has not been called), then it is considered that the call frequency of the algorithm model A is less than the preset call frequency threshold, and then the hierarchical structure information of the algorithm model A and the model parameter information related to the reasoning task are migrated to the CPU and memory, and the thread of the algorithm model A is put into a dormant state.

[0104] Optionally, when the idle algorithm model is migrated to the CPU and memory, the current running state of the idle algorithm model may be reset to a migrated state.

[0105] In the embodiment of the present application, considering that the setting of the preset call frequency threshold directly determines when the idle model is migrated from the GPU to the CPU and memory and enters the dormant state, the value of the reasonable preset call frequency threshold will directly affect the efficiency of resource scheduling and the responsiveness of the model. If the threshold is set too large, the residence time of the idle algorithm model on the GPU will be extended, which may cause the GPU resources to be occupied for a long time and unable to free up resources for other algorithm models or tasks, thereby affecting the resource utilization efficiency of the system; and if the threshold is set too small, the idle algorithm model may be migrated after being idle for a slightly longer time, increasing the frequency of algorithm model migration, thereby increasing the migration overhead and thread wake-up delay, thereby affecting the response time of the reasoning task and system performance.

[0106] In summary, the setting of preset call frequency threshold is not only the core of resource management, but also the key to balancing system performance and resource utilization. It needs to be dynamically adjusted in combination with various system operation states and algorithm model characteristics to ensure that different algorithm models can effectively and efficiently utilize system resources in different application scenarios. Especially in high-concurrency and heterogeneous resource environments, the dynamic and flexibility of this threshold is particularly important.

[0107] In order to solve the above problem, in an optional implementation, before executing step 12, the preset call frequency threshold may be determined in the following manner:

[0108] in, Indicates the preset call frequency threshold; Indicates GPU utilization; Indicates the maximum load limit of the GPU; Indicates CPU memory utilization; Indicates the maximum memory load limit; Represents the importance weight of the algorithm model reasoning result, which is used to adjust the threshold to match the real-time requirements of the task; Represents the historical usage frequency trend of the algorithm model. The Sigmoid function can be used to smooth the impact of the frequency trend on the threshold. Indicates the hit rate of the algorithm model's data in the cache, which is used to measure the data access efficiency of the algorithm model during its GPU residency period; Indicates the memory bandwidth pressure of the GPU, indicating the current tightness of the GPU's memory resources; Indicates the maximum value of GPU memory bandwidth pressure; Indicates the number of thread context switches of the algorithm model, which is used to reflect the scheduling overhead of the algorithm model in a multi-tasking environment; Indicates the maximum number of thread context switches that the system can withstand. a , b, c, d, e, f They are all weight coefficients, which are used to balance the impact of the above different features on the preset call frequency threshold.

[0109] Step 13, when a call request of the idle algorithm model is received, the thread in the dormant state is awakened, and the hierarchical structure information and model parameter information of the idle algorithm model are migrated from the CPU and the memory to the graphics processing unit.

[0110] In an embodiment of the present application, when a call request for an idle algorithm model is received, a thread in a dormant state can be awakened, and the current running state of the idle algorithm model can be reset to executing; then, the idle algorithm model can be migrated from the CPU and memory to the graphics processing unit.

[0111] Optionally, before migrating the hierarchical structure information and model parameter information of the idle algorithm model from the CPU and the memory to the graphics processing unit, the method provided in the embodiment of the present application further includes:

[0112] Determine whether the last running state of the idle algorithm model is a transferred state, wherein the last running state refers to the running state of the idle algorithm model at the historical moment closest to the current moment;

[0113] If it is in the transferred state, determining whether the current resources of the graphics processing unit meet the resource requirement conditions for the idle algorithm model to execute the reasoning task;

[0114] If the current resources do not meet the resource requirement conditions, an abnormal prompt message for prompting that the graphics processing unit resources are insufficient is sent.

[0115] By adopting the method provided in the embodiment of the present application, the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold can be migrated to the CPU and memory, and the thread of the idle algorithm model can be put into a dormant state until the idle algorithm model is requested to be called again to perform the reasoning task, and then the dormant thread is awakened, and the hierarchical structure information and model parameter information of the idle algorithm model are re-migrated to the graphics processing unit. In this way, the resources of the graphics processing unit occupied by the idle algorithm model can be released in time so that the graphics processing unit can have remaining resources to perform the reasoning task. In addition, by adopting the method provided in the embodiment of the present application, since it is not necessary to "slim down" the deep learning model, while ensuring that the graphics processing unit can have remaining resources to perform the reasoning task, the prediction ability of the deep learning model can also be guaranteed, which can be suitable for application scenarios with high accuracy and precision requirements.

[0116] In addition, when migrating the hierarchical information and model parameter information of the idle algorithm model in the present application, the thread of the idle algorithm model can enter a dormant state. In this way, when the idle algorithm model is subsequently requested to be called again to perform an inference task, it can immediately respond to the call request, thereby ensuring the processing efficiency of the inference task.

[0117] Example 2

[0118] The following describes how the method provided in the embodiment of the present application is applied in practice in combination with actual scenarios.

[0119] Please refer to the following content, which is a schematic diagram of an application process of the method provided in the embodiment of the present application in practice. The process specifically includes the following steps:

[0120] (1) Start the model registration machine in an asynchronous thread mode and receive model registration requests and model scheduling requests from algorithm models in different threads;

[0121] (2) When receiving model registration requests from algorithm models of different threads, register the required N (N≥1) algorithm models to the model registration machine;

[0122] At the same time, after registering each algorithm model, the model registration machine will return the corresponding model identification ID to each thread, initialize the state of the corresponding identification ID model to "executing", and start an independent timing task for each algorithm model.

[0123] The timing task mentioned here is to monitor / determine the number of times the algorithm model in the graphics processing unit is called to perform the reasoning task within a specified monitoring period.

[0124] (3) When a model call request is received from an algorithm model of a different thread, the model identification ID is submitted to the model registration machine to initiate the inference calculation, and the corresponding algorithm model status is reset to "executing".

[0125] It should be noted that when a model call request is received from an algorithm model of a different thread, if the previous state of the algorithm model corresponding to the model call request is "transferred" (that is, the hierarchical information and model parameter information of the algorithm model corresponding to the model call request have been transferred from the GPU to the CPU and memory area), and the current GPU resources are sufficient, the algorithm model in the main thread is woken up and the algorithm model corresponding to the model call request is migrated back to the GPU.

[0126] Alternatively, if the previous state of the algorithm model corresponding to the model call request is "transferred", but the current GPU resources are insufficient, an exception is thrown and insufficient resource information is returned, and then the algorithm model state is restored to "transferred".

[0127] Secondly, the above steps (2) and (3) are two steps executed in parallel. Here, step (2) is described first, and step (3) is described later. This is only for the convenience of description and does not limit the actual execution order of the two steps.

[0128] (4) After each round of timing cycle for each algorithm model ends, check the algorithm models that have not been called in this round. If the current status is "executing", update the status to "pending transfer"; if the current status is "pending transfer", update the status to "transferred".

[0129] (5) After each round of timing tasks for each model is completed, check the running status of the algorithm model that has not been called or has a low calling frequency in this round of timing tasks; if the current status of the algorithm model is "executing", update the status of the algorithm model to "pending transfer"; or, if the current status of the algorithm model is "pending transfer", update the running status of the algorithm model to "transferred". At the same time, notify each main thread to transfer the hierarchical structure information and model parameter information of the algorithm model with the status of "transferred" from the GPU to the CPU and memory area, until the call request of the algorithm model is received again, wake up the thread of the algorithm model in the dormant state, and migrate the hierarchical structure information and model parameter information of the algorithm model from the CPU and memory to the graphics processing unit.

[0130] By adopting the method provided in the embodiment of the present application, the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold can be migrated to the CPU and memory, and the thread of the idle algorithm model can be put into a dormant state until the idle algorithm model is requested to be called again to perform the reasoning task, and then the dormant thread is awakened, and the hierarchical structure information and model parameter information of the idle algorithm model are re-migrated to the graphics processing unit. In this way, the resources of the graphics processing unit occupied by the idle algorithm model can be released in time so that the graphics processing unit can have remaining resources to perform the reasoning task. In addition, by adopting the method provided in the embodiment of the present application, since it is not necessary to "slim down" the deep learning model, while ensuring that the graphics processing unit can have remaining resources to perform the reasoning task, the prediction ability of the deep learning model can also be guaranteed, which can be suitable for application scenarios with high accuracy and precision requirements.

[0131] In addition, when migrating the hierarchical information and model parameter information of the idle algorithm model in the present application, the thread of the idle algorithm model can enter a dormant state. In this way, when the idle algorithm model is subsequently requested to be called again to perform an inference task, it can immediately respond to the call request, thereby ensuring the processing efficiency of the inference task.

[0132] Example 3

[0133] In order to solve the problem that in the prior art, when the deep learning model is quantized so that the GPU has resources to perform reasoning tasks, the prediction ability and accuracy of the model are reduced, resulting in the model being unsuitable for application scenarios requiring high accuracy and precision, the embodiment of the present application provides a model migration device, the specific structural diagram of the device is as follows Figure 2 As shown, it includes a determination module 21, a first migration module 22 and a second migration module 23. The functions of each module are as follows:

[0134] A determination module 21, used to determine the calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task;

[0135] A first migration module 22 is used to migrate the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold to the CPU and memory, and put the thread of the idle algorithm model into a dormant state if the calling frequency is less than the preset calling frequency threshold;

[0136] The second migration module 23, when receiving a call request of the idle algorithm model, wakes up the thread in the dormant state, and migrates the hierarchical structure information and model parameter information of the idle algorithm model from the CPU and the memory to the graphics processing unit.

[0137] Optionally, the first migration module 22 is used to:

[0138] The current running state of the idle algorithm model is set to a pending transfer state, and the thread of the idle algorithm model enters a dormant state;

[0139] Migrate the idle algorithm model whose calling frequency is less than the preset calling frequency threshold to the CPU and memory.

[0140] Optionally, the device is also used for:

[0141] When the idle algorithm model is migrated to the CPU and memory, the current running state of the idle algorithm model is reset to the migrated state.

[0142] Optionally, the second migration module 23 is used to:

[0143] When receiving a call request of the idle algorithm model, the thread in the dormant state is awakened, and the current running state of the idle algorithm model is reset to being executed;

[0144] The idle algorithm model is then migrated from the CPU and memory to the graphics processing unit.

[0145] Optionally, the device is also used for:

[0146] Determine whether the last running state of the idle algorithm model is a transferred state, wherein the last running state refers to the running state of the idle algorithm model at the historical moment closest to the current moment;

[0147] If it is in the transferred state, determining whether the current resources of the graphics processing unit meet the resource requirement conditions for the idle algorithm model to execute the reasoning task;

[0148] If the current resources do not meet the resource requirement conditions, an abnormal prompt message for prompting that the graphics processing unit resources are insufficient is sent.

[0149] Optionally, the device is also used for:

[0150] When it is detected that the main thread registers the algorithm model to the model registration machine, the algorithm model identifier of the algorithm model is sent to the main thread;

[0151] Initialize the running state of the algorithm model to executing.

[0152] By using the device provided in the embodiment of the present application, the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold can be migrated to the CPU and memory, and the thread of the idle algorithm model can be put into a dormant state until the idle algorithm model is requested to be called again to perform the reasoning task, and then the dormant thread is awakened, and the hierarchical structure information and model parameter information of the idle algorithm model are migrated back to the graphics processing unit. In this way, the resources of the graphics processing unit occupied by the idle algorithm model can be released in time so that the graphics processing unit can have remaining resources to perform the reasoning task. In addition, by using the device provided in the embodiment of the present application, since it is not necessary to "slim down" the deep learning model, while ensuring that the graphics processing unit can have remaining resources to perform the reasoning task, the prediction ability of the deep learning model can also be guaranteed, and it can be suitable for application scenarios with high accuracy and precision requirements.

[0153] In addition, when migrating the hierarchical information and model parameter information of the idle algorithm model in the present application, the thread of the idle algorithm model can enter a dormant state. In this way, when the idle algorithm model is subsequently requested to be called again to perform an inference task, it can immediately respond to the call request, thereby ensuring the processing efficiency of the inference task.

[0154] Example 4

[0155] Figure 3 A hardware structure diagram of an electronic device for implementing various embodiments of the present application, the electronic device may include a processor 301 and a memory 302 storing computer program instructions. Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0156] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 302 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 302 may be inside or outside the electronic device. In a particular embodiment, the memory 302 may be a non-volatile solid-state memory.

[0157] In one embodiment, the memory 302 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0158] The processor 301 implements any one of the model migration methods in the above embodiments by reading and executing computer program instructions stored in the memory 302 .

[0159] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0160] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0161] The bus 310 includes hardware, software or both, coupling the components of the electronic device to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnect, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, the bus 310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnect.

[0162] In addition, in combination with the migration method of the model in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the migration of any one of the models in the above embodiment is implemented.

[0163] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0164] The above is only a specific implementation example of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, module and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A model migration method, characterized in that: include: Determine the calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task; If the calling frequency is less than a preset calling frequency threshold, the hierarchical structure information of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold and the model parameter information related to the reasoning task are migrated to the CPU and the memory, and the thread of the idle algorithm model is put into a dormant state; When receiving a call request of the idle algorithm model, waking up the thread in a dormant state, and migrating the hierarchical structure information and the model parameter information of the idle algorithm model from the CPU and the memory to the graphics processing unit; The step of determining the calling frequency of the algorithm model in the graphics processing unit being called to perform the inference task includes: Obtain the historical call frequency of the algorithm model in the graphics processing unit, the call load fluctuation rate, the priority of the processed tasks, the complexity of the algorithm model, the diversity of the data input dimensions of the algorithm model, the task asynchrony coefficient of the algorithm model, the uneven distribution of the inference time of the algorithm model, as well as the business real-time requirements and data loading frequency; Determine the designated monitoring period based on the historical call frequency of the algorithm model in the acquired graphics processing unit, the call load fluctuation rate, the priority of the processed task, the complexity of the algorithm model, the diversity of the data input dimensions of the algorithm model, the task asynchrony coefficient of the algorithm model, the uneven distribution of the inference time of the algorithm model, as well as the business real-time requirements and the data loading frequency; Determine the number of times an algorithm model in a graphics processing unit is called to perform an inference task within a specified monitoring period; The calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task is determined according to the number of calls and the specified monitoring period.

2. The method according to claim 1, characterized in that Migrating the hierarchical structure information of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold and the model parameter information related to the reasoning task to the CPU and the memory, and making the thread of the idle algorithm model enter a dormant state, including: Setting the current running state of the idle algorithm model to a pending transfer state, and making the thread of the idle algorithm model enter a dormant state; The idle algorithm model whose calling frequency is less than the preset calling frequency threshold is migrated to the CPU and the memory.

3. The method according to claim 2, characterized in that The method further comprises: When the idle algorithm model is migrated to the CPU and the memory, the current running state of the idle algorithm model is reset to a transferred state.

4. The method according to claim 1, characterized in that When a call request of the idle algorithm model is received, the thread in a dormant state is awakened, and the hierarchical structure information and the model parameter information of the idle algorithm model are migrated from the CPU and the memory to the graphics processing unit, including: When receiving a call request of the idle algorithm model, waking up the thread in a dormant state, and resetting the current running state of the idle algorithm model to being executed; The idle algorithm model is then migrated from the CPU and the memory to the graphics processing unit.

5. The method according to claim 1 or 4, characterized in that Before waking up the thread in the dormant state and migrating the hierarchical structure information and the model parameter information of the idle algorithm model from the CPU and the memory to the graphics processing unit, the method further includes: Determine whether the last running state of the idle algorithm model is a transferred state, wherein the last running state refers to the running state of the idle algorithm model at the historical moment closest to the current moment; If it is in the transferred state, determining whether the current resources of the graphics processing unit meet the resource requirement conditions for the idle algorithm model to execute the reasoning task; If the current resources do not meet the resource requirement condition, an abnormal prompt message for prompting that the graphics processing unit has insufficient resources is sent.

6. The method according to claim 1, characterized in that The method further comprises: When it is detected that the main thread registers the algorithm model to the model registration machine, the algorithm model identifier of the algorithm model is sent to the main thread; Initialize the running state of the algorithm model to being executed.

7. A model migration device, characterized in that: It includes a determination module, a first migration module and a second migration module, wherein: A determination module, used to determine the calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task; A first migration module is used for migrating the hierarchical structure information and model parameter information related to the reasoning task of the idle algorithm model whose calling frequency is less than the preset calling frequency threshold to the CPU and memory, and putting the thread of the idle algorithm model into a dormant state if the calling frequency is less than the preset calling frequency threshold; a second migration module, which, when receiving a call request of the idle algorithm model, wakes up the thread in a dormant state, and migrates the hierarchical structure information and the model parameter information of the idle algorithm model from the CPU and the memory to the graphics processing unit; Among them, the determination module is used to: Obtain the historical call frequency of the algorithm model in the graphics processing unit, the call load fluctuation rate, the priority of the processed tasks, the complexity of the algorithm model, the diversity of the data input dimensions of the algorithm model, the task asynchrony coefficient of the algorithm model, the uneven distribution of the inference time of the algorithm model, as well as the business real-time requirements and data loading frequency; Determine the designated monitoring period based on the historical call frequency of the algorithm model in the acquired graphics processing unit, the call load fluctuation rate, the priority of the processed task, the complexity of the algorithm model, the diversity of the data input dimensions of the algorithm model, the task asynchrony coefficient of the algorithm model, the uneven distribution of the inference time of the algorithm model, as well as the business real-time requirements and the data loading frequency; Determine the number of times an algorithm model in a graphics processing unit is called to perform an inference task within a specified monitoring period; The calling frequency of the algorithm model in the graphics processing unit being called to perform the reasoning task is determined according to the number of calls and the specified monitoring period.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the model migration method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the model migration method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the model migration method described in any one of claims 1 to 6.

Images