Inference model deployment system and inference model deployment method

Abstract

The present disclosure provides an inference model deployment system and an inference model deployment method. The method includes: obtaining an estimated resource usage of each of a plurality of model settings of an inference model; obtaining a capacity requirement; selecting one of the plurality of model settings as a specific model setting based on the capacity requirement, a device specification of an edge computing device, and the estimated resource usage of each of the plurality of model settings; and deploying the inference model configured as the specific model setting to the edge computing device. Thus, the method of the embodiments of the present disclosure can more accurately evaluate the inference model suitable for deployment to the edge computing device.

Description

Technical Field

[0001] This invention relates to a model deployment mechanism, and more particularly to an inference model deployment system and method. Background Technology

[0002] In deep learning applications, if a factory production line requires an inference computer with edge computing capabilities, the corresponding inference model will usually be deployed on that computer. If multiple models need to run simultaneously on a single inference computer, the relevant managers will manually calculate how many models the computer can support running at the same time, and then deploy the models to each inference computer accordingly.

[0003] The problem with this approach is that a factory's need for inference computers will vary depending on the product and application, and the inference computers that a factory purchases will not be consistent.

[0004] Generally speaking, inference computers used to perform edge computing do not necessarily have the same hardware specifications or requirements. Furthermore, for some products with smaller requirements, they may not use a single inference computer for processing, but may share the same inference computer with other products. Summary of the Invention

[0005] In view of this, the present disclosure provides an inference model deployment system and a method for deploying inference models, which can be used to solve the above-mentioned technical problems.

[0006] This disclosure provides an inference model deployment system, including an edge computing device and a model management server. The model management server is configured to: obtain the estimated resource usage of each of a plurality of model settings for an inference model; obtain a capacity requirement; select one of the plurality of model settings as a specific model setting based on the capacity requirement, the device specifications of the edge computing device, and the estimated resource usage of each model setting; and deploy the inference model configured with the specific model setting to the edge computing device.

[0007] In embodiments of this disclosure, the model management server is configured to: generate a first reference value based on the estimated resource usage of each model setting, the device specifications and test specifications of the edge computing device; generate a second reference value based on the capacity demand; compare the first reference value and the second reference value to select at least one candidate model setting from the plurality of model settings; and select the specific model setting from the at least one candidate model setting according to a default principle.

[0008] In embodiments of this disclosure, the default principle includes a performance principle, wherein the model management server is configured to obtain the estimated model performance of each of the at least one candidate model settings, and selects the specific model setting from the at least one candidate model settings based on the estimated model performance of each of the at least one candidate model settings.

[0009] In embodiments of this disclosure, the model management server includes a model training component and a model inference testing component. The model training component is used to train the inference model. The model inference testing component is used to apply the trained inference model individually to the plurality of model settings to perform pre-inference operations corresponding to each of the model settings, so as to obtain the estimated resource usage and the estimated model performance for each of the model settings.

[0010] In embodiments of this disclosure, the model inference testing component has test specifications, and the edge computing device runs multiple reference inference models. The model management server further includes a model inference deployment management component. The model inference deployment management component is used to: evaluate whether the edge computing device can deploy the inference model configured for the specific model based on the test specifications of the model inference testing component, the device specifications of the edge computing device, and resource usage; if yes, deploy the inference model configured for the specific model to the edge computing device; and if no, control the edge computing device to unload at least one of the multiple reference inference models, and re-evaluate whether the edge computing device can deploy the inference model configured for the specific model.

[0011] In embodiments of this disclosure, each of the reference inference models has idle time, and the model inference deployment management component is configured to: determine at least one of the plurality of reference inference models to be uninstalled based on the idle time of each of the reference inference models.

[0012] In embodiments of this disclosure, the edge computing device runs multiple reference inference models. The edge computing device includes an inference service interface component, an inference service database, a model data management component, and an inference service core component. The inference service interface component receives at least one request. The inference service database records each of the reference inference models and the usage time of each of the reference inference models. The model data management component is communicatively coupled to the model management server and is used to store and update each of the reference inference models. The inference service core component provides inference services corresponding to the edge computing device and adaptively optimizes or unloads at least one of the multiple reference inference models.

[0013] In embodiments of this disclosure, the edge computing device is deployed with multiple reference inference models, and the model management server is configured to: obtain production schedules for multiple products, identify multiple specific inference models for producing the multiple products from the multiple reference inference models, and control the edge computing device to preload the multiple specific inference models according to the production schedules.

[0014] This disclosure provides a method for deploying an inference model, comprising: obtaining the estimated resource usage of each of a plurality of model settings of an inference model; obtaining the capacity requirement; selecting one of the plurality of model settings as a specific model setting based on the capacity requirement, the device specifications of the edge computing device and the estimated resource usage of each model setting; and deploying the inference model configured as the specific model setting to the edge computing device.

[0015] In embodiments of this disclosure, the step of selecting the specific model setting includes: generating a first reference value based on the estimated resource usage of each model setting, the device specifications and test specifications of the edge computing device; generating a second reference value based on the capacity demand; comparing the first reference value and the second reference value to select at least one candidate model setting from the plurality of model settings; and selecting the specific model setting from the at least one candidate model setting according to a default principle.

[0016] In embodiments of this disclosure, the default principle includes a performance principle, wherein the inference model deployment method further includes: obtaining the estimated model performance of each of the at least one candidate model settings; and selecting the specific model setting from the at least one candidate model settings based on the estimated model performance of each of the at least one candidate model settings.

[0017] In embodiments of this disclosure, the method further includes: training the inference model; applying the trained inference model individually to the plurality of model settings to perform pre-inference operations corresponding to each of the model settings, so as to obtain the estimated resource usage and the estimated model performance of each of the model settings.

[0018] In embodiments of this disclosure, the edge computing device runs multiple reference inference models, and the inference model deployment method further includes: assessing whether the edge computing device can deploy the inference model configured for the specific model based on test specifications, the device specifications of the edge computing device, and resource usage; if yes, deploying the inference model configured for the specific model to the edge computing device; and if no, controlling the edge computing device to unload at least one of the multiple reference inference models, and reassessing whether the edge computing device can deploy the inference model configured for the specific model.

[0019] In embodiments of this disclosure, each of the reference inference models has an idle time, and the method includes: determining at least one of the plurality of reference inference models to be uninstalled based on the idle time of each of the reference inference models.

[0020] In embodiments of this disclosure, the edge computing device is deployed with multiple reference inference models, and the method further includes: obtaining production schedules for multiple products, identifying multiple specific inference models for producing the multiple products from the multiple reference inference models, and controlling the edge computing device to preload the multiple specific inference models according to the production schedules.

[0021] Therefore, compared to known methods of manual evaluation, the approach of this disclosure can more accurately evaluate inference models suitable for deployment on edge computing devices. Attached Figure Description

[0022] The accompanying drawings are included to further illustrate the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0023] Figure 1 This is a schematic diagram illustrating a deductive model deployment system based on embodiments of the present disclosure;

[0024] Figure 2 This is a flowchart illustrating a deductive model deployment method based on embodiments of the present disclosure;

[0025] Figure 3 It is based on Figure 1 A schematic diagram of an edge computing device is shown. Detailed Implementation

[0026] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component reference numerals are used in the drawings and description to denote the same or similar parts.

[0027] Please refer to Figure 1 This is a schematic diagram illustrating a deductive model deployment system based on embodiments of the present disclosure. Figure 1 In this invention, the inference model deployment system 100 includes a model management server 11 and at least one edge computing device 121-12K, where K is a positive integer. In embodiments of this disclosure, each edge computing device 121-12K is, for example, an inference computer with edge computing capabilities, and it may be located in the same or different environments (e.g., factories) and controlled by the model management server 11. In different embodiments, each edge computing device 121-12K may be implemented as various intelligent devices and / or computer devices, but is not limited thereto.

[0028] In one embodiment, each edge computing device 121-12K may be deployed with one or more corresponding reference inference models to realize corresponding inference / prediction functions.

[0029] For example, the edge computing device 121 may be deployed with reference inference models 1211 to 121M (M is a positive integer), and each reference inference model 1211 to 121M may have corresponding inference / prediction functions, such as screen defect detection, but may not be limited to this.

[0030] exist Figure 1 In the model management server 11, there are model training components 112, model inference testing components 114, model inference deployment management components 116 and model inference service interfaces 118. The model training components 112 are coupled to the model inference testing components 114, and the model inference deployment management components 116 are coupled to the model inference testing components 114 and the model inference service interfaces 118.

[0031] Please refer to Figure 2 This is a flowchart illustrating a method for deploying an inference model based on embodiments of the present disclosure. The method of this embodiment can be derived from... Figure 1 The model management server 11 is executed, and the following is the configuration. Figure 1 Component description shown Figure 2 Details of each step.

[0032] First, in step S210, the model management server 11 obtains the estimated resource usage of each of the multiple model settings of the inference model M1.

[0033] In one embodiment, the inference model M1 is, for example, an inference model to be deployed on one or more edge computing devices 121 to 12K. For ease of explanation, it is assumed below that the edge computing device to be deployed is edge computing device 121, but it is not limited thereto.

[0034] In embodiments of this disclosure, the model training component 112 can be used to train multiple inference models including inference model M1, and can publish the weights of each trained inference model and the corresponding multiple model settings to the model inference testing component 114.

[0035] In one embodiment, the model inference testing component 114 applies each trained inference model M1 to a plurality of corresponding model settings to perform pre-inference operations corresponding to each model setting, so as to obtain the estimated resource usage of each model setting. Furthermore, in one embodiment, when performing the aforementioned pre-inference operations corresponding to each model setting, the model inference testing component 114 can also obtain the estimated model performance of each model setting.

[0036] In one embodiment, the inference test component 114 may have its own test specifications, which may include, for example, a reference processor frequency and a reference number of floating-point operations per second (FLOPS). For ease of explanation, the reference processor frequency and the reference number of floating-point operations per second are respectively expressed in clock speeds. test and FLOPS test This indicates that, based on this, the inference test component 114 can perform the aforementioned pre-inference operation using its own test specifications.

[0037] For example, assuming that the inference model M1 has N (N is a positive integer) model settings S1 to SN, the inference test component 114 will apply these N model settings S1 to SN to the trained inference model M1 individually to obtain the estimated resource usage S11 to SN1 and the estimated model performance S12 to SN2 for each of these N model settings S1 to SN.

[0038] For example, the inference test component 114 can apply the inference model M1 configured for model setting S1 to perform pre-inference operations (such as screen defect detection) to obtain the estimated resource usage S11 and estimated model performance S12 of the corresponding model setting S1.

[0039] The model settings S1 to SN of the inference model M1 may include, for example, GPU model, model format, data type, and batch information. In one embodiment, the N model settings S1 to SN of the inference model M1 can be illustrated in Table 1 below.

[0040] Model setting GPU model Model Format Data type Batch information S1 P100 ONNX FLOAT16 8 S2 P100 Darknet FLOAT16 64 … … … … … SN P200 Torch FLOAT32 128

[0041] Table 1

[0042] In one embodiment, the estimated resource usage of each model in the inference model M1 includes at least one of the estimated cycle time and the estimated image memory usage. Additionally, the estimated model performance of each model in the inference model M1 includes at least one of the estimated accuracy, mean average precision (mAP), and recall.

[0043] For example, the estimated resource usage S11 of model setting S1 may include the estimated cycle time and estimated image memory usage corresponding to the inference model M1 applied to model setting S1. In addition, the estimated model performance S12 of model setting S1 may include the estimated accuracy, mean precision and recall corresponding to the inference model M1 of model setting S1.

[0044] In one embodiment, the N models of the inference model M1 are individually configured with estimated resource usage and estimated model performance, as illustrated in Table 2 below.

[0045]

[0046] Table 2

[0047] In step S220, the model management server 11 obtains the capacity requirement RQ. In one embodiment, the model management server 11 may obtain the capacity requirement RQ through the model inference deployment management component 116, for example. In one embodiment, the model inference deployment management component 116 may query a production management system to obtain the capacity requirement RQ, for example. In one embodiment, the capacity requirement RQ may be at least one of the number of units per hour (UPH) of a product and the number of images per unit, but is not limited to this.

[0048] In one embodiment, assuming that the inference model M1 is used to produce a product in a certain project, the model inference deployment management component 116 may, for example, query the production capacity requirement RQ of the project (e.g., the above-mentioned UPH and the number of pictures per unit) in the production management system based on the name and / or work order number of the project, but may not be limited to this.

[0049] In embodiments of this disclosure, the model management server 11 may request one or more of the edge computing devices 121-12K to provide corresponding device specifications and resource usage, and thereby assess whether these edge computing devices are suitable for deploying the inference model M1. For ease of explanation, the following description uses edge computing device 121 among the edge computing devices 121-12K as an example, but those skilled in the art should be able to understand the operations performed by the model management server 11 on other edge computing devices, but it is not limited to this.

[0050] In one embodiment, the model management server 11 obtains the device specifications and resource usage of the edge computing device 121. In one embodiment, the model management server 11 can obtain the device specifications P11 and resource usage P12 of the edge computing device 121 through the model inference deployment management component 116. In one embodiment, the model inference deployment management component 116 can request the edge computing device 121 to report its device specifications P11 and resource usage P12 to the model management server 11, but is not limited to this.

[0051] In one embodiment, the device specification P11 of the edge computing device 121 includes, for example, the total memory space size of the edge computing device 121 (in RAM). total (representation), image memory space size (in GRAM) total (represented by clock speed) processor frequency (in clock speed) edge (represented) and the number of floating-point operations per second (FLOPS) of the image processing unit. edge(represented by) at least one of them. For ease of explanation, the device specification P11 of the edge computing device 121 is assumed to be as illustrated in Table 3 below.

[0052] <![CDATA[RAM total ]]> <![CDATA[GRAM total ]]> <![CDATA[Clock edge ]]> <![CDATA[FLOPS edge ]]> Device Specification P11 32GB 16 GB 3.9GHz 9.3T

[0053] Table 3

[0054] In one embodiment, the resource usage P12 of the edge computing device 121 includes, for example, the current memory usage (in RAM) of each reference inference model 1211-121M. used (in GRAM) Current image memory usage used (represented by Idle_Time) and idle time. RAM for each reference inference model 1211–121M. used For example, this represents the space currently occupied in the memory of the edge computing device 121 by each reference inference model 1211 to 121M. The GRAM of each reference inference model 1211 to 121M. used For example, this represents the space currently occupied by each reference inference model 1211 to 121M in the image memory of the edge computing device 121. The idle time of each reference inference model 1211 to 121M is, for example, the time during which each reference inference model 1211 to 121M is not used to perform inference / prediction / recognition. In one embodiment, the resource usage P12 of the edge computing device 121 can be illustrated as shown in Table 4 below.

[0055] Reference inference model <![CDATA[RAM used ]]> <![CDATA[GRAM used ]]> Idle_Time 1211 0.986GB 3.79GB 0.5s … … … … 121M 1.1GB 6.48GB 7 days

[0056] Table 4

[0057] In step S230, the model management server 11 selects one of the multiple model settings S1 to SN as a specific model setting SS based on the capacity demand RQ, the device specifications of the edge computing device 121, and the estimated resource usage S11 to SN1 of each model setting S1 to SN.

[0058] In one embodiment, the model management server 11 can select one or more candidate model settings from model settings S1 to SN based on the device specifications P11 and resource usage P12 of the edge computing device 121, the estimated resource usage S11 to SN1 of each model setting S1 to SN, and the test specifications of the model inference test component 114, through the model inference deployment management component 116. Then, the model management server 11 can further select a specific model setting SS from the one or more candidate model settings.

[0059] In one embodiment, for a certain first model setting (e.g., model setting S1) among model settings S1 to S1N, the estimated cycle time (denoted as CT) in its estimated resource usage includes, for example, the first processor cycle time (denoted as CT).CPU (represented) and the cycle time of the first image processing unit (in CT) GPU (Representation). In one embodiment, the CT scan defined in the first model is, for example, the CT scan defined in the first model. CPU and CT GPU The sum, i.e., CT = CT CPU +CT GPU However, it is not limited to this.

[0060] In one embodiment, during the process of determining whether a first model setting belongs to a candidate model setting, the model inference deployment management component 116 may generate a first reference value RV1 based, for example, on the estimated resource usage of the first model setting, the device specifications of the edge computing device 121, and the test specifications. For instance, the model inference deployment management component 116 may generate a first reference value RV1 based on, for example, the CT value of the first model setting. CPU CT scan GPU Clock test FLOPS test Clock edge and FLOPS edge Estimate a first reference value RV1. In one embodiment, the first reference value RV1 can be characterized as: But it is not limited to this.

[0061] Furthermore, the model inference deployment management component 116 can also generate a second reference value RV2 based on the capacity demand RQ. For example, the model inference deployment management component 116 can estimate the second reference value RV2 based on the UPH, the number of images per unit (represented by Image) in the capacity demand RQ, and the batch information (represented by Batch) set in the first model. In one embodiment, the second reference value RV2 can be characterized as: in The time taken to produce one unit of product, measured in milliseconds, but not limited to these units.

[0062] In one embodiment, the model inference deployment management component 116 can compare the first reference value RV1 and the second reference value RV2 corresponding to each model setting S1 to SN to select one or more candidate model settings from the model settings S1 to SN. For example, the model inference deployment management component 116 can determine whether the first reference value RV1 of the first model setting (e.g., model setting S1) is less than the second reference value RV2. In response to determining that the first reference value RV1 is less than the second reference value RV2, the model inference deployment management component 116 can determine that the first model setting belongs to the candidate model settings. On the other hand, in response to determining that the first reference value RV1 is greater than the second reference value RV2, the model inference deployment management component 116 can determine that the first model setting does not belong to the candidate model settings.

[0063] In embodiments of this disclosure, the model inference deployment management component 116 can evaluate whether each model setting S1 to SN belongs to the candidate model settings based on the above teachings.

[0064] In different embodiments, the model inference deployment management component 116 may select a specific model setting SS from candidate model settings according to a default principle. The default principle may include a random principle or a performance principle, but is not limited to these. Taking the random principle as an example, the model inference deployment management component 116 may randomly select one from the candidate model settings as the specific model setting SS. Taking the performance principle as an example, the model inference deployment management component 116 may select the one with the best performance from the candidate model settings as the specific model setting SS based on the estimated model performance of each candidate model setting.

[0065] In some embodiments, the performance of the prediction model includes accuracy, precision, F1-score, mean precision, recall, and intersection over union (IoU).

[0066] Subsequently, in step S240, the model management server 11 deploys the inference model M1, configured with SS for a specific model, to the edge computing device 121.

[0067] In one embodiment, after determining a specific model setting SS, the model inference deployment management component 116 can deploy the inference model M1 configured with the specific model setting SS to the edge computing device 121. This allows the inference model M1 configured with the specific model setting SS to perform corresponding inference / preset / recognition behaviors on the edge computing device 121. For example, assuming the specific model setting SS selected by the model inference deployment management component 116 according to the above teaching is model setting S1, the model inference deployment management component 116 can deploy the inference model M1 configured with model setting S1 to the edge computing device 121. This allows the inference model M1 configured with model setting S1 to perform corresponding inference / preset / recognition behaviors on the edge computing device 121.

[0068] In one embodiment, before deploying the inference model M1 configured with a specific model setting SS to the edge computing device 121, the model inference deployment management component 116 may evaluate whether the edge computing device 121 is capable of deploying the inference model M1 configured with a specific model setting SS based on the test specifications of the model inference test component 114, the device specifications P11 of the edge computing device 121, and the resource usage P12.

[0069] In one embodiment, the model inference testing component 114 may have a test memory usage (in RAM) that sets the SS for this particular model. test (represented) and test image memory usage (in GRAM) test (This is indicated). Based on this, in the process of evaluating whether the edge computing device 121 can deploy the inference model M1 configured with SS set for a specific model, the model inference deployment management component 116 can determine the RAM. test RAM with each reference inference model 1211–121M used Is the first sum less than the RAM of the edge computing device 121? total That is, the model inference test component 114 can determine whether the following equation (1) holds true:

[0070] RAM test +∑ 1≤m≤M RAM used，m <RAM total (1)

[0071] RAM used，m It is the RAM of the m-th (m is the index value) reference inference model in reference inference models 1211~121M. used .

[0072] In addition, the model inference deployment management component 116 can also determine GRAM. test GRAM with each reference inference model 1211–121M used Is the second sum less than the GRAM of the edge computing device 121? total That is, the model inference test component 114 can determine whether the following equation (2) holds true:

[0073] GRAM test +∑ 1≤m≤M GRAM used，m <GRAM total (2)

[0074] In one embodiment, the reaction is to determine that the first sum is less than the RAM of the edge computing device 121. total (That is, equation (1) holds), and the second sum is less than the GRAM of the edge computing device 121. total (That is, equation (2) holds), which means that the edge computing device 121 has sufficient computing resources to run the inference model M1 configured with SS for a specific model. In this case, the model inference deployment management component 121 can determine that the edge computing device 121 can deploy the inference model M1 configured with SS for a specific model. Accordingly, the model inference deployment management component 116 can deploy the inference model M1 configured with SS for a specific model to the edge computing device 121.

[0075] On the other hand, if the decision equation (1) and / or equation (2) are not met, it means that there are not enough computing resources on the edge computing device 121 to run the inference model M1 configured with SS for a specific model. In this case, the model inference deployment management component 116 can determine that the edge computing device 121 cannot deploy the inference model M1 configured with SS for a specific model.

[0076] In this case, the model inference deployment management component 116 can control the edge computing device 121 to unload at least one of the reference inference models 1211 to 121M, and re-evaluate whether the edge computing device 121 can deploy the inference model M1 configured with SS for a specific model (i.e., whether equations (1) and (2) are valid). The details of the model inference deployment management component 116's evaluation of whether the edge computing device 121 can deploy the inference model M1 configured with SS for a specific model can be found in the above description and will not be repeated here.

[0077] In one embodiment, the model inference deployment management component 116 can determine the reference inference models to be unloaded based on the idle time of each reference inference model 1211-121M. For example, the model inference deployment management component 116 can select one or more of the reference inference models 1211-121M with the highest idle time as the reference inference models to be unloaded, and accordingly control the edge computing device 121 to unload these reference inference models to be unloaded. In one embodiment, the edge computing device 121 can unload these reference inference models to be unloaded by removing them from memory / image memory (but the models themselves are still retained in the edge computing device 121). As a result, the computing resources of the edge computing device 121 can be released accordingly, making the edge computing device 121 more suitable for deploying inference models M1 configured with specific model settings SS.

[0078] In one embodiment, after unloading some reference inference models from the edge computing device 121, if the model inference deployment management component 116 evaluates that the edge computing device 121 is still unable to deploy the inference model M1 configured with SS for a specific model (i.e., the decision (1) and / or the decision (2) are not valid), the model inference deployment management component 116 may again request the edge computing device 121 to unload other reference inference models in order to free up more computing resources, but is not limited to this.

[0079] In some embodiments, after an inference model M1 configured with a specific model setting SS is deployed to the edge computing device 121, the inference model M1 can also be considered as one of the reference inference models operating on the edge computing device 121. In one embodiment, the model inference deployment management component 116 can collect model key pointer information generated by each reference inference model on the edge computing device 121 after inference is performed. In one embodiment, this model key pointer information can be presented on the model inference service interface 118 so that users of the model management server 11 can track the current execution status and performance of each reference inference model, but it is not limited to this.

[0080] In one embodiment, the model management server 11 can obtain production schedules for multiple products and identify multiple specific inference models for producing these products from the reference inference models 1211-121M of the edge computing device 121. Then, the model management server 11 can control the edge computing device 121 to preload the aforementioned specific inference models based on this production schedule. For example, assuming the production schedule obtained by the model management server 11 requires the edge computing device 121 to produce products A, B, and C in sequence, the model management server 11 can identify multiple specific inference models for producing products A, B, and C from the reference inference models 1211-121M. In one embodiment, assuming that reference inference models 121M, 1211, and 1212 are used to produce products A, B, and C respectively, the model management server 11 can regard the reference inference models 121M, 1211, and 1212 as the specific inference models mentioned above, and require the edge computing device 121 to preload the reference inference models 121M, 1211, and 1212 so that the edge computing device 121 can be used to produce products A, B, and C in sequence, but it is not limited to this.

[0081] Please refer to Figure 3 It is based on Figure 1 A schematic diagram of an edge computing device is shown. In embodiments of this disclosure, the various edge computing devices 121-12K considered may have similar structures, while Figure 3 The example described uses edge computing device 121, but it is not limited to this.

[0082] exist Figure 3 In this embodiment, the edge computing device 121 may include an inference service interface component 311, an inference service database 312, a model data management component 313, and an inference service core component 314. In one embodiment, the inference service interface component 311 may support at least one request, such as a request for the edge computing device 121 to use one or more of the reference inference models 1211 to 121M to perform inference / prediction / recognition operations, but is not limited to this.

[0083] In addition, the inference service database 312 can record each reference inference model 1211 to 121M and its usage time. The model data management component 313 can be used with... Figure 1 The model management server 11 communicates with the model data management component 313 (i.e., the model data management component 313 is communicatively coupled to the model management server 11), and can store and update each reference inference model 1211 to 121M. The inference service core component 314 can provide inference services corresponding to the edge computing device 121, and can adaptively optimize or offload at least one of the reference inference models 1211 to 121M.

[0084] In embodiments of this disclosure, the inference service enables the edge computing device 121 to communicate with the model management server 11 and thereby collaborate with the model management server 11 to perform the technical means taught in the previous embodiments.

[0085] In some embodiments, when selecting an edge computing device from edge computing devices 121-12K for deploying the inference model M1, the model management server 11 may select the edge computing device with the most computing resources (e.g., the one with the most memory space) from the edge computing devices 121-12K as the edge computing device to be deployed. In one embodiment, in response to the determination that the resources of this edge computing device are still insufficient to deploy the inference model M1, the model management server 11 may unload a portion of the reference inference model on this edge computing device to release computing resources, thereby allowing this edge computing device to deploy the inference model M1, but this is not limited to this.

[0086] In summary, the embodiments of this disclosure allow a model management server to select a specific model setting suitable for an edge computing device from multiple model settings of the inference model, and accordingly deploy the inference model configured for this specific model setting to the edge computing device. Therefore, compared to the known method of manual evaluation, the approach of the embodiments of this disclosure can more accurately evaluate the inference model suitable for deployment on the edge computing device.

[0087] In some embodiments, the model management server may also adaptively require the edge computing device to offload a portion of the reference inference model to free up computing resources, thereby enabling the edge computing device to be deployed with an inference model configured for this particular model.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A system for deploying inference models, characterized in that, The inference model deployment system includes: Edge computing devices; and The model management server is communicatively coupled to the edge computing device, and the model management server is configured to: The estimated resource usage of each of the multiple models of the inference model is determined; Obtain production capacity requirements; Based on the stated capacity demand, the device specifications of the edge computing device, and the estimated resource usage of each of the stated models, one of the multiple model settings is selected as a specific model setting, including: A first reference value is generated based on the estimated resource usage set by each of the models, the device specifications and test specifications of the edge computing device; A second reference value is generated based on the aforementioned capacity demand; Compare the first reference value and the second reference value to select at least one candidate model setting from the plurality of model settings; and The specific model setting is selected from the at least one candidate model setting according to the default principle; and The inference model configured for the specific model is deployed to the edge computing device.

2. The inference model deployment system according to claim 1, characterized in that, The default principle includes a performance principle, in which the model management server is configured to obtain the estimated model performance of each of the at least one candidate model settings, and select the specific model setting from the at least one candidate model settings based on the estimated model performance of each of the at least one candidate model settings.

3. The inference model deployment system according to claim 2, characterized in that, The model management server mentioned above includes: Model training components are used to train the inference model; and The model inference testing component is used to apply the trained inference model to the plurality of model settings individually to perform pre-inference operations corresponding to each of the model settings, so as to obtain the estimated resource usage and the estimated model performance of each of the model settings.

4. The inference model deployment system according to claim 3, characterized in that, The model inference testing component has test specifications, and the edge computing device runs multiple reference inference models. The model management server also includes: Model inference deployment management components are used to: Based on the test specifications of the model inference test component, the device specifications and resource usage of the edge computing device, evaluate whether the edge computing device can deploy the inference model configured for the specific model; If so, the inference model configured for the specific model is deployed to the edge computing device; and If not, control the edge computing device to unload at least one of the plurality of reference inference models, and re-evaluate whether the edge computing device is able to deploy the inference model configured for the specific model.

5. The inference model deployment system according to claim 4, characterized in that, Each of the aforementioned reference inference models has idle time, and the model inference deployment management component is configured to: Based on the idle time of each of the aforementioned reference inference models, at least one of the plurality of reference inference models to be uninstalled is determined.

6. The inference model deployment system according to claim 1, characterized in that, The edge computing device runs multiple reference inference models, and the edge computing device includes: The inference service interface component receives at least one request. The inference service database records each of the aforementioned reference inference models and the usage time of each of the aforementioned reference inference models; The model data management component is communicatively coupled to the model management server and is used to store and update each of the reference inference models; and The inference service core component provides inference services corresponding to the edge computing device and adaptively optimizes or offloads at least one of the plurality of reference inference models.

7. The inference model deployment system according to claim 1, characterized in that, The edge computing device is deployed with multiple reference inference models, and the model management server is configured to: Obtain production schedules for multiple products, and identify multiple specific inference models for producing the multiple products from the multiple reference inference models; and The edge computing device is preloaded with the multiple specific inference models based on the production schedule.

8. A method for deploying inference models, suitable for deploying inference models to edge computing devices, characterized in that, The deployment method of the inference model includes: Multiple models of the aforementioned inference model are used to set individual estimated resource usage; Obtain production capacity requirements; Based on the stated capacity demand, the device specifications of the edge computing device, and the estimated resource usage of each of the stated models, one of the multiple model settings is selected as a specific model setting, including: A first reference value is generated based on the estimated resource usage set by each of the models, the device specifications and test specifications of the edge computing device; A second reference value is generated based on the aforementioned capacity demand; Compare the first reference value and the second reference value to select at least one candidate model setting from the plurality of model settings; and The specific model setting is selected from the at least one candidate model setting according to the default principle; and The inference model configured for the specific model is deployed to the edge computing device.

9. The inference model deployment method according to claim 8, characterized in that, The default principles mentioned therein include performance principles, and within these performance principles, the inference model deployment method further includes: Obtain the predicted model performance of at least one candidate model as specified in each of the above descriptions; and The specific model setting is selected from the at least one candidate model setting based on the estimated model performance of each of the at least one candidate model setting.

10. The inference model deployment method according to claim 9, characterized in that, Also includes: Train the inference model; The trained inference models are individually applied to the multiple model settings to perform pre-inference operations corresponding to each model setting, so as to obtain the estimated resource usage and the estimated model performance of each model setting.

11. The inference model deployment method according to claim 10, characterized in that, The edge computing device runs multiple reference inference models, and the inference model deployment method further includes: Based on the test specifications, the device specifications and resource usage of the edge computing device, the assessment evaluates whether the edge computing device can deploy the inference model configured for the specific model. If so, the inference model configured for the specific model is deployed to the edge computing device; and If not, control the edge computing device to unload at least one of the plurality of reference inference models, and re-evaluate whether the edge computing device is able to deploy the inference model configured for the specific model.

12. The inference model deployment method according to claim 11, characterized in that, Each of the aforementioned reference inference models has idle time, and the method includes: Based on the idle time of each of the aforementioned reference inference models, at least one of the plurality of reference inference models to be uninstalled is determined.

13. The inference model deployment method according to claim 8, characterized in that, The edge computing device is deployed with multiple reference inference models, and the method further includes: Obtain production schedules for multiple products, and identify multiple specific inference models for producing the multiple products from the multiple reference inference models; and The edge computing device is preloaded with the multiple specific inference models based on the production schedule.

Images