Inference task processing method and apparatus, and electronic device and storage medium

Abstract

The present disclosure relates to the technical field of artificial intelligence and provides an inference task processing method and apparatus, and an electronic device and a storage medium. The method is applied to an edge device, and comprises: receiving an inference task; determining a cost value for each of a plurality of candidate models to complete the inference task, wherein the state of each candidate model includes one of the following: created on an edge device, to be created on the edge device and to be created on a central cloud; determining a candidate model corresponding to the minimum cost value as a target model; and on the basis of the state of the target model, processing the inference task. Compared with the method of selecting a default model to process an inference task, the present method has higher flexibility, and is more suitable for resource-constrained edge devices.

Description

Reasoning task processing methods, devices, electronic equipment and storage media

[0001] Cross-references to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 202410783672.8, filed on June 18, 2024, entitled “Reasoning Task Processing Method, Apparatus, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of artificial intelligence technology, and in particular to a reasoning task processing method, apparatus, electronic device, and storage medium. Background Technology

[0004] With the development of artificial intelligence technology, the wave of large models has swept the globe, and the types of large models are increasing. AI (Artificial Intelligence) inference technology is widely used. How to select models to handle inference tasks on edge devices with limited resources is an important issue.

[0005] In related technologies, edge devices directly select a default model to process the inference task based on pre-defined rules and the type of inference task.

[0006] However, the approach of selecting a default model based on the type of inference task to handle inference tasks lacks flexibility.

[0007] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] This disclosure provides a reasoning task processing method, apparatus, electronic device, and storage medium, which at least to some extent overcomes the problem of poor flexibility in related technologies.

[0009] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0010] According to one aspect of this disclosure, a method for processing inference tasks is provided, applied to an edge device, comprising: receiving an inference task; determining the cost value of each candidate model among a plurality of candidate models for completing the inference task, wherein the state of each candidate model includes one of being created on the edge device, waiting to be created on the edge device, and waiting to be created in the central cloud; determining the candidate model corresponding to the minimum cost value as the target model; and processing the inference task according to the state of the target model.

[0011] In one embodiment of this disclosure, determining the cost of each candidate model completing the inference task from a plurality of candidate models includes: obtaining cost parameters of each candidate model; calculating the cost of each candidate model completing the inference task based on the cost parameters of each candidate model; wherein, when a candidate model is in a state of having been created on an edge device, the cost parameters include at least one of queuing time, inference resource cost, and inference accuracy; when a candidate model is in a state of waiting to be created on an edge device, the cost parameters include at least one of creation resource cost, creation time, inference resource cost, and inference accuracy; when a candidate model is in a state of waiting to be created in a central cloud, the cost parameters include at least one of transmission latency, creation time, and inference accuracy.

[0012] In one embodiment of this disclosure, the edge device stores model information and identifiers of models whose status is "already created on the edge device." The model information includes identifiers of multiple models and reference information. The identifiers of the multiple models include the identifiers of the models already created on the edge device. The reference information is used to indicate the creation location when a model needs to be created, where the creation location is either the edge device or the central cloud. The step of obtaining the cost parameters of each candidate model includes: obtaining the type of the inference task; selecting the identifiers of the multiple candidate models from the identifiers of the multiple models, where each candidate model is a model among the multiple models used to process the inference task of that type; determining the status of each candidate model based on the identifiers of the models whose status is "already created on the edge device," the identifiers of the multiple candidate models, and the reference information; and obtaining the cost parameters of each candidate model based on the status of each candidate model.

[0013] In one embodiment of this disclosure, the model information further includes the inference accuracy of multiple models; the inference accuracy of each of the multiple candidate models meets a preset condition.

[0014] In one embodiment of this disclosure, when the candidate model is in the state of having been created on the edge device, the cost parameters include queuing time, inference resource cost, and inference accuracy. The step of calculating the cost value of each candidate model in completing the inference task based on its cost parameters includes: when the candidate model is in the state of having been created on the edge device, the cost value of the candidate model in completing the inference task can be calculated according to the following formula: F1=αPL+nβPQ+γ1 / AC

[0015] Where F1 is the cost of completing the inference task when the candidate model is created on the edge device, αPL is the inference resource cost, nβPQ is the queuing time, AC is the inference accuracy, α, β, and γ1 are all positive parameters, P is the number of parameters of the candidate model, L is the number of tokens included in the inference task, Q is the time complexity of the candidate model, and n is the number of inference tasks in the waiting queue of the candidate model.

[0016] In one embodiment of this disclosure, when the candidate model is in the state of waiting to be created on the edge device, the cost parameters include creation resource cost, creation time, inference resource cost, and inference accuracy. The step of calculating the cost value of each candidate model in completing the inference task based on the cost parameters of each candidate model includes: when the candidate model is in the state of waiting to be created on the edge device, the cost value of the candidate model in completing the inference task can be calculated according to the following formula: F2=SD+TD1+αPL+γ2 / AC

[0017] Where F2 is the cost of completing the inference task when the candidate model is created on the edge device, SD is the creation resource cost of creating the candidate model, TD1 is the creation time of creating the candidate model, αPL is the inference resource cost, AC is the inference accuracy, α and γ2 are both positive parameters, P is the number of parameters of the candidate model, and L is the number of tokens included in the inference task.

[0018] In one embodiment of this disclosure, when the candidate model is in the state of waiting to be created in the central cloud, the cost parameters include transmission latency, creation time, and inference accuracy. The step of calculating the cost value of each candidate model in completing the inference task based on its cost parameters includes: when the candidate model is in the state of waiting to be created in the central cloud, the cost value of the candidate model in completing the inference task can be calculated using the following formula: F3=TR+TD2+γ3 / AC TR=q1 / R(p1)+q2 / R(p2)

[0019] Wherein, F3 is the cost of the candidate model being in a state of waiting to complete the inference task when it is created in the central cloud, TR is the transmission latency of uploading the inference task to the central cloud and downloading the feedback result, TD2 is the creation time of the candidate model created in the central cloud, AC is the inference accuracy, γ3 is a positive parameter, q1 is the data volume of the inference task, q2 is the data volume of the feedback result, R() is the Shannon formula, p1 is the upload power, and p2 is the download power.

[0020] According to another aspect of this disclosure, an inference task processing apparatus is provided, applied to an edge device, comprising: a receiving module configured to receive an inference task; a first determining module configured to determine the cost of each candidate model among a plurality of candidate models for completing the inference task, wherein the state of each candidate model includes one of being created on the edge device, waiting to be created on the edge device, and waiting to be created in the central cloud; a second determining module configured to determine the candidate model corresponding to the minimum cost as the target model; and a processing module configured to process the inference task according to the state of the target model.

[0021] In one embodiment of this disclosure, the first determining module is configured to obtain the cost parameters of each candidate model; and calculate the cost value of each candidate model in completing the inference task based on the cost parameters of each candidate model; wherein, when the candidate model is in the state of having been created on an edge device, the cost parameters include at least one of queuing time, inference resource cost, and inference accuracy; when the candidate model is in the state of waiting to be created on an edge device, the cost parameters include at least one of creation resource cost, creation time, inference resource cost, and inference accuracy; and when the candidate model is in the state of waiting to be created in the central cloud, the cost parameters include at least one of transmission latency, creation time, and inference accuracy.

[0022] In one embodiment of this disclosure, the edge device stores model information and identifiers of models whose status is "already created on the edge device." The model information includes identifiers of multiple models and reference information. The identifiers of the multiple models include the identifiers of the models already created on the edge device. The reference information is used to indicate the creation location when a model needs to be created, where the creation location is either the edge device or the central cloud. The first determining module is configured to: obtain the type of the inference task; select the identifiers of multiple candidate models from the identifiers of the multiple models, where each candidate model is a model among the multiple models used to process the inference task of the specified type; determine the status of each candidate model based on the identifiers of the models whose status is "already created on the edge device," the identifiers of the multiple candidate models, and the reference information; and obtain the cost parameters of each candidate model based on the status of each candidate model.

[0023] In one embodiment of this disclosure, when the candidate model is in the state of having been created on the edge device, the cost parameters include queuing time, inference resource cost, and inference accuracy. The first determining module is configured to calculate the cost of the candidate model completing the inference task according to the following formula when the candidate model is in the state of having been created on the edge device: F1=αPL+nβPQ+γ1 / AC

[0024] Where F1 is the cost of completing the inference task when the candidate model is created on the edge device, αPL is the inference resource cost, nβPQ is the queuing time, AC is the inference accuracy, α, β, and γ1 are all positive parameters, P is the number of parameters of the candidate model, L is the number of tokens included in the inference task, Q is the time complexity of the candidate model, and n is the number of inference tasks in the waiting queue of the candidate model.

[0025] In one embodiment of this disclosure, when the candidate model is in the state of waiting to be created on the edge device, the cost parameters include creation resource cost, creation time, inference resource cost, and inference accuracy. The first determining module is configured to calculate the cost of the candidate model completing the inference task according to the following formula when the candidate model is in the state of waiting to be created on the edge device: F2=SD+TD1+αPL+γ2 / AC

[0026] Where F2 is the cost of completing the inference task when the candidate model is created on the edge device, SD is the creation resource cost of creating the candidate model, TD1 is the creation time of creating the candidate model, αPL is the inference resource cost, AC is the inference accuracy, α and γ2 are both positive parameters, P is the number of parameters of the candidate model, and L is the number of tokens included in the inference task.

[0027] In one embodiment of this disclosure, when the candidate model is in the state of waiting to be created in the central cloud, the cost parameters include transmission latency, creation time, and inference accuracy. The first determining module is configured to calculate the cost of the candidate model completing the inference task according to the following formula when the candidate model is in the state of waiting to be created in the central cloud: F3=TR+TD2+γ3 / AC TR=q1 / R(p1)+q2 / R(p2)

[0028] Wherein, F3 is the cost of the candidate model being in a state of waiting to complete the inference task when it is created in the central cloud, TR is the transmission latency of uploading the inference task to the central cloud and downloading the feedback result, TD2 is the creation time of the candidate model created in the central cloud, AC is the inference accuracy, γ3 is a positive parameter, q1 is the data volume of the inference task, q2 is the data volume of the feedback result, R() is the Shannon formula, p1 is the upload power, and p2 is the download power.

[0029] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above-described inference task processing methods by executing the executable instructions.

[0030] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the inference task processing methods described above.

[0031] According to another aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program or computer instructions, the computer program or computer instructions being loaded and executed by a processor to enable a computer to implement any of the reasoning task processing methods described above.

[0032] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0033] The technical solution provided by the embodiments of this disclosure, by determining the cost of each candidate model in completing the inference task among multiple candidate models, can intuitively evaluate the cost of the candidate model in completing the inference task under a corresponding state using the cost value. By determining the candidate model corresponding to the minimum cost value as the target model, the model that can complete the inference task with less cost can be selected. Processing the inference task according to the state of the target model allows edge devices to complete the inference task with less cost. This method has greater flexibility and is more suitable for edge devices with limited resources compared to selecting a default model for processing the inference task.

[0034] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0036] Figure 1 shows a schematic diagram of an inference task processing system according to one embodiment of the present disclosure.

[0037] Figure 2 shows a flowchart of a reasoning task processing method in one embodiment of the present disclosure.

[0038] Figure 3 shows a flowchart of a reasoning task processing method according to another embodiment of this disclosure.

[0039] Figure 4 shows a schematic diagram of a reasoning task processing apparatus according to one embodiment of the present disclosure.

[0040] Figure 5 shows a structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0042] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0044] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0045] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0046] Figure 1 shows a schematic diagram of a reasoning task processing system according to an embodiment of the present disclosure. The system can apply the reasoning task processing method or reasoning task processing device in various embodiments of the present disclosure.

[0047] As shown in Figure 1, the inference task processing system may include: a task generation device 11, an edge device 12, and a central cloud 13.

[0048] The task generation device 11 can generate inference tasks and send them to the edge device 12. In one embodiment, the inference task can be generated by an application installed in the task generation device 11.

[0049] In one embodiment, the edge device 11 may include: an inference task processing device 111, a monitoring module 112, a storage module 113, and an interface 114.

[0050] In one embodiment, the monitoring module 112 can acquire the remaining resources of the edge device 12 and feed them back to the inference task processing device 111. The embodiments of this disclosure do not limit what resources are included in the remaining resources. For example, the remaining resources may include at least one of the following: remaining resources of memory, GPU (Graphics Processing Unit), and CPU (Central Processing Unit).

[0051] In one embodiment, the monitoring module 112 can monitor models that have been created on the edge device, obtain the identifier of the model whose status is "created on the edge device", and store the identifier of the model created on the edge device in the storage module 113.

[0052] In one embodiment, in addition to storing identifiers of models already created on the edge device, the storage module 113 may also store model information, including identifiers and reference information for multiple models. The reference information indicates the creation location when a model needs to be created, which is either the edge device 12 or the central cloud 13. The embodiments of this disclosure do not limit which specific models are included. In one embodiment, the multiple models include models that the edge device 12 can use for inference tasks, and models that the central cloud 13 can use for inference tasks.

[0053] The embodiments of this disclosure do not limit the specific type of model information that the reference information may refer to. For example, the reference information may be the size type of the model, such as a large model or a small model. A large model indicates that the creation location when the model needs to be created is the central cloud 13, while a small model indicates that the creation location when the model needs to be created is the edge device 12. Another example is the number of model parameters. If the number of model parameters is greater than a preset threshold, the creation location when the model needs to be created is the central cloud 13; conversely, if the number of model parameters is not greater than the preset threshold, the creation location when the model needs to be created is the edge device 12. Yet another example is the identifier of the model, either edge device 12 or central cloud 13. If the reference information is the identifier of edge device 12, the creation location when the model needs to be created is edge device 12; conversely, if the reference information is the identifier of central cloud 13, the creation location when the model needs to be created is central cloud 13.

[0054] In one embodiment, the model information may include, in addition to the identifiers and reference information of multiple models, the types and / or inference accuracy of the multiple models. The model category is used to indicate the type of inference task the model can handle.

[0055] In one embodiment, the monitoring module 112 can obtain model information by monitoring the historical data of each model's inference task and store the model information in the storage module 113.

[0056] The inference task processing device 111 can receive inference tasks sent by the task generation device 11, and data sent by the monitoring module 112, which may include remaining resources. The inference task processing device 111 can also retrieve data from the storage module 113, which may include model information and identifiers of models already created on the edge device.

[0057] The inference task processing device 111 can determine the cost of each candidate model among multiple candidate models to complete the inference task. Different candidate models can have different states or the same state. The model's state can include one of being created on an edge device, waiting to be created on an edge device, or waiting to be created in the central cloud. The inference task processing device 111 can also determine the candidate model corresponding to the minimum cost as the target model and process the inference task according to the state of the target model.

[0058] If the target model is in the state of waiting to be created in the central cloud, the inference task processing device 111 can send the inference task to the central cloud 13 through the interface 114, and instruct the central cloud 13 to use the target model to process the inference task, and download the feedback results for the inference task from the central cloud 13.

[0059] The central cloud 13 can receive inference tasks and instructions sent by the inference task processing device 111, create an instance of the target model in the central cloud 13 according to the target model in the instructions, apply the instance to process the inference task, and allow the inference task processing device 111 in the edge device 12 to download the feedback results of the inference task completion through the interface 114.

[0060] If the target model is in a state of waiting to be created on the edge device, the inference task processing device 111 can create an instance of the target model on the edge device 12 and apply the instance to process the inference task. If the target model is in a state of already being created on the edge device, the inference task processing device 111 can send the inference task to the target model's processing queue to queue the use of the target model to process the inference task.

[0061] The task generation device 11, the edge device 12, and the central cloud 13 communicate with each other through a network, which can be a wired network or a wireless network.

[0062] The inference task processing device 111, monitoring module 112, storage module 113 and interface 114 are connected via a network, which can be a wired network or a wireless network.

[0063] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0064] The task generation device 11 and the edge device 12 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, etc.

[0065] Task generation device 11 and edge device 12 can also be servers that provide various services. Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0066] The central cloud 13 can be a central cloud server cluster corresponding to the edge device 12.

[0067] Those skilled in the art will understand that the number of task generation devices 11 in FIG1 is merely illustrative, and any number of task generation devices 11 can be used according to actual needs. This disclosure does not limit this.

[0068] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.

[0069] This disclosure provides a method for processing inference tasks, which can be executed by any electronic device with computing capabilities. For example, the electronic device is an edge device.

[0070] Figure 2 shows a flowchart of a reasoning task processing method in one embodiment of the present disclosure. As shown in Figure 2, the reasoning task processing method provided in this embodiment of the present disclosure includes the following steps S201 to S204.

[0071] S201, Receive reasoning task.

[0072] The embodiments disclosed herein do not limit the specific type of reasoning task. For example, the reasoning task may be an image processing task or a question-answering task, etc.

[0073] In one embodiment, receiving an inference task may include: receiving an inference task sent by an inference task generating device. The inference task may be an inference task generated by any application within the inference task generating device.

[0074] S202, determine the cost of each candidate model in completing the inference task among multiple candidate models, where the state of each candidate model includes one of being created on an edge device, waiting to be created on an edge device, or waiting to be created in the central cloud.

[0075] The cost value represents the cost required for an edge device to complete the inference task using the alternative model. The larger the cost value, the greater the cost required; the smaller the cost value, the smaller the cost required.

[0076] In one embodiment, the cost can represent the latency and / or resource cost required for the edge device to complete the inference task using the alternative model.

[0077] The status of a candidate model is "Created on the edge device," indicating that an instance of the candidate model already exists on the edge device. The status of a candidate model "Pending creation on the edge device" indicates that no instance of the candidate model exists on the edge device. If the candidate model needs to be applied, an instance of the candidate model needs to be created on the edge device. The status of a candidate model "Pending creation in the central cloud" indicates that no instance of the candidate model exists on the edge device. If the candidate model needs to be applied, the inference task needs to be sent to the central cloud, where an instance of the candidate model will be created, and then the instance of the candidate model will be used to process the inference task.

[0078] The embodiments disclosed herein do not limit how to determine the cost of each candidate model in completing the inference task among multiple candidate models.

[0079] In one embodiment, determining the cost of each candidate model completing an inference task among multiple candidate models may include: obtaining cost parameters for each candidate model; and calculating the cost of each candidate model completing an inference task based on the cost parameters of each candidate model. Specifically, when a candidate model is in a state where it has already been created on an edge device, the cost parameters include at least one of queuing time, inference resource cost, and inference accuracy; when a candidate model is in a state where it is waiting to be created on an edge device, the cost parameters include at least one of creation resource cost, creation time, inference resource cost, and inference accuracy; and when a candidate model is in a state where it is waiting to be created in the central cloud, the cost parameters include at least one of transmission latency, creation time, and inference accuracy.

[0080] It should be noted that the cost parameters of the above-mentioned alternative models in the corresponding states are merely illustrative. For example, the cost parameters of the alternative models in each of the above states may also include inference latency.

[0081] Here, queuing time represents the time required to complete the inference task ahead of the one received in S201 in the processing queue using the candidate model. Inference resource cost represents the resource cost required for the edge device to complete the inference task using the candidate model. Inference accuracy is the accuracy of the result obtained by the candidate model processing the inference task. Creation resource cost represents the resource cost required to create a candidate model. Creation time represents the time required to create a candidate model. Transmission latency represents the time required to upload the inference task to the central cloud and download the feedback result, which is the result obtained by the central cloud based on the candidate model to complete the inference task.

[0082] In one embodiment, the plurality of alternative models includes the same model in different states.

[0083] In another embodiment, the multiple alternative models are all different models.

[0084] The embodiments of this disclosure do not limit how the cost parameters of each candidate model are obtained. In one embodiment, the edge device stores model information and identifiers of models whose status is "already created on the edge device." The model information includes identifiers of multiple models and reference information. The identifiers of the multiple models include identifiers of models already created on the edge device, and the reference information is used to indicate the creation location when a model needs to be created, which is either the edge device or the central cloud. In this case, obtaining the cost parameters of each candidate model may include: obtaining the type of inference task; selecting identifiers of multiple candidate models from the identifiers of multiple models, each candidate model being a model among the multiple models used to handle the inference task of that type; determining the status of each candidate model based on the identifiers of models whose status is "already created on the edge device," the identifiers of the multiple candidate models, and the reference information; and obtaining the cost parameters of each candidate model based on the status of each candidate model.

[0085] The type of inference task can be sent to the edge device along with the inference task itself. Obtaining the type of inference task can include receiving the type of inference task sent by the task generation device. Alternatively, the inference task includes a type parameter, and the type of inference task can be determined based on this type parameter.

[0086] The embodiments disclosed herein do not limit how to select multiple candidate model identifiers from multiple model identifiers.

[0087] In one embodiment, the model information includes not only identifiers and reference information for multiple models, but also capability information for multiple models. This capability information indicates the types of inference tasks the model can handle; for example, the capability information indicates that the model can handle two or three types of inference tasks, etc. Selecting multiple candidate model identifiers from the multiple model identifiers may include: selecting multiple candidate model identifiers from the multiple model identifiers based on the type of inference task and the capability information of the multiple models. The capability information of the candidate models indicates the types of inference tasks they can handle, including the types of inference tasks.

[0088] In another embodiment, the model information includes not only identifiers and reference information for multiple models, but also types for multiple models. Each model has a type, which indicates the type of inference task the model handles. Selecting multiple candidate model identifiers from the multiple model identifiers may include: selecting multiple candidate model identifiers from the multiple model identifiers based on the type of the inference task and the types of the multiple models, wherein the type of the candidate models is consistent with the type of the inference task.

[0089] In one embodiment, the model information may include not only the identifiers and reference information of multiple models, but also the inference accuracy of multiple models. The inference accuracy of each candidate model among these candidate models satisfies a preset condition; that is, when selecting candidate model identifiers from multiple model identifiers, the inference accuracy of the model needs to be considered. The embodiments of this disclosure do not limit the specific nature of this preset condition. For example, the preset condition may be that the inference accuracy is greater than a preset value. The embodiments of this disclosure do not limit the specific value of this preset value; for example, the preset value may be 0, or 0.8, etc.

[0090] In one embodiment, determining the status of each candidate model based on the identifier of the model whose status is "already created on the edge device," the identifiers of multiple candidate models, and reference information may include: if the identifier of the model whose status is "already created on the edge device" includes the identifier of the candidate model, then the status of the candidate model is "already created on the edge device"; otherwise, if the identifier of the model whose status is "already created on the edge device" does not include the identifier of the candidate model, then the status of the candidate model is determined to be either "waiting to be deployed on the edge device" or "waiting to be deployed in the central cloud" based on the reference information of the candidate model.

[0091] The embodiments of this disclosure do not limit the specific type of reference information. Please refer to the description of the reference information in the system embodiment corresponding to Figure 1, which will not be repeated here.

[0092] After determining the state of each candidate model among multiple candidate models, the cost parameters corresponding to each candidate model can be determined based on the state of each candidate model, thereby obtaining the corresponding cost parameters.

[0093] In one embodiment, the inference resource cost can be determined by the product of the number of parameters (P) of the candidate model and the number of tokens (L) included in the inference task. The number of parameters of the candidate model can be pre-stored on the edge device, and the number of tokens included in the inference task can be directly determined by the specific inference task. Obtaining the inference resource cost may include: obtaining the number of parameters of the candidate model and the number of tokens included in the inference task, calculating the product of the number of parameters and the number of tokens, and determining the inference resource cost (αPL, where α is a positive parameter, and the specific value of α can be calculated from the resource cost consumed by the candidate model when performing inference tasks in the past).

[0094] In one embodiment, the queuing time can be determined by the number (n) of inference tasks ahead of the inference tasks received in S201 in the processing queue of the candidate model, the number of parameters (P) of the candidate model, and the time complexity (Q) of the candidate model. The time complexity (Q) and the number of parameters (P) of the candidate model are only related to the specific candidate model and are fixed information for the candidate model. Q and P can be pre-stored in the edge device, while n can be directly determined based on the processing queue. Obtaining the queuing time can include: obtaining n, P, and Q; calculating the product of the three parameters n, P, and Q; and obtaining the queuing time (nβPQ, where β is a positive parameter that can be determined from the historical processing time of the candidate model for inference tasks).

[0095] In one embodiment, the model information may include, in addition to the identifiers and reference information of multiple models, the inference accuracy of those multiple models. Obtaining the inference accuracy of a candidate model may include retrieving the inference accuracy of the candidate model from the model information stored on the edge device.

[0096] In one embodiment, the creation resource cost is determined by alternative models, with each alternative model corresponding to a fixed creation resource cost (SD). The specific data of the creation resource cost of the alternative models can be pre-stored in the edge device. Obtaining the creation resource cost of the alternative models can include: directly retrieving the creation resource cost of the alternative models from memory.

[0097] It should be noted that when a candidate model is selected and its status is "awaiting creation on the edge device", the resource cost of creating the candidate model must be less than the remaining resources of the edge device.

[0098] In one embodiment, in either the edge device or the central cloud, the creation time is determined only by the time used to create the candidate model. Each candidate model corresponds to a creation time, and this time can be determined based on the historical creation time of the candidate model. Accordingly, the creation time can be pre-stored in the edge device. Obtaining the creation time of the candidate model may include: directly obtaining the creation time of the candidate model from memory (wherein, when the state is pending creation on the edge device, the creation time can be represented as TD1, and when the state is pending creation in the central cloud, the creation time can be represented as TD2).

[0099] In one embodiment, the transmission latency is determined by the upload rate of the edge device and the amount of data uploaded (q1) when uploading the inference task to the central cloud, the download rate and the amount of data downloaded (q2) when downloading the feedback result from the central cloud. The upload rate (R(p1), where R() is the Shannon formula (communication bandwidth and transmission distance are the parameters in the formula), and p1 is the upload power) is determined by the communication bandwidth, transmission distance and upload power between the edge device and the central cloud. The download rate (R(p2), where p2 is the upload power) is determined by the communication bandwidth, transmission distance and download power between the edge device and the central cloud.

[0100] The communication bandwidth and transmission distance between the edge device and the central cloud can be pre-stored in the edge device. The download rate and upload power can be determined based on the power used by the device during data upload and download, or the download and upload power can be fixed values ​​in the edge device, and these can also be pre-stored in the edge device. The amount of data corresponding to the inference task can be directly determined based on the inference task. The amount of data corresponding to the feedback result can be determined based on the amount of data obtained by the central cloud in historical inference tasks using the alternative model. Accordingly, the amount of data for the feedback result can also be pre-stored in the edge device. Obtaining the transmission latency can include: obtaining the communication bandwidth, transmission distance, download power, upload power, amount of data for the inference task, and amount of data for the feedback result between the edge device and the central cloud, and calculating the transmission latency (q1 / R(p1)+q2 / R(p2)).

[0101] The embodiments disclosed herein do not impose limitations on calculating the cost value of each alternative model in completing the inference task based on the cost parameters of each alternative model.

[0102] In one embodiment, when the candidate model is in the state of being created on the edge device, the cost of the candidate model completing the inference task can be positively correlated with queuing time and inference resource cost, and negatively correlated with inference accuracy.

[0103] In one embodiment, taking the state of the candidate model as already created on the edge device, the cost parameters include queuing time, inference resource cost, and inference accuracy. Calculating the cost of each candidate model completing the inference task based on its cost parameters can include:

[0104] When the candidate model is already created on the edge device, the cost of completing the inference task can be calculated using the following formula 1: F1=αPL+nβPQ+γ1 / AC (Formula 1)

[0105] Where F1 is the cost of completing the inference task when the candidate model is created on the edge device, αPL is the inference resource cost, nβPQ is the queuing time, AC is the inference accuracy, α, β, and γ1 are all positive parameters, P is the number of parameters of the candidate model, L is the number of tokens included in the inference task, Q is the time complexity of the candidate model, and n is the number of inference tasks in the waiting queue of the candidate model.

[0106] In one embodiment, when the candidate model is in a state of waiting to be created on the edge device, taking the cost parameters including creation resource cost, creation time, inference resource cost, and inference accuracy as an example, the cost of each candidate model completing the inference task is calculated based on the cost parameters of each candidate model, which may include:

[0107] When the candidate model is in the state of waiting to be created on the edge device, the cost of the candidate model completing the inference task can be calculated according to the following formula 2: F2=SD+TD1+αPL+γ2 / AC (Formula 2)

[0108] Where F2 is the cost of completing the inference task when the candidate model is created on the edge device, SD is the creation resource cost of creating the candidate model, TD1 is the creation time of creating the candidate model, αPL is the inference resource cost, AC is the inference accuracy, α and γ2 are both positive parameters, P is the number of parameters of the candidate model, and L is the number of tokens included in the inference task.

[0109] In one embodiment, when the candidate model is in the state of waiting to be created in the central cloud, taking the cost parameters including transmission latency, creation time, and inference accuracy as an example, the cost value of each candidate model to complete the inference task is calculated based on the cost parameters of each candidate model, including:

[0110] When the candidate model is in the state of waiting to be created in the central cloud, the cost of the candidate model to complete the inference task can be calculated according to the following formulas 3 and 4: F3=TR+TD2+γ3 / AC (Formula 3) TR=q1 / R(p1)+q2 / R(p2) (Formula 4)

[0111] Where F3 is the cost of completing the inference task when the candidate model is created in the central cloud, TR is the transmission latency of uploading the inference task to the central cloud and downloading the feedback result, TD2 is the creation time of the candidate model in the central cloud, AC is the inference accuracy, γ3 is the positive parameter, q1 is the data volume of the inference task, q2 is the data volume of the feedback result, R() is the Shannon formula, p1 is the upload power, and p2 is the download power.

[0112] When calculating the cost value, considering the inference accuracy of the candidate models and negatively correlating the cost value with the inference accuracy can reduce the cost value of the candidate model with higher inference accuracy in completing the inference task. This makes it easier to select the candidate model with higher inference accuracy as the target model to handle the inference task.

[0113] S203, the candidate model corresponding to the minimum cost value is determined as the target model.

[0114] The candidate model with the lowest cost among multiple candidate models is selected as the target model.

[0115] S204, Process the inference task based on the state of the target model.

[0116] Once the target model is determined, if the target model is in the state of waiting to be created in the central cloud, the edge device can send the inference task to the central cloud, instruct the central cloud to use the target model to process the inference task, and download the feedback results for the inference task from the central cloud.

[0117] If the target model is in a state of waiting to be created on the edge device, the edge device can create an instance of the target model and apply that instance to process the inference task.

[0118] If the target model has already been created on the edge device, the inference task can be sent to the target model's processing queue to queue for use by the target model to process the inference task.

[0119] The technical solution provided by the embodiments of this disclosure, by determining the cost of each candidate model in completing the inference task among multiple candidate models, can intuitively evaluate the cost of the candidate model in completing the inference task under a corresponding state using the cost value. By determining the candidate model corresponding to the minimum cost value as the target model, the model that can complete the inference task with less cost can be selected. Processing the inference task according to the state of the target model allows edge devices to complete the inference task with less cost. This method has greater flexibility and is more suitable for edge devices with limited resources compared to selecting a default model for processing the inference task.

[0120] As shown in Figure 3, the reasoning task processing method in another embodiment of this disclosure may include S301 to S307. The specific implementation of S301-S307 can be found in the embodiment corresponding to Figure 2, and will not be described again here.

[0121] S301, receive reasoning task.

[0122] S302, Obtain the type of reasoning task.

[0123] S303, select multiple alternative models.

[0124] S304, determine the status of each candidate model.

[0125] S305, based on the status of each candidate model, determine the cost of each candidate model completing the reasoning task.

[0126] S306, the candidate model corresponding to the minimum cost value is determined as the target model.

[0127] S307, Process the inference task based on the state of the target model.

[0128] Based on the same inventive concept, this disclosure also provides a reasoning task processing device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.

[0129] Figure 4 shows a schematic diagram of an inference task processing device according to an embodiment of the present disclosure, applied to an edge device. As shown in Figure 4, the device includes: a receiving module 41, configured to receive inference tasks; a first determining module 42, configured to determine the cost of each candidate model in a plurality of candidate models to complete the inference task, wherein the state of each candidate model includes one of being created on the edge device, waiting to be created on the edge device, and waiting to be created in the central cloud; a second determining module 43, configured to determine the candidate model corresponding to the minimum cost as the target model; and a processing module 44, configured to process the inference task according to the state of the target model.

[0130] In one embodiment of this disclosure, the first determining module 42 is configured to obtain the cost parameters of each candidate model; and calculate the cost value of each candidate model completing the inference task based on the cost parameters of each candidate model; wherein, when the candidate model is in the state of having been created on the edge device, the cost parameters include at least one of queuing time, inference resource cost, and inference accuracy; when the candidate model is in the state of waiting to be created on the edge device, the cost parameters include at least one of creation resource cost, creation time, inference resource cost, and inference accuracy; when the candidate model is in the state of waiting to be created in the central cloud, the cost parameters include at least one of transmission latency, creation time, and inference accuracy.

[0131] In one embodiment of this disclosure, the edge device stores model information and identifiers of models whose status is "already created on the edge device". The model information includes identifiers of multiple models and reference information. The identifiers of multiple models include the identifiers of models already created on the edge device. The reference information is used to indicate the creation location when a model needs to be created, which is either the edge device or the central cloud. The first determining module 42 is configured to obtain the cost parameters of each candidate model, including: obtaining the type of inference task; selecting the identifiers of multiple candidate models from the identifiers of multiple models, each candidate model being a model among the multiple models used to process the type of inference task; determining the status of each candidate model based on the identifiers of models whose status is "already created on the edge device", the identifiers of multiple candidate models and the reference information; and obtaining the cost parameters of each candidate model based on the status of each candidate model.

[0132] In one embodiment of this disclosure, when the state of the candidate model is that it has been created on the edge device, the types of cost parameters include queuing time, inference resource cost, and inference accuracy; the first determining module 42 is configured to calculate the cost of the candidate model completing the inference task according to Formula 1 above when the state of the candidate model is that it has been created on the edge device.

[0133] In one embodiment of this disclosure, when the candidate model is in the state of waiting to be created on the edge device, the cost parameters include creation resource cost, creation time, inference resource cost, and inference accuracy; the first determining module 42 is configured to calculate the cost of the candidate model completing the inference task according to the above formula 2 when the candidate model is in the state of waiting to be created on the edge device.

[0134] In one embodiment of this disclosure, when the candidate model is in the state of waiting to be created in the central cloud, the cost parameters include transmission latency, creation time, and inference accuracy. The first determining module 42 is configured to calculate the cost of the candidate model completing the inference task according to the above formulas 3 and 4 when the candidate model is in the state of waiting to be created in the central cloud.

[0135] It should be noted that the examples and application scenarios implemented by the modules in the above-described inference task processing device embodiments and the corresponding steps in the method embodiments are the same, but are not limited to the content disclosed in the above-described method embodiments. It should also be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0136] The technical solution provided by the embodiments of this disclosure, by determining the cost of each candidate model in completing the inference task among multiple candidate models, can intuitively evaluate the cost of the candidate model in completing the inference task under a corresponding state using the cost value. By determining the candidate model corresponding to the minimum cost value as the target model, the model that can complete the inference task with less cost can be selected. Processing the inference task according to the state of the target model allows edge devices to complete the inference task with less cost. This method has greater flexibility and is more suitable for edge devices with limited resources compared to selecting a default model for processing the inference task.

[0137] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0138] The electronic device 500 according to this embodiment of the present disclosure will now be described with reference to FIG5. The electronic device 500 shown in FIG5 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present disclosure.

[0139] As shown in Figure 5, the electronic device 500 is presented in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including storage unit 520 and processing unit 510).

[0140] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Detailed Description" section of this specification according to various exemplary embodiments of this disclosure.

[0141] Storage unit 520 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include a read-only memory (ROM) 5203.

[0142] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5205, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0143] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0144] Electronic device 500 can also communicate with one or more external devices 540 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown in Figure 5, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0145] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0146] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Detailed Description" section above.

[0147] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0149] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0150] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0151] In exemplary embodiments of this disclosure, a computer program product is also provided, comprising a computer program or computer instructions, which are loaded and executed by a processor to enable a computer to perform the steps of the various exemplary embodiments of this disclosure described in the foregoing “Detailed Description” section of this specification.

[0152] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0153] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0154] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0155] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope of this disclosure is indicated by the appended claims.

Claims

1. A method for processing an inference task, applied to an edge device, comprising: receiving an inference task; determining a value of each of a plurality of candidate models for completing the inference task, wherein a state of each candidate model comprises one of having been created at the edge device, to be created at the edge device, and to be created at a central cloud; determining a target model corresponding to a minimum value; processing the inference task according to the state of the target model.

2. The method of claim 1, wherein, The determination of the value of each of the plurality of candidate models for completing the inference task comprises: obtaining a cost parameter of each candidate model; calculating the value of each candidate model for completing the inference task according to the cost parameter of each candidate model; wherein when the state of a candidate model is having been created at the edge device, the cost parameter comprises at least one of queuing time, inference resource cost, and inference accuracy; when the state of a candidate model is to be created at the edge device, the cost parameter comprises at least one of creation resource cost, creation time, inference resource cost, and inference accuracy; and when the state of a candidate model is to be created at the central cloud, the cost parameter comprises at least one of transmission delay, creation time, and inference accuracy.

3. The method of claim 2, wherein, The edge device stores model information and an identifier of a model having been created at the edge device, wherein the model information comprises identifiers and reference information of a plurality of models, the identifiers of the plurality of models comprise the identifier of the model having been created at the edge device, and the reference information is used to indicate a creation location when a model needs to be created, the creation location being the edge device or the central cloud. The obtaining of the cost parameter of each candidate model comprises: obtaining a type of the inference task; selecting identifiers of the plurality of candidate models from the identifiers of the plurality of models, each candidate model being a model of the plurality of models for processing an inference task of the type; determining the state of each candidate model according to the identifier of the model having been created at the edge device, the identifiers of the plurality of candidate models, and the reference information; obtaining the cost parameter of each candidate model according to the state of each candidate model.

4. The method of claim 3, wherein, The model information further comprises inference accuracies of the plurality of models; the inference accuracy of each candidate model in the plurality of candidate models satisfies a preset condition.

5. The method of claim 2, wherein, When the state of a candidate model is having been created at the edge device, the type of the cost parameter comprises queuing time, inference resource cost, and inference accuracy; and the calculation of the value of each candidate model for completing the inference task according to the cost parameter of each candidate model comprises: when the state of a candidate model is having been created at the edge device, the value of the candidate model for completing the inference task can be calculated according to the following formula: F1 = αPL + nβPQ + γ1 / AC. ​ F1=SD+TD1+αPL+nβPQ+γ1 / AC wherein, F1 is a value of a cost of completing the inference task when the state of the candidate model is that the inference task has been created at the edge device, αPL is an inference resource cost, nβPQ is a queuing time, AC is an inference accuracy rate, α, β, γ1 are all positive parameters, P is a parameter quantity of the candidate model, L is a number of tokens Token contained in the inference task, Q is a time complexity of the candidate model, and n is a number of inference tasks in a queue of the candidate model.

6. The method of claim 2, wherein, When the state of the candidate model is that the candidate model is to be created at the edge device, the types of the cost parameters include a creation resource cost, a creation time, an inference resource cost, and an inference accuracy rate. The calculating of the value of the cost of completing the inference task according to the cost parameters of each candidate model includes: When the state of the candidate model is that the candidate model is to be created at the edge device, the value of the cost of completing the inference task of the candidate model can be calculated according to the following formula: F2=SD+TD1+αPL+γ2 / AC wherein, F2 is a value of a cost of completing the inference task when the state of the candidate model is that the inference task is to be created at the edge device, SD is a creation resource cost of creating the candidate model, TD1 is a creation time of creating the candidate model, αPL is an inference resource cost, AC is an inference accuracy rate, α, γ2 are all positive parameters, P is a parameter quantity of the candidate model, and L is a number of tokens Token contained in the inference task. When the state of the candidate model is that the candidate model is to be created at the central cloud, the types of the cost parameters include a transmission delay, a creation time, and an inference accuracy rate.

7. The method of claim 2, wherein, The calculating of the value of the cost of completing the inference task according to the cost parameters of each candidate model includes: When the state of the candidate model is that the candidate model is to be created at the central cloud, the value of the cost of completing the inference task of the candidate model can be calculated according to the following formula: F3=TR+TD2+γ3 / AC TR=q1 / R(p1)+q2 / R(p2) wherein, F3 is a value of a cost of completing the inference task when the state of the candidate model is that the inference task is to be created at the central cloud, TR is a transmission delay of uploading the inference task to the central cloud and downloading a feedback result, TD2 is a creation time of creating the candidate model at the central cloud, AC is an inference accuracy rate, γ3 is a positive parameter, q1 is a data quantity of the inference task, q2 is a data quantity of the feedback result, R() is a Shannon formula, p1 is an upload power, and p2 is a download power.

8. An inference task processing apparatus, applied to an edge device, comprising: a receiving module, configured to receive an inference task; a first determining module, configured to determine a value of a cost of completing the inference task of each candidate model in a plurality of candidate models, a state of each candidate model including one of having been created at the edge device, being to be created at the edge device, and being to be created at a central cloud; a second determining module, configured to determine a candidate model corresponding to a minimum value of the cost as a target model; a processing module, configured to process the inference task according to a state of the target model.

9. An electronic device, comprising: a processor; and a memory, configured to store executable instructions of the processor. ​ ​ ​ The processor is configured to execute the inference task processing method of any one of claims 1-7 by executing the executable instructions. 10.A computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the inference task processing method of any one of claims 1-7. 11.A computer program product, the computer program product comprising a computer program or computer instructions, the computer program or the computer instructions being loaded and executed by a processor to cause a computer to implement the inference task processing method of any one of claims 1-7.

Images