Reasoning task processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a method, device, electronic device and storage medium for processing reasoning tasks, and relates to the field of artificial intelligence technology. The method is applied to an edge device, and includes: receiving a reasoning task; determining the cost of each candidate model in a plurality of candidate models to complete the reasoning task, and the status of each candidate model includes one of having been created on the edge device, to be created on the edge device, and to be created on the central cloud; determining the candidate model corresponding to the minimum cost value as the target model; and processing the reasoning task according to the status of the target model. Compared with selecting the default model to process the reasoning task, this method has higher flexibility and is more suitable for edge devices with limited resources.

Description

Background Art

[0002] With the development of artificial intelligence technology, the wave of big models has swept the world. There are more and more types of big models, and AI (Artificial Intelligence) reasoning technology has been widely used. How to select models to handle reasoning tasks on edge devices with limited resources is an important issue.

[0003] In the related art, the edge device directly selects a default model to process the reasoning task according to the type of reasoning task based on manually pre-set rules.

[0004] However, the method of selecting a default model to handle reasoning tasks according to the type of reasoning tasks has poor flexibility.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a method, device, electronic device and storage medium for processing reasoning tasks, which at least to some extent overcome the problem of poor flexibility in related technologies.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, there is provided a method for processing an inference task, which is applied to an edge device, and includes: receiving an inference task; determining a cost value of each candidate model among multiple candidate models to complete the inference task, wherein the status of each candidate model includes one of having been created on the edge device, to be created on the edge device, and to be created on the central cloud; determining the candidate model corresponding to the minimum cost value as the target model; and processing the inference task according to the status of the target model.

[0009] In one embodiment of the present disclosure, determining the cost of each alternative model among multiple alternative models to complete the reasoning task includes: obtaining the cost parameters of each alternative model; calculating the cost of each alternative model to complete the reasoning task according to the cost parameters of each alternative model; wherein, when the status of the alternative model is that it has been created on the edge device, the cost parameters include: at least one of queuing time, inference cost and inference accuracy; when the status of the alternative model is to be created on the edge device, the cost parameters include: at least one of creation cost, creation time, inference cost and inference accuracy; when the status of the alternative model is to be created on the central cloud, the cost parameters include: at least one of transmission delay, creation time and inference accuracy.

[0010] In one embodiment of the present disclosure, the edge device stores model information and an identifier of a model whose status is that it has been created on the edge device, the model information includes identifiers and reference information of multiple models, the identifiers of the multiple models include the identifier of the model that has been created on the edge device, and the reference information is used to indicate the creation location when the model needs to be created, and the creation location is the edge device or the central cloud; obtaining the cost parameters of each alternative model includes: obtaining the type of the reasoning task; selecting the identifiers of the multiple alternative models from the identifiers of the multiple models, each alternative model is a model among the multiple models used to process the reasoning task of the type; determining the status of each alternative model according to the identifier of the model whose status is that it has been created on the edge device, the identifiers of the multiple alternative models and the reference information; obtaining the cost parameters of each alternative model according to the status of each alternative model.

[0011] In one embodiment of the present disclosure, the model information also includes reasoning accuracy of multiple models; the reasoning accuracy of each candidate model in the multiple candidate models meets a preset condition.

[0012] In one embodiment of the present disclosure, when the status of the candidate model is that it has been created on the edge device, the types of cost parameters include queuing time, inference cost, and inference accuracy; the cost value of each candidate model to complete the inference task is calculated according to the cost parameters of each candidate model, including: when the status of the candidate model is that it has been created on the edge device, the cost value of the candidate model to complete the inference task can be calculated according to the following formula:

[0013] F1= aPL + nβPQ + c 1 / AC

[0014] Among them, F1 is the cost value of the candidate model that has completed the reasoning task when the edge device is created, α PL is the inference cost, nβPQ For the queue time, AC is the inference accuracy, α , β , c 1 are all positive parameters, P is the number of parameters of the alternative model, L is the number of tokens contained in the reasoning task, Q is the time complexity of the alternative model, n is the number of inference tasks in the waiting queue of the candidate model.

[0015] In one embodiment of the present disclosure, when the state of the candidate model is to be created on the edge device, the types of cost parameters include creation cost, creation time, inference cost and inference accuracy; the cost value of each candidate model to complete the inference task is calculated according to the cost parameters of each candidate model, including: when the state of the candidate model is to be created on the edge device, the cost value of the candidate model to complete the inference task can be calculated according to the following formula:

[0016] F2= SD + TD 1+ aPL + c 2 / AC

[0017] Among them, F2 is the state of the candidate model, which is the cost value of completing the reasoning task when the edge device is created. SD To create the cost of creating alternative models, TD 1 is the creation time of the alternative model, aPL is the inference cost, AC is the inference accuracy, α , c 2 are all positive parameters, P is the number of parameters of the alternative model, L The number of tokens included in the reasoning task.

[0018] In one embodiment of the present disclosure, when the state of the candidate model is to be created in the central cloud, the types of cost parameters include transmission delay, creation time and inference accuracy; the cost value of each candidate model to complete the inference task is calculated according to the cost parameters of each candidate model, including: when the state of the candidate model is to be created in the central cloud, the cost value of the candidate model to complete the inference task can be calculated according to the following formula:

[0019] F3= TR + TD 2+ c 3 / AC

[0020] TR = q 1 / R ( p 1)+ q 2 / R ( p 2)

[0021] Among them, F3 is the cost value of the candidate model to complete the reasoning task when the central cloud is created. TR The transmission delay for uploading the inference task to the central cloud and downloading the feedback results is TD 2 is the creation time of the candidate model created by the central cloud, AC is the inference accuracy, c 3 is a positive parameter, q 1 is the data volume of the reasoning task, q 2 is the data volume of the feedback result, R () is Shannon's formula, p 1 is the upload power, p 2 is the download power.

[0022] According to another aspect of the present disclosure, there is provided an inference task processing apparatus, which is applied to an edge device, and includes: a receiving module, which is used to receive an inference task; a first determination module, which is used to determine the cost value of each alternative model among multiple alternative models to complete the inference task, and the status of each alternative model includes one of having been created in the edge device, to be created in the edge device, and to be created in the central cloud; a second determination module, which is used to determine the alternative model corresponding to the minimum cost value as the target model; and a processing module, which is used to process the inference task according to the status of the target model.

[0023] In one embodiment of the present disclosure, the first determination module is used to obtain the cost parameters of each alternative model; according to the cost parameters of each alternative model, the cost value of each alternative model to complete the reasoning task is calculated; wherein, when the status of the alternative model is that it has been created on the edge device, the cost parameters include: at least one of queuing time, inference cost and inference accuracy; when the status of the alternative model is to be created on the edge device, the cost parameters include: at least one of creation cost, creation time, inference cost and inference accuracy; when the status of the alternative model is to be created on the central cloud, the cost parameters include: at least one of transmission delay, creation time and inference accuracy.

[0024] In one embodiment of the present disclosure, the edge device stores model information and an identifier of a model whose status is that it has been created on the edge device, the model information includes identifiers and reference information of multiple models, the identifiers of the multiple models include the identifier of the model that has been created on the edge device, and the reference information is used to indicate the creation location when the model needs to be created, and the creation location is the edge device or the central cloud; the first determination module is used to obtain the cost parameters of each alternative model, including: obtaining the type of the reasoning task; selecting the identifiers of the multiple alternative models from the identifiers of the multiple models, each alternative model is a model among the multiple models used to process the reasoning task of the type; determining the status of each alternative model according to the identifier of the model whose status is that it has been created on the edge device, the identifiers of the multiple alternative models and the reference information; obtaining the cost parameters of each alternative model according to the status of each alternative model.

[0025] In one embodiment of the present disclosure, when the status of the candidate model is that it has been created on the edge device, the types of cost parameters include queuing time, inference cost, and inference accuracy; the first determination module is used to calculate the cost value of the candidate model to complete the inference task according to the following formula when the status of the candidate model is that it has been created on the edge device:

[0026] F1= aPL + nβPQ + c 1 / AC

[0027] Among them, F1 is the cost value of the candidate model that has completed the reasoning task when the edge device is created, α PL is the inference cost, nβPQ For the queue time, AC is the inference accuracy, α , β , c 1 are all positive parameters, P is the number of parameters of the alternative model, L is the number of tokens contained in the reasoning task, Q is the time complexity of the alternative model, n is the number of inference tasks in the waiting queue of the candidate model.

[0028] In one embodiment of the present disclosure, when the state of the candidate model is to be created on the edge device, the types of cost parameters include creation cost, creation time, inference cost and inference accuracy; the first determination module is used to calculate the cost value of the candidate model to complete the inference task according to the following formula when the state of the candidate model is to be created on the edge device:

[0029] F2= SD + TD 1+ aPL + c 2 / AC

[0030] Among them, F2 is the state of the candidate model, which is the cost value of completing the reasoning task when the edge device is created. SD To create the cost of creating alternative models, TD 1 is the creation time of the alternative model, aPL is the inference cost, AC is the inference accuracy, α , c 2 are all positive parameters, P is the number of parameters of the alternative model, L The number of tokens included in the reasoning task.

[0031] In one embodiment of the present disclosure, when the state of the candidate model is to be created in the central cloud, the types of cost parameters include transmission delay, creation time and inference accuracy; the first determination module is used to calculate the cost value of the candidate model to complete the inference task according to the following formula when the state of the candidate model is to be created in the central cloud:

[0032] F3= TR + TD 2+ c 3 / AC

[0033] TR = q 1 / R ( p 1)+ q 2 / R ( p 2)

[0034] Among them, F3 is the cost value of the candidate model to complete the reasoning task when the central cloud is created. TR The transmission delay for uploading the inference task to the central cloud and downloading the feedback results is TD 2 is the creation time of the candidate model created by the central cloud, AC is the inference accuracy, c 3 is a positive parameter, q 1 is the data volume of the reasoning task, q 2 is the data volume of the feedback result, R () is Shannon's formula, p 1 is the upload power, p 2 is the download power.

[0035] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-mentioned reasoning task processing methods by executing the executable instructions.

[0036] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned reasoning task processing methods is implemented.

[0037] According to another aspect of the present disclosure, a computer program product is provided, wherein the computer program product includes a computer program or computer instructions, and the computer program or the computer instructions are loaded and executed by a processor so that a computer implements any of the above-mentioned reasoning task processing methods.

[0038] The technical solution provided by the embodiments of the present disclosure includes at least the following beneficial effects:

[0039] The technical solution provided by the embodiments of the present disclosure can intuitively evaluate the cost of an alternative model to complete an inference task under a corresponding state by using the cost value, by determining the cost value of each alternative model among multiple alternative models to complete the inference task. By determining the alternative model corresponding to the minimum cost value as the target model, a model that can complete the inference task at a lower cost can be selected. The method of processing the inference task according to the state of the target model allows the edge device to complete the inference task at a lower cost. Compared with selecting the default model to process the inference task, this method has higher flexibility and is more suitable for edge devices with limited resources.

[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0042] Figure 1 A schematic diagram of an inference task processing system in one embodiment of the present disclosure is shown.

[0043] Figure 2 A flow chart of a method for processing an inference task in one embodiment of the present disclosure is shown.

[0044] Figure 3 A flow chart of a method for processing an inference task in another embodiment of the present disclosure is shown.

[0045] Figure 4 A schematic diagram of an inference task processing device in an embodiment of the present disclosure is shown.

[0046] Figure 5 A structural block diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0049] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0050] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0051] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0052] Figure 1 A schematic diagram of a reasoning task processing system in an embodiment of the present disclosure is shown, and the system can apply the reasoning task processing method or reasoning task processing device in various embodiments of the present disclosure.

[0053] like Figure 1 As shown, the reasoning task processing system may include: a task generating device 11, an edge device 12 and a central cloud 13.

[0054] The task generation device 11 may generate an inference task and send the inference task to the edge device 12. In one embodiment, the inference task may be generated by an application installed in the task generation device 11.

[0055] In one embodiment, the edge device 11 may include: an inference task processing device 111 , a monitoring module 112 , a storage module 113 , and an interface 114 .

[0056] In one embodiment, the monitoring module 112 may obtain the remaining resources of the edge device 12 and feed the remaining resources back to the reasoning task processing device 111. The embodiments of the present disclosure do not limit which resources the remaining resources include. For example, the remaining resources may include at least one of the following: the remaining resources of a memory, a GPU (Graphic Processing Unit), and a CPU (Central Processing Unit).

[0057] In one embodiment, the monitoring module 112 may monitor the models created on the edge device, obtain the identifier of the model whose status is "created on the edge device", and store the identifier of the model created on the edge device in the storage module 113.

[0058] In one embodiment, in addition to storing the identification of the model that has been created on the edge device, the storage module 113 can also store model information, which includes the identification and reference information of multiple models, and the reference information is used to indicate the creation location when the model needs to be created, and the creation location is the edge device 12 or the central cloud 13. The embodiments of the present disclosure do not limit which models the multiple models specifically include. In one embodiment, the multiple models include models that can be used by the edge device 12 to process reasoning tasks, and models that can be used by the central cloud 13 to process reasoning tasks.

[0059] The embodiments of the present disclosure do not limit what kind of information of the model the reference information is specifically. For example, the reference information may be the size type of the model, the size type being a large model or a small model, the large model being used to indicate that the creation location when the model needs to be created is the central cloud 13, and the small model being used to indicate that the creation location when the model needs to be created is the edge device 12. For another example, the reference information is the parameter amount of the model, if the parameter amount of the model is greater than a preset threshold, it indicates that the creation location when the model needs to be created is the central cloud 13, otherwise, if the parameter amount of the model is not greater than the preset threshold, it indicates that the creation location when the model needs to be created is the edge device 12. For another example, the reference information of the model is the identifier of the edge device 12 or the central cloud 13, the reference information is the identifier of the edge device 12, then it indicates that the creation location when the model needs to be created is the edge device 12, otherwise, the reference information is the identifier of the central cloud 13, then it indicates that the creation location when the model needs to be created is the central cloud 13.

[0060] In one embodiment, the model information may include, in addition to the identification and reference information of the multiple models, the types and / or inference accuracy of the multiple models. The type of the model is used to indicate the type of inference task that the model can handle.

[0061] In one embodiment, the monitoring module 112 may obtain model information by monitoring the historical data of each model processing reasoning task, and store the model information in the storage module 113 .

[0062] The reasoning task processing device 111 can receive the reasoning task sent by the task generation device 11, and receive the data sent by the monitoring module 112, which may include the remaining resources. The reasoning task processing device 111 can also retrieve data from the storage module 113, which may include model information and the identification of the model created in the edge device.

[0063] The reasoning task processing device 111 can determine the cost value of each candidate model in multiple candidate models to complete the reasoning task, wherein different candidate models can have different states or the same state, and the model state can include one of having been created in the edge device, to be created in the edge device, and to be created in the central cloud. The reasoning task processing device 111 can also determine the candidate model corresponding to the minimum cost value as the target model, and process the reasoning task according to the state of the target model.

[0064] If the state of the target model is to be created in the central cloud, the reasoning task processing device 111 can send the reasoning task to the central cloud 13 through the interface 114, and instruct the central cloud 13 to use the target model to process the reasoning task, and download the feedback result for the reasoning task from the central cloud 13.

[0065] The central cloud 13 can receive the reasoning tasks and instructions sent by the reasoning task processing device 111, create an instance of the target model in the central cloud 13 according to the target model in the instruction, and apply the instance to process the reasoning task, and allow the reasoning task processing device 111 in the edge device 12 to download the feedback results of the completion of the reasoning task through the interface 114.

[0066] If the state of the target model is to be created in the edge device, the reasoning task processing device 111 can create an instance of the target model in the edge device 12 and use the instance to process the reasoning task. If the state of the target model is already created in the edge device, the reasoning task processing device 111 can send the reasoning task to the processing queue of the target model to queue up and use the target model to process the reasoning task.

[0067] The task generation device 11, the edge device 12 and the central cloud 13 are connected to each other through a network, which can be a wired network or a wireless network.

[0068] The reasoning task processing device 111 , the monitoring module 112 , the storage module 113 and the interface 114 are connected to each other through a network, and the network may be a wired network or a wireless network.

[0069] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocols. The network is typically the Internet, but may also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. may be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may also be used to replace or supplement the above data communication technologies.

[0070] The task generating device 11 and the edge device 12 may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, etc.

[0071] The task generation device 11 and the edge device 12 may also be servers that provide various services. Optionally, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0072] The central cloud 13 may be a central cloud server cluster corresponding to the edge device 12 .

[0073] Those skilled in the art will know that Figure 1 The number of task generation devices 11 in the example is only illustrative, and any number of task generation devices 11 may be provided according to actual needs. This embodiment of the present disclosure does not limit this.

[0074] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0075] The present disclosure provides a method for processing an inference task, which can be executed by any electronic device with computing and processing capabilities. For example, the electronic device is an edge device.

[0076] Figure 2 A flowchart of a method for processing an inference task in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the reasoning task processing method provided in the embodiment of the present disclosure includes the following S201 to S204.

[0077] S201, receiving an inference task.

[0078] The embodiments of the present disclosure do not limit what kind of reasoning task is. For example, the reasoning task is an image processing task, or a question-answering task, etc.

[0079] In one embodiment, receiving the reasoning task may include: receiving the reasoning task sent by the reasoning task generating device. The reasoning task may be a reasoning task generated by any application in the reasoning task generating device.

[0080] S202, determining the cost of each candidate model among multiple candidate models to complete the reasoning task, the status of each candidate model including one of having been created in the edge device, to be created in the edge device, and to be created in the central cloud.

[0081] The cost value is used to indicate the cost required for the edge device to complete the reasoning task using the alternative model. The larger the cost value, the greater the required cost; the smaller the cost value, the smaller the required cost.

[0082] In one embodiment, the cost value may represent the latency cost and / or resource cost required for the edge device to complete the reasoning task using the alternative model.

[0083] The status of the alternative model is "Created in the edge device", which indicates that the edge device has an instance of the created alternative model. The status of the alternative model is "To be created in the edge device", which indicates that there is no instance of the alternative model in the edge device. If the alternative model needs to be applied, an instance of the alternative model needs to be created in the edge device. The status of the alternative model is "To be created in the central cloud", which indicates that there is no instance of the alternative model in the edge device. If the alternative model needs to be applied, the reasoning task needs to be sent to the central cloud. After the central cloud creates an instance of the alternative model, the instance of the alternative model is applied to process the reasoning task.

[0084] The embodiments of the present disclosure do not limit how to determine the cost of each candidate model among multiple candidate models to complete the reasoning task.

[0085] In one embodiment, determining the cost of each candidate model among multiple candidate models to complete the reasoning task may include: obtaining the cost parameters of each candidate model; and calculating the cost of each candidate model to complete the reasoning task according to the cost parameters of each candidate model. Wherein, when the status of the candidate model is that it has been created on the edge device, the cost parameters include: at least one of the queuing time, the reasoning cost, and the reasoning accuracy; when the status of the candidate model is to be created on the edge device, the cost parameters include: at least one of the creation cost, the creation time, the reasoning cost, and the reasoning accuracy; when the status of the candidate model is to be created on the central cloud, the cost parameters include: at least one of the transmission delay, the creation time, and the reasoning accuracy.

[0086] It should be noted that the cost parameters of the candidate models in the corresponding states are merely exemplary. For example, the cost parameters of the candidate models in the above states may also include inference delay.

[0087] Among them, the queuing time represents the time required to complete the reasoning task in the processing queue that is ahead of the reasoning task received in S201 using the alternative model. The reasoning cost represents the resource cost required for the edge device to complete the reasoning task using the alternative model. The reasoning accuracy is the accuracy of the result obtained by the alternative model processing the reasoning task. The creation cost is used to represent the resource cost required to create the candidate model. The creation time is used to represent the time required to create the alternative model. The transmission delay is used to represent the time required to upload the reasoning task to the central cloud and download the feedback result, which is the result obtained by the central cloud after completing the reasoning task based on the alternative model.

[0088] In one embodiment, the plurality of candidate models include the same model in different states.

[0089] In another embodiment, the plurality of candidate models are all different models.

[0090] The embodiments of the present disclosure do not limit how to obtain the cost parameters of each alternative model. In one embodiment, the edge device stores model information and an identifier of a model whose status is that it has been created on the edge device. The model information includes identifiers and reference information of multiple models. The identifiers of multiple models include an identifier of a model that has been created on the edge device. The reference information is used to indicate the creation location when the model needs to be created. The creation location is the edge device or the central cloud. At this time, obtaining the cost parameters of each alternative model may include: obtaining the type of reasoning task; selecting identifiers of multiple alternative models from the identifiers of multiple models, each alternative model is a model for processing a type of reasoning task among multiple models; determining the status of each alternative model according to the identifier of the model whose status is that it has been created on the edge device, the identifiers of multiple alternative models, and the reference information; obtaining the cost parameters of each alternative model according to the status of each alternative model.

[0091] The type of the inference task may be sent to the edge device along with the inference task, and obtaining the type of the inference task may include: receiving the type of the inference task sent by the task generation device. Alternatively, the inference task includes a type parameter, and the type of the inference task may be determined according to the type parameter.

[0092] The embodiments of the present disclosure do not limit how to select multiple identifiers of candidate models from multiple identifiers of models.

[0093] In one embodiment, in addition to the identifiers and reference information of the multiple models, the model information also includes capability information of the multiple models, and the capability information is used to indicate the types of reasoning tasks that the models can process. For example, the capability information indicates that the types of reasoning tasks that the models can process include two, three, etc. Selecting identifiers of multiple candidate models from the identifiers of the multiple models may include: selecting identifiers of multiple candidate models from the identifiers of the multiple models according to the types of reasoning tasks and the capability information of the multiple models, and the capability information of the candidate models indicates that the types of reasoning tasks that can be processed include the types of reasoning tasks.

[0094] In another embodiment, the model information includes the types of the multiple models in addition to the identifiers and reference information of the multiple models, and one model has one type, which indicates the type of reasoning task processed by the model. Selecting the identifiers of multiple candidate models from the identifiers of the multiple models may include: selecting the identifiers of multiple candidate models from the identifiers of the multiple models according to the type of reasoning task and the types of the multiple models, and the types of the candidate models are consistent with the type of reasoning task.

[0095] In one embodiment, in addition to the identification and reference information of multiple models, the model information may also include the reasoning accuracy of multiple models, and the reasoning accuracy of each candidate model in the multiple candidate models meets the preset conditions, that is, when selecting the identification of the candidate model from the identifications of multiple models, the reasoning accuracy of the model needs to be considered. The embodiments of the present disclosure do not limit what the preset condition is. For example, the preset condition is that the reasoning accuracy is greater than a preset value, and the embodiments of the present disclosure do not limit what the preset value is. For example, the preset value is 0, or 0.8, etc.

[0096] In one embodiment, the status of each alternative model is determined based on the identifier of the model whose status is already created on the edge device, the identifiers of multiple alternative models, and reference information, which may include: if the identifier of the model whose status is already created on the edge device includes the identifier of the alternative model, then the status of the alternative model is already created on the edge device; conversely, if the identifier of the model whose status is already created on the edge device does not include the identifier of the alternative model, then based on the reference information of the alternative model, it is determined that the status of the alternative model is to be deployed on the edge device or to be deployed in the central cloud.

[0097] The embodiments of the present disclosure do not limit what kind of information the reference information is. Figure 1 The description of the reference information in the corresponding system embodiment will not be repeated here.

[0098] After determining the status of each candidate among the multiple candidate models, the cost parameters corresponding to the candidate models can be determined according to the status of each candidate model, thereby obtaining the corresponding cost parameters.

[0099] In one embodiment, the inference cost can be calculated by the number of parameters of the candidate model ( P ) and the number of tokens contained in the reasoning task ( L ), and the parameter quantity of the candidate model can be pre-stored on the edge device, and the number of tokens included in the reasoning task can be directly determined by the specific reasoning task. Obtaining the reasoning cost may include: obtaining the parameter quantity of the candidate model and the number of tokens included in the reasoning task, calculating the product of the parameter quantity and the number of tokens, and determining the reasoning cost based on the product ( aPL ,in, α is a positive parameter, α The specific size can be calculated from the resource cost consumed by the alternative model history when performing reasoning tasks).

[0100] In one embodiment, the queuing time may be determined by the number of inference tasks in the processing queue of the candidate model that precede the inference task received in S201 ( n ), the number of parameters of the alternative model ( P) and the time complexity of the alternative models ( Q ) is determined. Among them, the time complexity of the alternative model ( Q ) and the number of parameters of the alternative models ( P ) is only related to the specific model of the alternative model, which is fixed information of the alternative model. Q and P can be pre-stored in the edge device, and n It can be determined directly based on the processing queue. Getting the queue time can include: n , P , Q ,calculate n , P , Q The product of the three parameters, based on which the queue time is obtained ( nβPQ ,in, β is a positive parameter that can be determined by the time the candidate model has historically used to process inference tasks).

[0101] In one embodiment, the model information may include the inference accuracy of the multiple models in addition to the identification and reference information of the multiple models. Obtaining the inference accuracy of the candidate model may include obtaining the inference accuracy of the candidate model from the model information stored in the edge device.

[0102] In one embodiment, the creation cost is determined by the candidate model, and one candidate model corresponds to a fixed creation cost ( SD ), the specific data of the creation cost of the candidate model can be pre-stored in the edge device. Then obtaining the creation cost of the candidate model can include: directly obtaining the creation cost of the candidate model from the memory.

[0103] It should be noted that when the state of the selected candidate model is to be created in the edge device, the creation cost of the candidate model needs to be less than the remaining resources of the edge device.

[0104] In one embodiment, in the edge device or the central cloud, the creation time is determined only by the time used to create the candidate model. Each candidate model corresponds to a creation time, and the time used can be determined based on the time used to create the candidate model in history. Accordingly, the creation time can be pre-stored in the edge device. Obtaining the creation time of the candidate model can include: directly obtaining the creation time of the candidate model from the memory (wherein, when the state is to be created on the edge device, the creation time can be represented as TD 1. When the status is to be created in the central cloud, the creation time can be expressed as TD 2).

[0105] In one embodiment, the transmission delay is determined by the upload rate of the edge device and the amount of data uploaded when uploading the inference task to the central cloud ( q 1) Download rate and amount of data downloaded when downloading feedback results from the central cloud ( q 2) Determine the upload rate ( R ( p 1), R () is the Shannon formula (communication bandwidth and transmission distance are parameters in this formula), p 1 is the upload power) is determined by the communication bandwidth, transmission distance, and upload power between the edge device and the central cloud. The download rate ( R ( p 2), p 2 is the upload power) is determined by the communication bandwidth, transmission distance, and download power between the edge device and the central cloud.

[0106] Among them, the communication bandwidth and transmission distance between the edge device and the central cloud can be pre-stored in the edge device, and the download rate and upload power can be determined based on the power of the current device when uploading and downloading data, or the download power and upload power in the edge device are both fixed values, and the download rate and upload power can also be pre-stored in the edge device. The amount of data corresponding to the reasoning task can be directly determined based on the reasoning task, and the amount of data corresponding to the feedback result can be determined based on the amount of data of the result obtained by the central cloud in the history of this type of reasoning task based on the alternative model. Accordingly, the amount of data of the feedback result can also be pre-stored in the edge device. Obtaining the transmission delay may include: obtaining the communication bandwidth, transmission distance, download power, upload power, the amount of data of the reasoning task and the amount of data of the feedback result between the edge device and the central cloud, and calculating the transmission delay ( q 1 / R ( p 1)+ q 2 / R ( p 2)).

[0107] The embodiments of the present disclosure do not limit the calculation of the cost value of each candidate model to complete the reasoning task based on the cost parameters of each candidate model.

[0108] In one embodiment, when the status of the candidate model is that it has been created on the edge device, the cost of the candidate model to complete the reasoning task may be positively correlated with the queuing time and the reasoning cost, and negatively correlated with the reasoning accuracy.

[0109] In one embodiment, when the state of the candidate model is that it has been created on the edge device, the types of cost parameters include queuing time, inference cost, and inference accuracy. According to the cost parameters of each candidate model, the cost value of each candidate model to complete the inference task can be calculated, which may include:

[0110] When the status of the candidate model is that it has been created on the edge device, the cost of the candidate model to complete the reasoning task can be calculated according to the following formula 1:

[0111] F1= aPL + nβPQ + c 1 / AC (Formula 1)

[0112] Among them, F1 is the cost value of the candidate model that has completed the inference task when the edge device is created. aPL is the inference cost, nβPQ For the queue time, AC is the inference accuracy, α , β , c 1 are all positive parameters, P is the number of parameters of the alternative model, L is the number of tokens included in the reasoning task, Q is the time complexity of the alternative model, n is the number of inference tasks in the waiting queue of the candidate model.

[0113] In one embodiment, when the state of the candidate model is to be created on the edge device, the cost parameter types include creation cost, creation time, inference cost and inference accuracy. According to the cost parameters of each candidate model, the cost value of each candidate model to complete the inference task is calculated, which may include:

[0114] When the status of the candidate model is to be created on the edge device, the cost of the candidate model to complete the reasoning task can be calculated according to the following formula 2:

[0115] F2= SD + TD 1+ aPL + c 2 / AC (Formula 2)

[0116] Among them, F2 is the cost value of the candidate model to complete the inference task when the edge device is created. SD To create the cost of creating alternative models, TD 1 is the creation time of the alternative model, aPL is the inference cost, AC is the inference accuracy, α , c 2 are all positive parameters, P is the number of parameters of the alternative model, L The number of tokens included in the reasoning task.

[0117] In one embodiment, when the state of the candidate model is to be created in the central cloud, the cost parameter types include transmission delay, creation time and reasoning accuracy. According to the cost parameters of each candidate model, the cost value of each candidate model to complete the reasoning task is calculated, including:

[0118] When the status of the candidate model is to be created in the central cloud, the cost of the candidate model to complete the reasoning task can be calculated according to the following formula 3 and formula 4:

[0119] F3= TR + TD 2+ c 3 / AC (Formula 3)

[0120] TR = q 1 / R ( p 1)+ q 2 / R ( p 2) (Formula 4)

[0121] Among them, F3 is the cost value of the candidate model to complete the reasoning task when the central cloud is created. TR The transmission delay for uploading the inference task to the central cloud and downloading the feedback results is TD 2 is the creation time of the candidate model created by the central cloud, AC is the inference accuracy, c 3 is a positive parameter, q 1 is the data volume of the reasoning task, q 2 is the amount of data for the feedback results, R () is Shannon's formula, p 1 is the upload power, p 2 is the download power.

[0122] When calculating the cost value, the reasoning accuracy of the alternative model is taken into consideration, and the cost value is negatively correlated with the reasoning accuracy. This can make the alternative model with higher reasoning accuracy have a smaller cost value for completing the reasoning task, thereby facilitating the selection of the alternative model with higher reasoning accuracy as the target model to process the reasoning task.

[0123] S203: Determine the candidate model corresponding to the minimum cost value as the target model.

[0124] The candidate model corresponding to the minimum cost value among the cost values ​​of the multiple candidate models is determined as the target model.

[0125] S204: Process the reasoning task according to the state of the target model.

[0126] After determining the target model, if the status of the target model is to be created in the central cloud, the edge device can send the reasoning task to the central cloud, instruct the central cloud to use the target model to process the reasoning task, and download the feedback results for the reasoning task from the central cloud.

[0127] If the state of the target model is to be created on the edge device, the edge device can create an instance of the target model and apply the instance to process the reasoning task.

[0128] If the state of the target model is that it has been created on the edge device, the inference task can be sent to the processing queue of the target model to queue up and use the target model to process the inference task.

[0129] The technical solution provided by the embodiments of the present disclosure can intuitively evaluate the cost of an alternative model to complete an inference task under a corresponding state by using the cost value, by determining the cost value of each alternative model among multiple alternative models to complete the inference task. By determining the alternative model corresponding to the minimum cost value as the target model, a model that can complete the inference task at a lower cost can be selected. The method of processing the inference task according to the state of the target model allows the edge device to complete the inference task at a lower cost. Compared with selecting the default model to process the inference task, this method has higher flexibility and is more suitable for edge devices with limited resources.

[0130] like Figure 3 As shown, the reasoning task processing method in another embodiment of the present disclosure may include S301 to S307. The specific implementation of S301-S307 can be found in Figure 2 The corresponding embodiments will not be described in detail here.

[0131] S301, receiving an inference task.

[0132] S302: Obtain the type of reasoning task.

[0133] S303, selecting multiple candidate models.

[0134] S304: Determine the status of each candidate model.

[0135] S305: Determine the cost of each candidate model to complete the reasoning task according to the status of each candidate model.

[0136] S306: Determine the candidate model corresponding to the minimum cost value as the target model.

[0137] S307: Process the reasoning task according to the state of the target model.

[0138] Based on the same inventive concept, the present disclosure also provides an inference task processing device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0139] Figure 4 A schematic diagram of an inference task processing device in an embodiment of the present disclosure is shown, which is applied to edge devices, such as Figure 4 As shown, the device includes: a receiving module 41, used to receive an inference task; a first determining module 42, used to determine the cost value of each alternative model among multiple alternative models to complete the inference task, and the status of each alternative model includes one of having been created in an edge device, to be created in an edge device, and to be created in a central cloud; a second determining module 43, used to determine the alternative model corresponding to the minimum cost value as the target model; a processing module 44, used to process the inference task according to the status of the target model.

[0140] In one embodiment of the present disclosure, the first determination module 42 is used to obtain the cost parameters of each alternative model; according to the cost parameters of each alternative model, the cost value of each alternative model to complete the reasoning task is calculated; wherein, when the status of the alternative model is that it has been created on the edge device, the cost parameters include: at least one of queuing time, inference cost and inference accuracy; when the status of the alternative model is to be created on the edge device, the cost parameters include: at least one of creation cost, creation time, inference cost and inference accuracy; when the status of the alternative model is to be created on the central cloud, the cost parameters include: at least one of transmission delay, creation time and inference accuracy.

[0141] In one embodiment of the present disclosure, an edge device stores model information and an identifier of a model whose status is that it has been created on the edge device, the model information includes identifiers and reference information of multiple models, the identifiers of multiple models include the identifier of the model that has been created on the edge device, and the reference information is used to indicate the creation location when the model needs to be created, and the creation location is the edge device or the central cloud; a first determination module 42 is used to obtain cost parameters of each candidate model, including: obtaining the type of reasoning task; selecting identifiers of multiple candidate models from the identifiers of multiple models, each candidate model is a model for processing type reasoning tasks among multiple models; determining the status of each candidate model according to the identifier of the model whose status is that it has been created on the edge device, the identifiers of multiple candidate models and the reference information; obtaining the cost parameters of each candidate model according to the status of each candidate model.

[0142] In one embodiment of the present disclosure, when the status of the alternative model is that it has been created on the edge device, the types of cost parameters include queuing time, inference cost and inference accuracy; the first determination module 42 is used to calculate the cost value of the alternative model to complete the inference task according to the above formula 1 when the status of the alternative model is that it has been created on the edge device.

[0143] In one embodiment of the present disclosure, when the status of the alternative model is to be created on the edge device, the types of cost parameters include creation cost, creation time, inference cost and inference accuracy; the first determination module 42 is used to calculate the cost value of the alternative model to complete the inference task according to the above formula 2 when the status of the alternative model is to be created on the edge device.

[0144] In one embodiment of the present disclosure, when the status of the alternative model is to be created in the central cloud, the types of cost parameters include transmission delay, creation time and inference accuracy; the first determination module 42 is used to calculate the cost value of the alternative model to complete the reasoning task according to the above formulas 3 and 4 when the status of the alternative model is to be created in the central cloud.

[0145] The technical solution provided by the embodiments of the present disclosure can intuitively evaluate the cost of an alternative model to complete an inference task under a corresponding state by using the cost value, by determining the cost value of each alternative model among multiple alternative models to complete the inference task. By determining the alternative model corresponding to the minimum cost value as the target model, a model that can complete the inference task at a lower cost can be selected. The method of processing the inference task according to the state of the target model allows the edge device to complete the inference task at a lower cost. Compared with selecting the default model to process the inference task, this method has higher flexibility and is more suitable for edge devices with limited resources.

[0146] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0147] Refer to the following Figure 5 The electronic device 500 according to this embodiment of the present disclosure is described. Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0148] like Figure 5As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510).

[0149] The storage unit stores program codes, which can be executed by the processing unit 510, so that the processing unit 510 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Specific Implementation Methods” section of this specification.

[0150] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202 , and may further include a read-only storage unit (ROM) 5203 .

[0151] The storage unit 520 may also include a program / utility 5204 having a set (at least one) of program modules 5205, such program modules 5205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0152] Bus 530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0153] The electronic device 500 may also communicate with one or more external devices 540 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 500, and / or any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 560. Figure 5 As shown, the network adapter 560 communicates with other modules of the electronic device 500 via the bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0154] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0155] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above-mentioned method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Specific Implementation Methods” section of this specification.

[0156] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0158] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0159] In specific implementation, the program code for performing the operation of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0160] In an exemplary embodiment of the present disclosure, a computer program product is also provided, which includes a computer program or computer instructions, and the computer program or computer instructions are loaded and executed by a processor to enable a computer to implement the steps of various exemplary embodiments of the present disclosure described in the above "Specific Implementation" section of this specification.

[0161] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0162] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0163] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the example implementation modes described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation mode of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation mode of the present disclosure.

[0164] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope of the present disclosure is indicated by the appended claims.

Claims

1. A method for processing an inference task, characterized in that: Applied to edge devices, including: receiving an inference task; Determine a cost value of each candidate model among a plurality of candidate models to complete the reasoning task, wherein the status of each candidate model includes one of being created on an edge device, being created on an edge device, and being created on a central cloud; The alternative model corresponding to the minimum cost value is determined as the target model; Processing the reasoning task according to the state of the target model; The edge device stores model information and an identifier of a model that has been created on the edge device, the model information includes identifiers and reference information of multiple models, the identifiers of the multiple models include the identifier of the model that has been created on the edge device, and the reference information is used to indicate a creation location when a model needs to be created, and the creation location is an edge device or a central cloud; The step of determining the cost of each candidate model among the plurality of candidate models to complete the reasoning task includes: Obtaining the type of the reasoning task; Selecting identifiers of the plurality of candidate models from the identifiers of the plurality of models, each candidate model being a model among the plurality of models for processing the type of reasoning task; Determine the status of each candidate model according to the identifier of the model whose status is already created on the edge device, the identifiers of the multiple candidate models and the reference information; According to the status of each candidate model, obtain the cost parameters of each candidate model; According to the cost parameters of each candidate model, the cost value of each candidate model to complete the reasoning task is calculated.

2. The method according to claim 1, characterized in that: When the status of the alternative model is that it has been created on the edge device, the cost parameters include: at least one of queuing time, inference cost and inference accuracy; when the status of the alternative model is to be created on the edge device, the cost parameters include: at least one of creation cost, creation time, inference cost and inference accuracy; when the status of the alternative model is to be created in the central cloud, the cost parameters include: at least one of transmission delay, creation time and inference accuracy.

3. The method according to claim 1, characterized in that: The model information also includes the inference accuracy of multiple models; The inference accuracy of each candidate model in the multiple candidate models meets the preset conditions.

4. The method according to claim 2, characterized in that: When the status of the candidate model is that it has been created on the edge device, the types of cost parameters include queuing time, inference cost, and inference accuracy; the cost value of each candidate model to complete the inference task is calculated according to the cost parameters of each candidate model, including: When the status of the candidate model is that it has been created on the edge device, the cost of the candidate model to complete the reasoning task is calculated according to the following formula: F1= αPL + nβP + γ 1 / AC Among them, F1 is the cost value of the candidate model that has completed the reasoning task when the edge device is created, αPL is the inference cost, nβP For the queue time, AC is the inference accuracy, α , β , γ 1 are all positive parameters, P is the number of parameters of the alternative model, L is the number of tokens contained in the reasoning task, Q is the time complexity of the alternative model, n is the number of inference tasks in the waiting queue of the candidate model.

5. The method according to claim 2, characterized in that: When the state of the candidate model is to be created on the edge device, the types of cost parameters include creation cost, creation time, inference cost and inference accuracy; the cost value of each candidate model to complete the inference task is calculated according to the cost parameters of each candidate model, including: When the state of the candidate model is to be created on the edge device, the cost of the candidate model to complete the reasoning task is calculated according to the following formula: F2= SD + TD 1+ αPL + γ 2 / AC Among them, F2 is the state of the candidate model, which is the cost value of completing the reasoning task when the edge device is created. SD To create the cost of creating alternative models, TD 1 is the creation time of the alternative model, αPL is the inference cost, AC is the inference accuracy, α , γ 2 are all positive parameters, P is the number of parameters of the alternative model, L The number of tokens included in the reasoning task.

6. The method according to claim 2, characterized in that When the candidate model is in the state of being created in the central cloud, the types of cost parameters include transmission delay, creation time and inference accuracy; The step of calculating the cost of each candidate model to complete the reasoning task according to the cost parameters of each candidate model includes: When the status of the candidate model is to be created in the central cloud, the cost of the candidate model to complete the reasoning task is calculated according to the following formula: F3= TR + TD 2+ γ 3 / AC TR = q 1 / R ( p 1)+ q 2 / R ( p 2) Among them, F3 is the cost value of the candidate model to complete the reasoning task when the central cloud is created. TR The transmission delay for uploading the inference task to the central cloud and downloading the feedback results is TD 2 is the creation time of the candidate model created by the central cloud, AC is the inference accuracy, γ 3 is a positive parameter, q 1 is the data volume of the reasoning task, q 2 is the data volume of the feedback result, R () is Shannon's formula, p 1 is the upload power, p 2 is the download power.

7. A reasoning task processing device, characterized in that: Applied to edge devices, including: A receiving module, used for receiving reasoning tasks; A first determination module is used to determine the cost of each candidate model among multiple candidate models to complete the reasoning task, and the status of each candidate model includes one of being created on the edge device, being created on the edge device, and being created on the central cloud; A second determination module is used to determine the candidate model corresponding to the minimum cost value as the target model; A processing module, used for processing the reasoning task according to the state of the target model; The edge device stores model information and an identifier of a model that has been created on the edge device, the model information includes identifiers and reference information of multiple models, the identifiers of the multiple models include the identifier of the model that has been created on the edge device, and the reference information is used to indicate a creation location when a model needs to be created, and the creation location is an edge device or a central cloud; Among them, the first determination module is also used to: obtain the type of the reasoning task; select the identifiers of the multiple candidate models from the identifiers of the multiple models, each candidate model is a model among the multiple models used to process the reasoning task of the type; determine the status of each candidate model according to the identifier of the model whose status is created on the edge device, the identifiers of the multiple candidate models and the reference information; obtain the cost parameters of each candidate model according to the status of each candidate model; and calculate the cost value of each candidate model to complete the reasoning task according to the cost parameters of each candidate model.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the reasoning task processing method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for processing an inference task described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and the computer program or computer instructions are loaded and executed by a processor to enable a computer to implement the reasoning task processing method according to any one of claims 1 to 6.

Images