Method, device, equipment, medium and product for calling pre-training model service

Abstract

The present disclosure relates to the technical field of computer technology, large model technology, large language model technology and artificial intelligence technology, in particular to a pre-training model service calling method and device, equipment, medium and product, the method comprising: obtaining a model edge gateway corresponding to an end-side device; when receiving a model service calling request initiated by the end-side device, if it is found through the model edge gateway that there is pre-training model cache information matched with the model service calling request in the cache server, returning the pre-training model cache information to the end-side device; otherwise, obtaining an intranet environment where the model edge gateway is located, sending the model service calling request to a target model server in the intranet environment, and returning a target model service matched with the model service calling request through the target model server. Through the implementation of the technical solution, the calling time is reduced, the application response speed of the end-side device is improved, the inference calculation of the model service is saved, and the resource utilization rate is optimized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, in particular to a method and apparatus for calling a pre-trained model service. BACKGROUND

[0002] Currently, model gateway services are mainly deployed in central areas. When end-side devices in different edge areas call model services, they all need to pass through the central network, even if the cache is hit. The delay of the end-side devices to the central network will be high, thereby causing the application response speed of the end-side devices in the edge area to be low. SUMMARY

[0003] Therefore, the present disclosure provides a method and apparatus for calling a pre-trained model service to solve the problem of low application response speed caused by high latency of end-side devices in edge areas calling model services.

[0004] In a first aspect, the present disclosure provides a method for calling a pre-trained model service, comprising: obtaining a model edge gateway corresponding to an end-side device, the model edge gateway and the end-side device being deployed in the same edge area; when receiving a model service calling request initiated by the end-side device, querying, through the model edge gateway, whether there is pre-trained model cache information matched with the model service calling request in a cache server, the cache server and the model edge gateway being deployed in the same edge area; if the cache server has the pre-trained model cache information matched with the model service calling request, returning the pre-trained model cache information to the end-side device; if the cache server does not have the pre-trained model cache information matched with the model service calling request, obtaining an intranet environment where the model edge gateway is located; sending the model service calling request to a target model server in the intranet environment, and returning, through the target model server, a target model service matched with the model service calling request.

[0005] In a second aspect, the present disclosure provides a pre-trained model service calling device, comprising: an obtaining module configured to obtain a model edge gateway corresponding to an end-side device, the model edge gateway and the end-side device being deployed in a same edge area; a cache querying module configured to, when receiving a model service calling request initiated by the end-side device, query whether there is pre-trained model cache information matched with the model service calling request in a cache server through the model edge gateway, the cache server and the model edge gateway being deployed in the same edge area; a model feedback module configured to, if the cache server has the pre-trained model cache information matched with the model service calling request, return the pre-trained model cache information to the end-side device; an intranet information obtaining module configured to, if the cache server does not have the pre-trained model cache information matched with the model service calling request, obtain an intranet environment in which the model edge gateway is located; and a model matching module configured to send the model service calling request to a target model server in the intranet environment, and return a target model service matched with the model service calling request through the target model server.

[0006] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, which are communicatively connected with each other, and the memory stores computer instructions; the processor executes the computer instructions to perform the pre-trained model service calling method of the first aspect or any of the corresponding embodiments thereof.

[0007] In a fourth aspect, the present disclosure provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer perform the pre-trained model service calling method of the first aspect or any of the corresponding embodiments thereof.

[0008] In a fifth aspect, the present disclosure provides a computer program product, which comprises computer instructions, and the computer instructions are used to make a computer perform the pre-trained model service calling method of the first aspect or any of the corresponding embodiments thereof.

[0009] The application provides a pre-training model service calling method, device, equipment, medium and product. A model edge gateway is deployed in an edge area where an end-side device is located, pre-training model cache information matched with a model service calling request is queried from a cache server in the edge area through the model edge gateway, and when it is determined that the cache server has the pre-training model cache information matched with the model service calling request, the pre-training model cache information is directly returned to the end-side device, so that the model gateway and the related cache server are sunk from the center to the edge area closer to the end-side device, the end-side device can access and infer locally and provide local cache when calling the model service, the calling time is greatly reduced compared with calling the model from the central cache, and the application response speed of the end-side device is improved. When it is determined that the cache server does not have the pre-training model cache information matched with the model service calling request, inference is performed on a model server in the same internal network environment as the model edge gateway to obtain a corresponding target model service, the time delay is greatly reduced compared with the inference after returning to the central model server. Meanwhile, the model inference calculation is saved, the resource utilization is optimized, the throughput is improved, the calling time is greatly reduced and the response speed is greatly improved in the similar query scene. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the specific embodiments of the present disclosure or the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0011] Figure 1 is a model calling method schematic diagram in the related art;

[0012] Figure 2 is a flowchart of a pre-training model service calling method according to an embodiment of the present disclosure;

[0013] Figure 3 is a specific schematic diagram of model calling according to an embodiment of the present disclosure;

[0014] Figure 4 is a flowchart of another pre-training model service calling method according to an embodiment of the present disclosure;

[0015] Figure 5 is a flowchart of another pre-training model service calling method according to an embodiment of the present disclosure;

[0016] Figure 6is a model calling schematic diagram under a delay shortest priority strategy according to an embodiment of the present disclosure;

[0017] Figure 7 is a model calling schematic diagram under a throughput maximum priority strategy according to an embodiment of the present disclosure;

[0018] Figure 8 is a model calling schematic diagram under a resource consumption optimization strategy according to an embodiment of the present disclosure;

[0019] Figure 9 is a model index issuing schematic diagram according to an embodiment of the present disclosure;

[0020] Figure 10 is a structural block diagram of a pre-trained model service calling apparatus according to an embodiment of the present disclosure;

[0021] Figure 11 is a hardware structure schematic diagram of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described below in a clear and complete manner with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0023] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario and the like of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0024] For example, when responding to the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be executed will need to obtain and use the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the electronic device, application program, server or storage medium and the like software or hardware that execute the operation of the technical solutions of the present disclosure according to the prompt information.

[0025] As an optional but not limited implementation manner, the manner of sending prompt information to the user in response to the active request of the user may, for example, be a pop-up window manner, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide the personal information to the electronic device.

[0026] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other methods that meet relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0027] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.

[0028] As the use of large language models (LLMs) in various industries and applications on various end-side devices continues to increase, it becomes increasingly important to improve the response speed and throughput of LLMs to enable them to operate at a larger scale and with higher cost efficiency. Therefore, large model gateways have emerged, and large model gateways play a crucial role in connecting various applications. They provide important functions such as greater range of LLM support, query acceleration, cache mechanism construction, monitoring, and security protection.

[0029] The large model gateway service in the related art is mainly deployed in the center, as shown in Figure 1 The capabilities it provides mainly include: 1) proxying large models from different public cloud providers; 2) monitoring and log tracking; 3) API authentication and authorization; 4) flow limiting strategy; 5) security protection, etc. Based on this, the overall link of its implementation is roughly as follows: 1) the end-side device initiates a large model service call request; 2) the request is forwarded to the center gateway server; 3) after the gateway server authentication and authorization pass, the cache recall is performed based on the vector similarity, and the cache result is directly returned if it hits, otherwise it is returned to the large model of each different public cloud vendor; 4) the corresponding request is responded by the large model of each public cloud vendor, and then returned to the end-side device through the gateway. However, in this process, the end-side devices in different edge areas call the large model through the center network, and even if it hits the cache, the latency from the end-side device to the center network will be high. As a result, for scenarios with high latency requirements, performing inference in the center will to some extent affect the response speed of the end-side device application. Moreover, the application program of the end-side device usually uses multiple large models from different providers, which will consume a lot of resources during the calling process, and when frequently calling a specific model of a vendor, it often produces an error that the request cannot be processed temporarily.

[0030] Based on this, the technical solution of the present disclosure sinks the model gateway service, related cache infrastructure, and model server from the center to the edge area where the end-side device is located, to realize near access, near inference, and near cache, greatly reducing the latency of the end-side device calling the model service. In addition, by tracking the changes in latency and resource consumption of each model service, the model service can be automatically optimized according to actual needs, reducing the calling latency and resource consumption of the model service while ensuring the availability of the end-side device application.

[0031] According to the embodiments of the present disclosure, a pre-trained model service calling method is provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0032] In the present embodiment, a pre-trained model service calling method is provided, which can be used in computer devices such as gateways, servers, etc. Figure 2 The flowchart of the pre-trained model service calling method according to the embodiments of the present disclosure is shown in FIG. 1, which includes the following steps: Figure 2

[0033] In step S101, the model edge gateway corresponding to the end-side device is obtained, and the model edge gateway and the end-side device are deployed in the same edge area.

[0034] The model edge gateway is deployed in the edge area where the end-side device is located, i.e., the model edge gateway and the end-side device are deployed in the same edge area, which is a region with similar geographical locations, such as the same city A. The end-side device refers to a device running in the edge part of data processing, i.e., a device close to the data source or the end user. The end-side device can be a personal device such as a tablet computer or a notebook computer, or an Internet of Things device such as a smart home device or a wearable device. A smart application is deployed on the end-side device, which uses multiple network models (such as large language models, etc.) of the same manufacturer or different manufacturers.

[0035] For end-side devices in different edge areas, the model edge gateway deployed in the edge area is determined by detecting the edge area where the end-side device is located, so that when the end-side device initiates a model service calling request, the model service calling request is routed to the model edge gateway through the gateway domain name.

[0036] In step S102, when receiving the model service calling request initiated by the end-side device, the pre-trained model cache information matched with the model service calling request is queried in the cache server through the model edge gateway, and the cache server and the model edge gateway are deployed in the same edge area.

[0037] The cache server is a server in communication connection with the model edge gateway and in the same edge area and the same intranet environment. In the server, there is stored model information called by the end-side device in the historical time, i.e., pre-trained model cache information. The model service calling request is a request initiated by the application in the end-side device when the model needs to be called.

[0038] Specifically, as shown in FIG. 2, the model edge gateway is deployed in the same edge area as the end-side device, and the cache server is deployed in the same edge area as the model edge gateway.​Figure 3 As shown, when the end-side device initiates a model service call request to the model edge gateway through the gateway domain name, the model edge gateway can receive the model service call request initiated by the end-side device, parse the information carried in the model service call request, and determine the model information requested by the model service call request. Then, according to the information carried in the model service call request, the existing cache information in the cache server is queried, and it is determined whether there is pre-trained model cache information matching the same through similarity calculation.

[0039] If the cache server has pre-trained model cache information matching the model service call request, the pre-trained model cache information is returned to the end-side device.

[0040] If the cache server has pre-trained model cache information matching the model service call request, it indicates that the end-side device has previously requested related model services, and at this time, the pre-trained model cache information can be directly called from the cache server and returned to the end-side device, as shown in Figure 3 If the cache server does not have pre-trained model cache information matching the model service call request, other operations are performed, such as calling from the model server in the edge area and matching again, which is not limited here.

[0041] The pre-trained model service call method provided in this embodiment deploys a model edge gateway in the edge area where the end-side device is located, so as to query the pre-trained model cache information matching the model service call request from the cache server in the edge area where the model edge gateway is located. When it is determined that the cache server has pre-trained model cache information matching the model service call request, the pre-trained model cache information is directly returned to the end-side device, thereby realizing the sinking of the model gateway and the related cache server from the center to the edge area closer to the end-side device. The end-side device can access nearby and infer nearby when calling a model, and provide nearby cache. Compared with calling a model from the center cache, the calling time is greatly reduced, and the application response speed of the end-side device is improved. At the same time, the model inference calculation is saved, the resource utilization rate is optimized, and the throughput is improved. In the scene with more similar queries, the access is nearby, which can greatly reduce the calling time and greatly improve the response speed.

[0042] In this embodiment, a pre-trained model service call method is provided, which can be used in computer devices such as gateways, servers, and the like, Figure 4 is a flowchart of the pre-trained model service call method according to the embodiments of the present disclosure, as shown in Figure 4 The flowchart includes the following steps:

[0043] At step S201, a model edge gateway corresponding to the end-side device is acquired, and the model edge gateway and the end-side device are deployed in the same edge area. For details, please refer to the related description of the corresponding step of the above-mentioned embodiment, which will not be repeated here.

[0044] At step S202, when receiving the model service call request initiated by the end-side device, it is queried through the model edge gateway whether there is pre-trained model cache information matched with the model service call request in the cache server, and the cache server and the model edge gateway are deployed in the same edge area. For details, please refer to the related description of the corresponding step of the above-mentioned embodiment, which will not be repeated here.

[0045] At step S203, if the cache server has pre-trained model cache information matched with the model service call request, the pre-trained model cache information is returned to the end-side device. For details, please refer to the related description of the corresponding step of the above-mentioned embodiment, which will not be repeated here.

[0046] At step S204, if the cache server does not have pre-trained model cache information matched with the model service call request, the internal network environment where the model edge gateway is located is acquired.

[0047] The internal network environment where the model edge gateway is located refers to a network infrastructure supporting edge computing and data processing. Specifically, switches, routers, firewalls and other network devices are deployed in the internal network environment, so as to connect the model edge gateway, model server, cache server, end-side device and the like accessing the internal network environment through various network devices. At the same time, the internal network environment is also deployed with corresponding access control, encryption, defense system and other security measures to ensure the data security of the internal network environment.

[0048] If the cache server does not have pre-trained model cache information matched with the model service call request, the corresponding model information needs to be requested from the nearest model server, at which time the network topology where the model edge gateway is located needs to be parsed to acquire the internal network environment where the model edge gateway is located.

[0049] At step S205, the model service call request is sent to the target model server in the internal network environment, and the target model service matched with the model service call request is returned through the target model server.

[0050] The target model server refers to the model server feeding back the model information to the end-side device, and the target model service refers to the model information required by the model service call request and inferred by the target model server. After determining that there is no corresponding pre-trained model cache information in the cache server, the model edge gateway routes the model service call request to the target model server in the internal network environment, so that the target model server performs inference calculation according to the model service call request and returns the corresponding target model service.

[0051] The calling method of the pre-trained model service provided by the embodiment determines that the cache server does not have pre-trained model cache information matched with the model service calling request, and then performs inference on the model server in the intranet environment to obtain the corresponding target model service. Compared with the inference after the source is returned to the central model server, the time delay is greatly reduced.

[0052] In some optional embodiments, as shown in Figure 5 The step S205 includes the following steps:

[0053] In step S2051, a plurality of model servers deployed in the intranet environment and a load request quantity of each model server are obtained.

[0054] The load request quantity is the number of model service calling requests to be processed currently loaded by the model server. In the intranet environment, a plurality of model servers are deployed to provide model inference services. For each model server, model service calling requests from different end-side devices can be received, and the number and processing state of the model service calling requests received by each model server are updated in real time to determine the model service calling requests that have been processed and the model service calling requests to be processed, so as to obtain the load request quantity of each model server.

[0055] In step S2052, based on the load request quantity, a target model server is determined from each model server according to load balancing.

[0056] When a new model service calling request enters the intranet environment, the load request quantity corresponding to each model server is analyzed, and one or more model servers currently allocable for the model service calling request are determined according to the load balancing. If there is only one model server currently allocable for the model service calling request, the model server is determined as the target model server. If there are multiple model servers currently allocable for the model service calling request, the optimal model server is determined based on the data processing speed, memory usage and other parameter information of the model server, and the optimal model server is determined as the target model server.

[0057] In step S2053, a model service calling strategy corresponding to the end-side device and a plurality of candidate model services matched with the model service calling request are obtained.

[0058] The model service calling strategy is a strategy for the end-side device to call the model service, such as delay shortest first, throughput maximum first, and the like. The model service calling strategy corresponding to the end-side device is pre-configured, for example, the end-side device 1 corresponds to the delay shortest first strategy, and the end-side device 2 corresponds to the throughput maximum first strategy. When the end-side device initiates a model service calling request, the model service calling strategy is encapsulated in the model service calling request. Therefore, when the model edge gateway receives the model service calling request, the model service calling strategy corresponding to the end-side device can be obtained by analyzing the content carried in the model service calling request.

[0059] The candidate model service is model information capable of providing the required function of the model service calling request. For the same model service calling request, there can be multiple models capable of meeting its requirements. Specifically, when the model server performs inference calculation according to the model service calling request, all model information meeting the model service calling request, i.e., multiple candidate model services, are filtered out.

[0060] In step S2054, the target model service is determined from the multiple candidate model services according to the model service calling strategy.

[0061] According to the model service calling strategy, the model service parameters of each candidate model service under the model service calling strategy are determined, such as model service delay, model service throughput, and the like. The model service parameters corresponding to each candidate model service are sorted to generate a corresponding parameter sorting result, and the model service calling priority is represented by the parameter sorting result. Then, the target model service required is filtered out from the multiple candidate model services according to the model service calling priority.

[0062] The calling method of the pre-trained model service provided in this embodiment determines the target model server from each model server according to load balancing, so as to process the model service calling request through the target model server, avoids the request overload of the model server in the internal network environment, optimizes the resource utilization of each model server in the internal network environment, minimizes the throughput and the response time, thereby greatly shortens the delay of the end-side device calling the model service, and improves the application response speed. By configuring multiple candidate model services matched with the model service calling request, the problem of too many calling failures affecting user experience is avoided. When there are multiple candidate model services, the target model service is automatically optimized according to the model service calling strategy, which greatly improves the application response speed and application availability of the edge area end-side device.

[0063] In some optional embodiments, when the model service calling strategy is the delay shortest first strategy, the above step S2054 can include:

[0064] Step a1, obtain the model service latency of each candidate model service under the latency shortest first strategy.

[0065] Step a2, sort the model service latency, and determine the first call priority of each candidate model service based on the delay sorting result.

[0066] Step a3, determine the target model service according to the first call priority.

[0067] The model service latency is the time consumed by the model to output the result of task inference. The model service latency of each candidate model service is determined based on the processing speed of the candidate model service. Specifically, the center gateway can monitor the task processing state of each candidate model service, determine the average latency required for task processing of each candidate model service in real time, and synchronize the model service latency corresponding to each candidate model service to the model server in each edge area in real time.

[0068] The obtained model service latency of each candidate model service under the latency shortest first strategy is sorted to obtain a delay sorting result of the model service latency from small to large. The call priority of each candidate model service in calling can be determined through the delay sorting result.

[0069] Taking a large model as an example, the model service latency can be represented by the first token latency (TTFT). As shown in Figure 6 , the end-side device 1 enables the TTFT shortest first strategy, and the candidate model services are LLM1, LLM2 and LLM3. When the end-side device 1 accesses the model edge gateway to call the large model, the three candidate model services are sorted based on the shortest average TTFT, and are called in turn. For example, LLM3 is 100 milliseconds, LLM1 is 150 milliseconds, and LLM2 is 200 milliseconds. Since LLM3 has the shortest TTFT, LLM3 is called first. If LLM3 fails to be called, LLM1 with the second shortest TTFT is automatically called, and so on.

[0070] The calling method of the pre-trained model service provided in the embodiment supports automatic optimization of the target model service according to the latency shortest first strategy, thereby ensuring the application availability of the end-side device under the latency shortest first strategy and improving the application response speed of the end-side device.

[0071] In some optional embodiments, when the model service calling strategy is the throughput maximum first strategy, the above step S2054 can include:

[0072] Step b1, obtain the model service throughput of each candidate model service under the throughput maximum first strategy.

[0073] Step b2: Sort the throughput of each model service and determine the second call priority of each candidate model service based on the throughput ranking results.

[0074] Step b3: Determine the target model service according to the second call priority.

[0075] Model service throughput refers to the amount of data a model can process per unit of time. The model service throughput of each candidate model service is determined based on its processing speed. Specifically, the central gateway can monitor the task processing status of each candidate model service, determine the model service throughput generated by each candidate model service for task processing in real time, and synchronize the model service throughput corresponding to each candidate model service to the model servers in each edge region in real time.

[0076] The throughput of each candidate model service is sorted under the maximum throughput priority strategy, and the throughput ranking results are obtained from large to small. The invocation priority of each candidate model service can be determined by the throughput ranking results.

[0077] Taking a large model as an example, the service throughput of this model can be characterized by tokens per second (TPS). Figure 7 As shown, edge device 2 has enabled a maximum TPS priority strategy, with candidate model services being LLM1, LLM2, and LLM3. When edge device 2 accesses the model edge gateway to call a large model, it sorts the three candidate model services based on their average maximum TPS and calls them sequentially. For example, LLM3 has a TPS of 20, LLM2 has a TPS of 15, and LLM1 has a TPS of 10. Since LLM3 has the shortest TPS, it is called first. If the LLM3 call fails, LLM2 with the second highest TPS is automatically called, and so on.

[0078] The method for calling the pre-trained model service provided in this embodiment supports automatic selection of the target model service according to the maximum throughput priority strategy, thereby ensuring the application availability of the end device under the maximum throughput priority strategy and ensuring the application response speed of the end device.

[0079] In some optional implementations, when the model service invocation strategy is a resource consumption optimization strategy, step S2054 above may include:

[0080] Step c1: Obtain the resource consumption of each candidate model service under the resource consumption optimization strategy.

[0081] Step c2, ranking the resource consumption of each model service resource, and determining the third invocation priority of each candidate model service based on the ranking result of resource consumption.

[0082] Step c3, determining the target model service according to the third invocation priority.

[0083] The model service resource consumption is the resource consumed in the model using process. The model service resource consumption of each candidate model service is determined based on the intrinsic attributes of the candidate model service. Specifically, the central gateway can analyze the attribute information of each candidate model service, determine the model service resource consumption of each candidate model service, and synchronize the model service resource consumption corresponding to each candidate model service to the model server in each edge area in real time.

[0084] The model service resource consumption of each candidate model service under the resource consumption optimization strategy is ranked to obtain a resource consumption ranking result from low to high. The invocation priority of each candidate model service when invoked can be determined through the resource consumption ranking result.

[0085] Taking a large model as an example, the model service resource consumption can be represented by the model using cost. As shown in Figure 8 , the end-side device 3 enables the resource consumption optimization strategy, and the candidate model services are LLM1, LLM2 and LLM3. When the end-side device 3 accesses the model edge gateway to invoke the large model, the three candidate model services are ranked based on the model service resource consumption, and are invoked in turn. For example, the model service using cost of LLM3 is 0.0003 yuan per thousand tokens, the model service using cost of LLM2 is 0.005 yuan per thousand tokens, and the model service using cost of LLM1 is 0.001 yuan per thousand tokens. Since the model service using cost of LLM3 is the least, LLM3 is invoked first. If LLM3 fails to be invoked, LLM1 with the second least model service using cost is automatically invoked, and so on.

[0086] The invocation method of the pre-trained model service provided in this embodiment supports automatic optimization of the target model service according to the resource consumption optimization strategy, thereby reducing the resource consumption while ensuring the application availability of the end-side device under the resource consumption optimization strategy.

[0087] In some optional embodiments, as shown in Figure 9 , the model edge gateway corresponding to the central gateway issues the model indicators corresponding to the candidate model services in real time, including the model service latency, the model service throughput, and the resource consumption, etc.

[0088] When the central gateway service is deployed, the model indicators of the model (such as model service delay TTFT, model service throughput TPS, etc.) and the model service resource consumption (such as model cost) of each vendor are set with corresponding tracking services, so that the model delay, model service throughput, and resource consumption of each model are synchronously issued in real time to the model servers in the edge area where each model edge gateway is located through the central gateway configuration service, to provide the decision target model service.

[0089] The calling method of the pre-trained model service provided in the embodiment ensures the accuracy of each model indicator in the edge area by issuing the model indicators corresponding to the candidate model service in real time through the central gateway corresponding to the model edge gateway, and improves the decision accuracy of the target model service.

[0090] In the embodiment, a calling device of a pre-trained model service is also provided, which is used to implement the above-described embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is contemplated.

[0091] The calling device of the pre-trained model service provided in the embodiment, as shown in Figure 10 includes:

[0092] The acquisition module 301 is configured to acquire a model edge gateway corresponding to an end-side device, the model edge gateway and the end-side device being deployed in the same edge area.

[0093] The cache query module 302 is configured to, when receiving a model service calling request initiated by the end-side device, query whether there is pre-trained model cache information matched with the model service calling request in a cache server through the model edge gateway, the cache server and the model edge gateway being deployed in the same edge area.

[0094] The model feedback module 303 is configured to, if the cache server has the pre-trained model cache information matched with the model service calling request, return the pre-trained model cache information to the end-side device.

[0095] The intranet information acquisition module 304 is configured to, if the cache server does not have the pre-trained model cache information matched with the model service calling request, acquire an intranet environment where the model edge gateway is located.

[0096] The model matching module 305 is configured to send the model service calling request to a target model server in the intranet environment, and return a target model service matched with the model service calling request through the target model server.

[0097] In some optional embodiments, the model matching module comprises:

[0098] a load obtaining unit configured to obtain a plurality of model servers deployed in the intranet environment and a load request amount of each model server.

[0099] a load balancing unit configured to determine a target model server from the plurality of model servers according to load balancing based on the load request amount.

[0100] In some optional embodiments, the model matching module further comprises:

[0101] a policy obtaining unit configured to obtain a model service invocation policy corresponding to the end-side device and a plurality of candidate model services matched with the model service invocation request.

[0102] a model determining unit configured to determine a target model service from the plurality of candidate model services according to the model service invocation policy.

[0103] In some optional embodiments, when the model service invocation policy is a shortest-latency-first policy, the model determining unit comprises:

[0104] a model service latency obtaining sub-unit configured to obtain a model service latency of each candidate model service under the shortest-latency-first policy.

[0105] a first priority determining sub-unit configured to sort the model service latencies, and determine a first invocation priority of each candidate model service based on a latency sorting result.

[0106] a first determining sub-unit configured to determine the target model service according to the first invocation priority.

[0107] In some optional embodiments, when the model service invocation policy is a maximum-throughput-first policy, the model determining unit comprises:

[0108] a throughput obtaining sub-unit configured to obtain a model service throughput of each candidate model service under the maximum-throughput-first policy.

[0109] a second priority determining sub-unit configured to sort the model service throughputs, and determine a second invocation priority of each candidate model service based on a throughput sorting result.

[0110] a second determining sub-unit configured to determine the target model service according to the second invocation priority.

[0111] In some optional embodiments, when the model service invocation policy is a resource consumption optimization policy, the model determining unit comprises:

[0112] The resource consumption obtaining subunit is configured to obtain model service resource consumption of each candidate model service under a resource consumption optimization strategy.

[0113] The third priority determining subunit is configured to sort the model service resource consumption, and determine a third calling priority of each candidate model service based on the resource consumption sorting result.

[0114] The third determining subunit is configured to determine the target model service according to the third calling priority.

[0115] In some optional embodiments, the apparatus further includes:

[0116] The index issuing module is configured to issue the model index corresponding to the candidate model service in real time through the center gateway corresponding to the model edge gateway.

[0117] Further function descriptions of the above modules and subunits are the same as those of the above corresponding embodiments, and will not be described here.

[0118] The calling apparatus of the pre-trained model service in the embodiment is presented in the form of a functional unit. The unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0119] The calling apparatus of the pre-trained model service provided in the embodiment is configured to deploy a model edge gateway in an edge area where an end-side device is located, to query pre-trained model cache information matched with a model service calling request from a cache server in the edge area where the model edge gateway is located, and to return the pre-trained model cache information to the end-side device directly when it is determined that the cache server has the pre-trained model cache information matched with the model service calling request. Thus, the model gateway and the related cache server are sunk from the center to the edge area closer to the end-side device, so that the end-side device can access and infer locally and provide local cache when calling a model. Compared with calling a model from the center cache, the calling time is greatly reduced, and the application response speed of the end-side device is improved. At the same time, the model inference calculation is saved, the resource usage is optimized, and the throughput is improved. In a similar query scenario, local access can greatly reduce the calling time and greatly improve the response speed.

[0120] The disclosure also provides a computer device having Figure 10 the calling apparatus of the pre-trained model service.

[0121] Please refer to Figure 11 , Figure 11Fig. 1 is a schematic diagram of a computer device according to an example embodiment of the present disclosure. Figure 11 As shown in Fig. 1, the computer device includes one or more processors 10, memory 20, and interfaces 30 including a high-speed interface and a low-speed interface. The various components communicate over various buses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 10 can process instructions for execution within the computer device, including instructions stored in the memory 20 or elsewhere to implement processes, such as graphical information of a GUI stored in the memory 20 to be displayed on an external input / output device, such as a display device coupled to the high-speed interface. In some embodiments, multiple processors and / or multiple buses can be employed as appropriate, as will be apparent to those of ordinary skill in the art. Also, various elements of the computer device, as well as the input / output device itself, can be implemented with a level of redundancy to fail-safe or fail-over in the event one of these elements malfunctions. Figure 11 The processor 10 is taken as an example in the following description.

[0122] The processor 10 can be a central processing unit, a network processing unit, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0123] The memory 20 stores instructions that are executable by the at least one processor 10, so as to enable the at least one processor 10 to perform the method shown in the above embodiments.

[0124] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory that is remotely arranged with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0125] The memory 20 can include a volatile memory, such as a random access memory, and can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned kinds of memories.

[0126] The computer device also comprises a communication interface 30 for the computer device to communicate with other devices or communication networks.

[0127] The embodiments of the present disclosure further provide a computer readable storage medium, and the method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer code originally stored in a remote storage medium or non-transitory machine readable storage medium and downloaded to a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor, or hardware, the method shown in the above embodiments is implemented.

[0128] Part of the present disclosure can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present disclosure can be invoked or provided. Those skilled in the art should understand that the form of computer program instructions in a computer readable medium includes but is not limited to source files, executable files, installation package files, etc. Correspondingly, the way of computer program instructions executed by a computer includes but is not limited to: the computer directly executes the instructions, or the computer executes the corresponding compiled program after compiling the instructions, or the computer reads and executes the instructions, or the computer reads and executes the corresponding installed program after installing the instructions. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0129] Although the embodiments of the present disclosure are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present disclosure, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for calling a pre-trained model service, characterized in that, The method comprises: acquiring a model edge gateway corresponding to the end-side device, the model edge gateway being deployed in the same edge area as the end-side device; when receiving a model service call request initiated by the end-side device, querying, by the model edge gateway, a cache server whether there is pre-trained model cache information matched with the model service call request, the cache server being deployed in the same edge area as the model edge gateway, the pre-trained model cache information being used to represent model information called by the end-side device in a historical time; if the cache server has the pre-trained model cache information matched with the model service call request, returning the pre-trained model cache information to the end-side device; if the cache server does not have the pre-trained model cache information matched with the model service call request, acquiring an intranet environment where the model edge gateway is located; sending the model service call request to a target model server in the intranet environment, and returning, by the target model server, a target model service matched with the model service call request; wherein the returning, by the target model server, the target model service matched with the model service call request comprises: acquiring a model service call strategy corresponding to the end-side device and a plurality of candidate model services matched with the model service call request, the model service call strategy being a strategy for the end-side device to call a model service, the model service call strategy being encapsulated in the model service call request, the model service call strategy being any one of a shortest-latency-first strategy, a maximum-throughput-first strategy and a resource consumption optimization strategy, the model service latency under the shortest-latency-first strategy being used to represent time consumed by a model to output a result of task inference, the model service throughput under the maximum-throughput-first strategy being used to represent an amount of data that can be processed by the model in a unit of time, and the model service resource consumption under the resource consumption optimization strategy being used to represent resources consumed by the model in a use process; determining the target model service from the plurality of candidate model services according to the model service call strategy.

2. The method of claim 1, wherein, sending the model service call request to the target model server in the intranet environment comprises: acquiring a plurality of model servers deployed in the intranet environment and a load request amount of each model server; determining the target model server from each model server according to load balancing based on the load request amount.

3. The method of claim 1, wherein, when the model service call strategy is the shortest-latency-first strategy, the determining the target model service from the plurality of candidate model services according to the model service call strategy comprises: acquiring model service latency of each candidate model service under the shortest-latency-first strategy; sorting each model service latency, and determining a first call priority of each candidate model service based on a delay sorting result; determining the target model service according to the first call priority.

4. The method of claim 1, wherein, When the model service calling strategy is a throughput maximum priority strategy, the determining the target model service from the plurality of candidate model services according to the model service calling strategy comprises: obtaining model service throughputs of each candidate model service under the throughput maximum priority strategy; ranking each model service throughput, and determining a second calling priority of each candidate model service based on a ranking result of the throughputs; determining the target model service according to the second calling priority.

5. The method of claim 1, wherein, When the model service calling strategy is a resource consumption optimization strategy, the determining the target model service from the plurality of candidate model services according to the model service calling strategy comprises: obtaining model service resource consumptions of each candidate model service under the resource consumption optimization strategy; ranking each model service resource consumption, and determining a third calling priority of each candidate model service based on a ranking result of the resource consumptions; determining the target model service according to the third calling priority.

6. The method according to any one of claims 1 to 5, characterized in that, Further comprising: distributing, by the model edge gateway, model indexes corresponding to the candidate model services in real time through a corresponding central gateway.

7. A pre-trained model service calling apparatus, characterized by comprising: The apparatus comprises: an obtaining module configured to obtain a model edge gateway corresponding to an end-side device, the model edge gateway being deployed in a same edge area as the end-side device; a cache query module configured to, when receiving a model service calling request initiated by the end-side device, query, through the model edge gateway, whether there is pre-trained model cache information matched with the model service calling request in a cache server, the cache server being deployed in the same edge area as the model edge gateway, the pre-trained model cache information being used to represent model information called by the end-side device in a historical time; a model feedback module configured to, if the cache server has the pre-trained model cache information matched with the model service calling request, return the pre-trained model cache information to the end-side device; an intranet information obtaining module configured to, if the cache server does not have the pre-trained model cache information matched with the model service calling request, obtain an intranet environment in which the model edge gateway is located; a model matching module configured to send the model service calling request to a target model server in the intranet environment, and return, by the target model server, a target model service matched with the model service calling request; wherein the model matching module comprises: A policy obtaining unit is configured to obtain a model service invocation policy corresponding to the end-side device and a plurality of candidate model services matched with the model service invocation request, the model service invocation policy being a policy for invoking a model service by the end-side device, the model service invocation policy being encapsulated in the model service invocation request, the model service invocation policy being any one of a shortest-latency-first policy, a maximum-throughput-first policy, and a resource consumption optimization policy, the model service latency in the shortest-latency-first policy being used to represent a time consumed by a model to output a result of task inference, the model service throughput in the maximum-throughput-first policy being used to represent an amount of data that can be processed by a model in a unit of time, and the model service resource consumption in the resource consumption optimization policy being used to represent a resource consumed by a model in a process. A model determining unit is configured to determine the target model service from the plurality of candidate model services according to the model service invocation policy.

8. A computer device, comprising: The method comprises the following steps: A memory and a processor are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the method for invoking a pre-trained model service according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer perform the method for invoking a pre-trained model service according to any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer instructions are used to make a computer perform the method for invoking a pre-trained model service according to any one of claims 1 to 6.

Images