Large model service calling method, device and computer equipment

Abstract

The present application relates to a large model service calling method, device and computer equipment. The method includes: in response to the current service call request initiated for the large model service instance, obtaining the request number of the service call request contained in the large model request queue, and when the request number does not reach the preset request threshold, adding the current service call request to the large model request queue, and finally obtaining the target large model service instance through the preset scheduler based on the real-time request number of the large model service instance corresponding to each service call request contained in the large model request queue, and finally calling the target large model service instance. By setting the large model request queue, the request volume processed by the current backend system is effectively controlled, thereby avoiding the problem of system service overload caused by excessive request volume, and improving the feasibility and high availability of the system.

Description

Technical Field

[0001] The present application relates to the field of cloud big data technology, and in particular to a large model service calling method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] In modern distributed computing systems, with the widespread use of big models in natural language, image processing and other fields, various applications are trying to access big model services to improve the intelligence and convenience of application services. In the process of calling big model services, the existing calling method is direct scheduling, which directly sends the front-end request to the big model service, and the back-end simply calls the big model service.

[0003] However, the current direct scheduling method has the problem that the system is prone to service overload. Summary of the invention

[0004] Based on this, it is necessary to provide a large model service calling method, device, computer equipment, computer-readable storage medium and computer program product that can avoid system service overload in response to the above technical problems.

[0005] In a first aspect, the present application provides a large model service calling method, which is applied to a backend server, comprising:

[0006] In response to a current service call request initiated for a large model service instance, obtaining a request quantity of service call requests included in a large model request queue;

[0007] When the number of requests does not reach the preset request number threshold, the current service call request is added to the large model request queue;

[0008] Obtain the target big model service instance and call the target big model service instance; the target big model service instance is determined from the big model service instances corresponding to each service call request through a preset scheduler based on the real-time request number of the big model service instances corresponding to each service call request contained in the big model request queue.

[0009] In one embodiment, obtaining a target large model service instance includes:

[0010] Obtain the preset maximum number of concurrent requests for each model service instance and the real-time number of requests for each model service instance through the scheduler;

[0011] The large model service instance whose real-time request number is less than the maximum concurrent number is used as the callable large model service instance;

[0012] According to the scheduling strategy of the scheduler, the target large model service instance is obtained from the callable large model service instances.

[0013] In one embodiment, obtaining a target large model service instance from a callable large model service instance includes:

[0014] The callable large model service instance with the smallest number of real-time requests is used as the target large model service instance.

[0015] In one embodiment, obtaining a target large model service instance from a callable large model service instance includes:

[0016] Obtain the call number associated with each callable big model service instance and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time;

[0017] When the call number associated with the callable big model service instance includes a historical call number, the historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance.

[0018] In one embodiment, obtaining a target large model service instance from a callable large model service instance includes:

[0019] Obtain the call number associated with each callable big model service instance and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time;

[0020] If the call number does not include the historical call number, the historical call number is postponed or decremented, and the postponed or decremented historical call number is used as the new historical call number until the call number associated with the callable large model service instance includes the historical call number;

[0021] The historical call number is determined as the target call number, and the callable large model service instance associated with the target call number is used as the target large model service instance.

[0022] In an exemplary embodiment, after calling the target large model service instance, the method further includes:

[0023] When the target large model service instance is successfully called and the call response time of the target large model service instance does not exceed the preset time, the call response result of the target large model service instance is obtained, and the call response result is returned to the front end that triggers the current service call request;

[0024] When the target large model service instance fails to be called, or the call response time exceeds the preset time, the preset business exception handling logic is executed.

[0025] In one of the embodiments, after calling the target large model service instance, the process includes:

[0026] When the target large model service instance call is completed, the current service call request contained in the large model request queue is removed.

[0027] In an exemplary embodiment, after obtaining the number of service call requests included in the large model request queue, the method further includes:

[0028] When the number of requests reaches the preset request number threshold, stop adding the current service call request to the large model request queue and execute the pre-set rejection strategy.

[0029] In a second aspect, the present application also provides a large model service calling device, which is applied to a backend server, including:

[0030] An acquisition module, used to obtain the number of service call requests contained in the large model request queue in response to a current service call request initiated for the large model service instance;

[0031] An adding module is used to add the current service call request into the large model request queue when the number of requests does not reach the preset request number threshold;

[0032] A calling module is used to obtain a target big model service instance and call the target big model service instance; the target big model service instance is determined from the big model service instances corresponding to each service call request through a preset scheduler based on the real-time request number of the big model service instances corresponding to each service call request contained in the big model request queue.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] In response to a current service call request initiated for a large model service instance, obtaining a request quantity of service call requests included in a large model request queue;

[0035] When the number of requests does not reach the preset request number threshold, the current service call request is added to the large model request queue;

[0036] Obtain the target big model service instance and call the target big model service instance; the target big model service instance is determined from the big model service instances corresponding to each service call request through a preset scheduler based on the real-time request number of the big model service instances corresponding to each service call request contained in the big model request queue.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0038] In response to a current service call request initiated for a large model service instance, obtaining a request quantity of service call requests included in a large model request queue;

[0039] When the number of requests does not reach the preset request number threshold, the current service call request is added to the large model request queue;

[0040] Obtain the target big model service instance and call the target big model service instance; the target big model service instance is determined from the big model service instances corresponding to each service call request through a preset scheduler based on the real-time request number of the big model service instances corresponding to each service call request contained in the big model request queue.

[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0042] In response to a current service call request initiated for a large model service instance, obtaining a request quantity of service call requests included in a large model request queue;

[0043] When the number of requests does not reach the preset request number threshold, the current service call request is added to the large model request queue;

[0044] Obtain the target big model service instance and call the target big model service instance; the target big model service instance is determined from the big model service instances corresponding to each service call request through a preset scheduler based on the real-time request number of the big model service instances corresponding to each service call request contained in the big model request queue.

[0045] The above-mentioned large model service call method, device, computer equipment, computer-readable storage medium and computer program product, in response to the current service call request initiated for the large model service instance, obtains the request number of the service call request contained in the large model request queue, and when the request number does not reach the preset request threshold, adds the current service call request to the large model request queue, and finally obtains the target large model service instance through the preset scheduler based on the real-time request number of the large model service instance corresponding to each service call request contained in the large model request queue, and finally calls the target large model service instance. By setting the large model request queue, the request volume processed by the current backend system is effectively controlled, thereby avoiding the problem of system service overload due to excessive request volume, and improving the feasibility and high availability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 An application environment diagram of a large model service calling method in an embodiment;

[0048] Figure 2 A schematic diagram of a flow chart of a large model service calling method in one embodiment;

[0049] Figure 3 A schematic diagram of a process of adding a large model request queue in one embodiment;

[0050] Figure 4 A schematic diagram of a process for determining a target large model service instance in one embodiment;

[0051] Figure 5 A schematic diagram of a process for determining a target large model service instance in another embodiment;

[0052] Figure 6 A schematic diagram of a process of calling a target large model service instance in one embodiment;

[0053] Figure 7 A complete flow chart of a large model service calling method in another embodiment;

[0054] Figure 8 A structural block diagram of a large model service calling device in one embodiment;

[0055] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The large model service calling method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the front end n102 is installed with multiple interactive applications, which are displayed as multiple application icons on the front end interface. Users can initiate service call requests by clicking on the application icons. The front end n102 interacts with the back end server 104 through the network. In addition, a scheduler can be preset in the back end server, and the scheduler can also be a separate device independent of the server. The data storage system can store data that the back end server 104 needs to process, as well as multiple large model service instances for responding to service call requests. The data storage system can be integrated on the back end server 104, or it can be placed on the cloud or other network servers.

[0058] The backend server 104 responds to the current service call request for the large model service instance initiated by the user through the front end n102, and then obtains the number of service call requests contained in the large model request queue, and if the number of requests does not reach the preset request number threshold, adds the current service call request to the large model request queue, and finally obtains the target large model service instance through the preset scheduler based on the real-time number of requests of the large model service instance corresponding to each service call request contained in the large model request queue, and calls the target large model service instance. Among them, the front end n102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The backend server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0059] In an exemplary embodiment, Figure 2 As shown, a large model service calling method is provided, which is applied to Figure 1 The backend server 104 in the example is used as an example to illustrate, including the following steps S201 to S203. Among them:

[0060] Step S201, in response to a current service call request initiated for a large model service instance, obtain the request quantity of the service call request contained in the large model request queue.

[0061] Among them, the current service call request can be understood as a bridge for interaction between the front-end n102 and the back-end server 104. The user selects the corresponding service through the service page designed by the front-end n102. The front-end n102 generates the corresponding service call request and sends it to the back-end server 104. The back-end server 104 determines whether to implement this service call.

[0062] Optionally, the back-end server 104 obtains the number of service call requests contained in the large model request queue in response to the current service call request initiated by the user through the front-end n102 for the large model service instance. The back-end server 104 obtains the number of service call requests contained in the large model request queue only when it receives the current service call request, that is, when it needs to call the large model service instance. In this way, the cost of obtaining the number of requests at all times can be reduced, and a data foundation is laid for whether to respond to the service call request subsequently.

[0063] Step S202: When the number of requests does not reach a preset request number threshold, the current service call request is added to the large model request queue.

[0064] Step S203, obtain the target big model service instance and call the target big model service instance; the target big model service instance is determined from the big model service instances corresponding to each service call request through a preset scheduler based on the real-time request number of the big model service instances corresponding to each service call request contained in the big model request queue.

[0065] Among them, the preset request quantity threshold can be understood as the maximum number of service call requests that the back-end server 104 can respond to within its own tolerance; the scheduler can be understood as a device for determining the target large model service instance, which can be a component integrated in the back-end server 104 or a separate device. Different schedulers are configured with different target large model service instance selection strategies.

[0066] For example, when the number of requests does not reach the preset request number threshold, that is, the current performance of the backend server 104 can still support the response to the current service call request, the current service call request is added to the big model request queue, and the preset scheduler responds to the current service call request. Based on the real-time request number of the big model service instance corresponding to each service call request contained in the big model request queue, the target big model service instance is determined from the big model service instance corresponding to each service call request, and the backend server 104 calls the target big model service instance. By adding a big model request queue to buffer the current service call request directly into the backend server 104, the problem of system processing crash caused by the simultaneous influx of too many requests is avoided, thereby enhancing the robustness of the system and improving the availability of the system.

[0067] In the above-mentioned large model service calling method, in response to the current service calling request initiated for the large model service instance, the request number of the service calling request contained in the large model request queue is obtained. If the request number does not reach the preset request threshold, the current service calling request is added to the large model request queue. Finally, the preset scheduler is used to obtain the target large model service instance based on the real-time request number of the large model service instance corresponding to each service calling request contained in the large model request queue, and finally the target large model service instance is called. By setting the large model request queue, the request volume processed by the current backend system is effectively controlled, thereby avoiding the problem of system service overload caused by excessive request volume, and improving the feasibility and high availability of the system.

[0068] In one embodiment, obtaining a target big model service instance includes: obtaining, through a scheduler, a preset maximum number of concurrencies for each big model service instance and a real-time request number for each big model service instance; using a big model service instance whose real-time request number is less than the maximum number of concurrencies as a callable big model service instance; and obtaining a target big model service instance from the callable big model service instance according to a scheduling strategy of the scheduler.

[0069] Among them, the maximum concurrency can be understood as the service call requests that the model service instance can process simultaneously under the premise of maintaining normal service response; the real-time request number can be understood as the number of service call requests that the large model service instance is currently responding to; the scheduling strategy can be understood as the target large model service instance selection strategy, such as polling strategy, minimum number of connections strategy, priority strategy, first-come-first-served strategy, short job priority strategy, etc.

[0070] For example, the backend server 104 obtains the preset maximum number of concurrency of each major model service instance and the number of real-time requests of each major model service instance through the scheduler, and uses the large model service instance with a real-time request number less than the maximum number of concurrency as the callable large model service instance, and obtains the target large model service instance from the callable large model service instance according to the scheduling strategy matched by the scheduler. The maximum number of concurrency and the number of real-time requests of each major model service instance are used to perform preliminary screening of model service instances, and the callable model service instance that can still respond to the service call request is determined, thereby reducing the cost of obtaining the target large model service instance and improving the accuracy of determining the target large model service instance. In addition, the scheduling strategy matched by the scheduler is used to obtain the target large model service instance from the callable large model service instance, and the scheduling strategy is used for selection instead of random selection, so as to achieve load balancing of the major callable large model service instances.

[0071] In one of the embodiments, obtaining a target large model service instance from the callable large model service instances includes: taking the callable large model service instance with the smallest number of real-time requests as the target large model service instance.

[0072] Optionally, based on the least number of connections strategy, the server 104 uses the callable large model service instance with the smallest number of real-time requests as the target large model service instance. The least number of connections strategy can be understood as preferentially allocating new requests to the callable large model service instance with the smallest number of current real-time requests. The large model service instance with the smallest number of real-time requests is currently under relatively light load. Using it as the target large model service instance for processing the current service call request can improve the call response speed to the current service call request and ensure the load balancing of the large model service instances.

[0073] In an exemplary embodiment, a target big model service instance is obtained from the callable big model service instance, including: obtaining a call number associated with each callable big model service instance and a pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time; when the call number associated with the callable big model service instance includes the historical call number, the historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance.

[0074] For example, based on the polling strategy, the server 104 obtains the call number associated with each callable big model service instance (the call number associated with each big model service instance is a sequential number, but the call number associated with each big model service instance that can be called is not necessarily a sequential number), and the pre-recorded historical call number of the big model service instance called last time. When the call number associated with the callable big model service instance includes the historical call number, the historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance. Among them, the polling strategy can be understood as usually maintaining a list of big model service instances, each of which has a sequential number. When a request arrives, the request will be sent to the big model service instance with the current number, and the number will be incremented after sending (after reaching the end of the list, it will be passed back to the starting server). The polling strategy determines that the target big model service instance is simple to implement, guarantees fairness for each callable big model service instance, and does not need to track the status of the callable big model service instance, making the determination of the target big model service instance simpler and more efficient.

[0075] In one embodiment, obtaining a target large model service instance from a callable large model service instance includes:

[0076] Get the call number associated with each callable big model service instance and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time; if the historical call number is not included in the call number, postpone or decrement the historical call number, and use the postponed or decremented historical call number as the new historical call number until the call number associated with the callable big model service instance includes the historical call number; determine the historical call number as the target call number, and use the callable big model service instance associated with the target call number as the target big model service instance.

[0077] Optionally, server 104 obtains the call number associated with each callable big model service instance and the call number of the last called big model service instance recorded in advance. If the call number does not contain the historical call number, the historical call number is postponed or decremented, and the postponed or decremented historical call number is used as the new historical call number until the call number associated with the callable big model service instance contains the updated historical call number. Finally, the updated historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance.

[0078] For example, assuming the historical call number is 3, when the call number associated with the callable large model service instance includes the historical call number 3, the historical call number 3 is determined as the target call number. If the historical call number 3 is not included, the historical call number 4, 5... or the historical call number 2, 1 is decremented, and the deferred or decremented historical call number is used as the new historical call number until the historical call number is included in the call number associated with the callable large model service instance. For example, the call number associated with the callable large model service instance is 5 but not 4, or the call number associated with the callable large model service instance is 1 but not 2, then the historical call number 5 or the historical call number 1 is used as the target call number.

[0079] The means of determining the target call number mentioned in the above embodiments include selecting any one of the minimum number of connections strategy and the polling strategy, and another method is to prioritize the use of the historical call number according to the polling strategy to directly determine the target call number, followed by deferring or decreasing the historical call number, and finally determining the target call number; when the above methods cannot determine, directly select the large model service instance with the smallest number of requests as the target large model service instance. By increasing the methods for determining the target large model service instance, the efficiency and reliability of determining the target large model service instance are guaranteed, and the high reliability and high availability of the system are improved.

[0080] In one of the embodiments, after calling the target large model service instance, it also includes: when the target large model service instance is called successfully and the call response time of the target large model service instance does not exceed the preset time, obtaining the call response result of the target large model service instance, and returning the call response result to the front end that triggers the current service call request; when the target large model service instance fails to call, or the call response time exceeds the preset time, executing the preset business exception handling logic.

[0081] Among them, the business exception handling logic can be understood as a set of processing mechanisms designed to deal with various business exceptions (such as errors, failures or unexpected situations) that may occur during the operation of the system.

[0082] For example, when the target large model service instance is successfully called and the call response time of the target large model service instance does not exceed the preset time, the call response result of the target large model service instance is obtained, and the call response result is returned to the front end that triggers the current service call request. When the target large model service instance fails to call or the call response time exceeds the preset time, the preset business exception handling logic is executed. The specific examples are as follows: First, the call of the target large model service instance fails. At this time, there is no call response time, and the preset business exception handling logic is called for exception handling; second, the target large model service instance is successfully called, but the call response time exceeds the preset time, and the preset business exception handling logic is still called for exception handling. The business exception handling logic set in the two cases is different. Whether it is a call failure or a response timeout, the system can immediately start the preset business exception handling logic to avoid waste of resources, ensure that the system can quickly handle errors when they occur, thereby reducing the impact of sudden errors on the system, and through timely feedback response and effective abnormal business processing, users can feel the fluency and reliability when using the service, thereby enhancing the user's trust and satisfaction with the system.

[0083] In one embodiment, after calling the target large model service instance, the method includes: when the call of the target large model service instance is completed, removing the current service call request contained in the large model request queue.

[0084] For example, when calling the target large model service instance, the backend server 104 adds 1 to the real-time request number of the target large model service instance, and when the call of the target large model service instance is completed (both failed and successful calls are counted as completed calls), the real-time request number of the target large model service instance is reduced by 1, and the current service call request contained in the large model request queue is removed, and the number of requests in the large model request queue is reduced by 1. By adjusting the real-time request number before and after the call of the large model service instance, the load of each instance can be accurately controlled, and the status of the large model request queue can be updated in time, which ensures the continuity of business processing and avoids excessive consumption and waste of resources.

[0085] In one of the embodiments, after obtaining the number of service call requests contained in the large model request queue, it also includes: when the number of requests reaches a preset request number threshold, stopping adding the current service call request to the large model request queue and executing a preset rejection strategy.

[0086] Optionally, when the number of requests reaches a preset request number threshold, the backend server 104 stops adding the current service call request to the large model request queue and executes a preset rejection strategy, including not processing the current service call request until the number of requests does not reach the preset request number threshold, and then adding it to the large model request queue for processing; and calling other large model services to guarantee a timely response to the current service call request. By presetting the maximum value of the request queue, new requests are rejected once the threshold is reached, thereby protecting the stability of the large model service and ensuring the reliability and high availability of the system. At the same time, by calling other large model service instances to respond as a backup, the user's service experience is also guaranteed to a certain extent.

[0087] This application is mainly a scheduling framework for backend servers to call large model services. Through configuration, the path, priority, routing strategy and concurrency of large model service instances can be flexibly preset. During the large model scheduling process, the preset scheduler will call the optimal large model service instance based on the path combined with the routing strategy and the preset concurrency. If the request concurrency has reached the preset concurrency peak of all instances, the rejection strategy will be executed to avoid unnecessary service requests, while avoiding waste of backend resources and unnecessary request waiting on the front end.

[0088] The detailed scheduling process is as follows:

[0089] Step 1: If Figure 3 As shown in the figure, the judgment process is as follows: set the large model request queue. If the number of requests in the request queue has reached the preset value, execute the rejection strategy and no longer call the large model service; if the request queue is not full, put the large model request into the request queue. The scheduler obtains the request from the request queue.

[0090] Step 2: The scheduler determines the optimal large model calling instance based on the number of concurrent calls being processed by the large model service instance, combined with the scheduling strategy and the preset concurrency.

[0091] The detailed scheduling logic is as follows:

[0092] Execute different schedulers according to the scheduling strategy:

[0093] ① Such as Figure 4 As shown, a polling strategy is adopted: the large model service instance with a real-time concurrency less than the set concurrency is obtained as the callable service instance. Each model service instance has a number. According to the model service instance called last time, the model service instance that should be called this time is selected from the callable model service instances according to the last call number, and the model service number called this time is recorded.

[0094] ② If Figure 5 As shown, the minimum number of connections strategy is adopted: the model service instance whose real-time processing concurrency is less than the preset maximum concurrency is obtained as the callable model service instance, and the service instance with the smallest number of requests processed in real time among all callable model service instances is found, and the service instance is selected as the service instance for this call, and the number of requests processed in real time by the service instance is increased by 1.

[0095] Step 3: If Figure 6 As shown, the large model service is called, and the real-time request count of the service instance is increased by 1. If the large model is called but no response is received within the timeout period, the abnormal business logic is executed; if the response is received within the preset timeout period, the large model result is returned normally. After the call is completed, the real-time request count of the large model service is reduced by 1, and the request task is removed from the request queue.

[0096] Based on the above content, the complete process of the large model service call method is as follows Figure 7 shown.

[0097] Compared with the prior art, this application has the following advantages:

[0098] 1. By setting the large model request queue and rejection strategy, the concurrent request volume is effectively controlled, avoiding the service overload problem caused by excessive requests. The traditional direct call method easily causes the large model service to respond late or crash due to too many requests. This solution presets the maximum value of the request queue and rejects new requests once the threshold is reached, thereby protecting the stability of the large model service and ensuring the reliability and high availability of the system.

[0099] 2. By flexibly configuring the path, priority and routing strategy of the large model service instance, this patent realizes the intelligent scheduling of the large model service. According to different scheduling strategies, such as polling strategy and minimum number of connections strategy, the system can dynamically select the optimal large model service instance for calling, thereby achieving load balancing. This not only improves the utilization rate of hardware resources, but also reduces the load of a single service instance and avoids resource bottleneck problems. In particular, the minimum number of connections strategy helps to prioritize the calling of service instances with lower current loads, further improving the system's response speed and service quality.

[0100] 3. By monitoring and managing the real-time concurrent number of each large model service instance, the orderliness and stability of service calls are guaranteed. By adjusting the real-time request number before and after the call of the large model service instance, the load of each instance can be accurately controlled to avoid excessive consumption and waste of resources. In addition, setting a reasonable timeout time and exception handling logic ensures that the large model service can handle exceptions in a timely manner, reducing the negative impact on the overall system and improving the system's fault tolerance and stability.

[0101] 4. Through the strategy of reducing peak loads and filling valleys, the pressure brought by the peak request period is effectively alleviated and the request processing process is smoothed. This strategy not only ensures the request processing efficiency during the peak period, but also improves the resource utilization of the system during the off-peak period, saving hardware resources and costs.

[0102] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a large model service invocation device for implementing the large model service invocation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more large model service invocation device embodiments provided below can refer to the limitations on the large model service invocation method above, and will not be repeated here.

[0104] In an exemplary embodiment, Figure 8As shown, a large model service calling device is provided, including: an acquisition module 801, an adding module 802 and a calling module 803, wherein:

[0105] The acquisition module 801 is used to obtain the number of service call requests contained in the large model request queue in response to the current service call request initiated for the large model service instance.

[0106] The adding module 802 is used to add the current service call request into the large model request queue when the number of requests does not reach a preset request number threshold.

[0107] The calling module 803 is used to obtain the target big model service instance and call the target big model service instance; the target big model service instance is determined from the big model service instances corresponding to each service call request through a preset scheduler based on the real-time request number of the big model service instances corresponding to each service call request contained in the big model request queue.

[0108] In one embodiment, the calling module 802 further includes:

[0109] The acquisition unit is used to obtain the preset maximum concurrency number of each major model service instance and the real-time request number of each major model service instance through the scheduler.

[0110] The initial screening unit is used to select the large model service instances whose real-time request number is less than the maximum concurrent number as callable large model service instances.

[0111] The determination unit is used to obtain the target large model service instance from the callable large model service instances according to the scheduling strategy of the scheduler.

[0112] In one of the embodiments, the determination unit is further configured to use the callable large model service instance with the smallest number of real-time requests as the target large model service instance.

[0113] In an exemplary embodiment, the determination unit is also used to obtain the call number associated with each callable big model service instance, and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time; when the call number associated with the callable big model service instance includes the historical call number, the historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance.

[0114] In one embodiment, the determination unit is also used to obtain the call number associated with each callable big model service instance, and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time; when the historical call number is not included in the call number, the historical call number is postponed or decremented, and the postponed or decremented historical call number is used as the new historical call number until the call number associated with the callable big model service instance includes the historical call number; the historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance.

[0115] In one embodiment, the large model service calling device further includes:

[0116] The normal feedback unit is used to obtain the call response result of the target large model service instance when the call of the target large model service instance is successful and the call response time of the target large model service instance does not exceed the preset time, and return the call response result to the front end that triggers the current service call request.

[0117] The exception handling unit is used to execute the preset business exception handling logic when the target large model service instance fails to call or the call response time exceeds the preset time.

[0118] In one embodiment, the large model service calling device further includes: a request removal module, which is used to remove the current service calling request contained in the large model request queue when the target large model service instance call is completed.

[0119] In one of the embodiments, the large model service calling device also includes: a request stopping module, which is used to stop adding the current service calling request to the large model request queue and execute a preset rejection strategy when the number of requests reaches a preset request number threshold.

[0120] Each module in the above-mentioned large model service calling device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0121] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store service call requests, request quantities, large model service instance data and related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large model service call method is implemented.

[0122] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the large model service calling method of the above embodiment is implemented.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the large model service calling method of the above embodiment is implemented.

[0125] In one embodiment, a computer program product is provided, including a computer program, which implements the large model service calling method of the above embodiment when executed by a processor.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0127] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0129] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A large model service calling method, characterized in that: Applied to a backend server, the method comprises: In response to a current service call request initiated for a large model service instance, obtaining a request quantity of service call requests included in a large model request queue; When the number of requests does not reach a preset request number threshold, adding the current service call request to the large model request queue; Obtaining, through a preset scheduler, the preset maximum concurrency of the big model service instance corresponding to each service call request contained in the big model request queue, and the real-time request number of each big model service instance; the real-time request number represents the number of service call requests currently being responded to by the big model service instance; The large model service instance whose number of real-time requests is less than the maximum concurrent number is used as a callable large model service instance; Obtaining a target large model service instance from the callable large model service instance according to the scheduling strategy of the scheduler, including: Obtaining the call number associated with each of the callable big model service instances and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time; In the case where the call number associated with the callable big model service instance includes the historical call number, the historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance; The target large model service instance is called.

2. The method according to claim 1, characterized in that The step of obtaining a target large model service instance from the callable large model service instance includes: The callable large model service instance with the smallest number of real-time requests is used as the target large model service instance.

3. The method according to claim 1, characterized in that: The step of obtaining a target large model service instance from the callable large model service instance includes: Obtaining the call number associated with each of the callable big model service instances and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time; In the case that the historical call number is not included in the call number, the historical call number is postponed or decremented, and the postponed or decremented historical call number is used as a new historical call number until the call number associated with the callable large model service instance includes the historical call number; The historical call number is determined as the target call number, and the callable large model service instance associated with the target call number is used as the target large model service instance.

4. The method according to claim 1, characterized in that: After calling the target large model service instance, it also includes: When the target large model service instance is successfully called and the call response time of the target large model service instance does not exceed the preset time, obtain the call response result of the target large model service instance, and return the call response result to the front end that triggers the current service call request; When the target large model service instance fails to be called, or the call response time exceeds the preset time, the preset business exception handling logic is executed.

5. The method according to claim 1, characterized in that After calling the target large model service instance, it includes: When the target large model service instance call is completed, the current service call request contained in the large model request queue is removed.

6. The method according to claim 1, characterized in that After obtaining the number of service call requests included in the large model request queue, the method further includes: When the number of requests reaches the preset request number threshold, stop adding the current service call request into the large model request queue and execute the preset rejection strategy.

7. A large model service calling device, characterized in that: Applied to a backend server, the device comprises: An acquisition module, used to obtain the number of service call requests contained in the large model request queue in response to a current service call request initiated for the large model service instance; An adding module, configured to add the current service call request into the large model request queue if the number of requests does not reach a preset request number threshold; A calling module is used to obtain the preset maximum concurrency of the big model service instance corresponding to each service call request contained in the big model request queue and the real-time request number of each big model service instance through a preset scheduler; the real-time request number represents the number of service call requests currently being responded to by the big model service instance; the big model service instance whose real-time request number is less than the maximum concurrency number is used as the callable big model service instance; according to the scheduling strategy of the scheduler, the target big model service instance is obtained from the callable big model service instance, including: obtaining the call number associated with each of the callable big model service instances, and the pre-recorded historical call number; the historical call number is the call number of the big model service instance called last time; when the call number associated with the callable big model service instance includes the historical call number, the historical call number is determined as the target call number, and the callable big model service instance associated with the target call number is used as the target big model service instance; calling the target big model service instance.

8. The device according to claim 7, characterized in that The device also includes: The calling module is also used to use the callable large model service instance with the smallest number of real-time requests as the target large model service instance.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images