Service request scheduling method and device, electronic equipment and storage medium
By assigning task queue priority based on scenario information, the response delay problem caused by high load on the service server of large language model is solved, and the user experience of terminal devices is improved.
Patent Information
- Application Number
- CN202410955286.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
AI Technical Summary
When the server load of large language model service is deployed is high, the model service response delay is high, affecting the user experience of end device users.
By obtaining the scene information of the call request, the target task queue is determined, and the processing task is sent to the task queue with corresponding call priority. Based on the priority call and execution of tasks, we ensure that requests in different application scenarios are processed with appropriate priority.
It improves the overall processing efficiency and response speed of the model service system and improves the user experience of terminal device users.
Smart Images

Figure CN118916130A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of Internet technology, and in particular to a service request scheduling method, device, electronic device, and storage medium. Background Art
[0002] Currently, with the rapid development and popularization of large language models, more and more applications (Application, APP) begin to provide functions based on large language models, such as text processing, image processing, intelligent voice, etc. In practical applications, model services based on large language models are usually deployed on the server side to provide corresponding data processing services by responding to client requests.
[0003] However, in actual application, when the server where the model service is deployed has too many pending tasks and requests, the server load will increase, which will cause high response delay of the model service and affect the user experience on the terminal device side. Summary of the invention
[0004] The embodiments of the present disclosure provide a service request scheduling method, device, electronic device and storage medium to overcome the problem of high model service response delay.
[0005] In a first aspect, an embodiment of the present disclosure provides a service request scheduling method, including:
[0006] In response to a call request for a target function, at least one processing task is generated, and the processing task is used to call the model service corresponding to the target function to process target data; according to the scenario information corresponding to the call request, a target task queue is determined from multiple task queues, and the processing task is sent to the corresponding target task queue, wherein the scenario information represents an application scenario when running the target function, and the target task queue has a calling priority corresponding to the scenario information; based on the calling priority of the target task queue, the target task queue is called, and the processing task in the target task queue is executed to obtain the execution result of the target function returned by the corresponding model service.
[0007] In a second aspect, an embodiment of the present disclosure provides a service request scheduling device, including:
[0008] A response module, configured to generate at least one processing task in response to a call request for a target function, wherein the processing task is used to call a model service corresponding to the target function to process target data;
[0009] a processing module, configured to determine a target task queue from a plurality of task queues according to scenario information corresponding to the call request, and send the processing task to the corresponding target task queue, wherein the scenario information represents an application scenario when the target function is run, and the target task queue has a call priority corresponding to the scenario information;
[0010] The calling module is used to call the target task queue based on the calling priority of the target task queue, and execute the processing tasks in the target task queue to obtain the execution result of the target function returned by the corresponding model service.
[0011] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;
[0012] The memory stores computer-executable instructions;
[0013] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the service request scheduling method as described in the first aspect and various possible designs of the first aspect.
[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the service request scheduling method described in the first aspect and various possible designs of the first aspect is implemented.
[0015] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the service request scheduling method as described in the first aspect and various possible designs of the first aspect.
[0016] The service request scheduling method, device, electronic device and storage medium provided in this embodiment generate at least one processing task in response to a call request for a target function, wherein the processing task is used to call the model service corresponding to the target function to process target data; determine the target task queue from multiple task queues according to the scenario information corresponding to the call request, and send the processing task to the corresponding target task queue, wherein the scenario information represents the application scenario when the target function is running, and the target task queue has a call priority corresponding to the scenario information; based on the call priority of the target task queue, call the target task queue, and execute the processing task in the target task queue to obtain the execution result of the target function returned by the corresponding model service. By obtaining the scenario information corresponding to the call request, and sending the processing task to the corresponding target task queue for processing based on the scenario information, the matching between the use scenario of the call request and the call priority of the task queue is achieved, so that the call request in different application scenarios can be processed with the corresponding priority, thereby improving the overall processing efficiency and response speed of the model service system, thereby improving the user experience on the terminal device side. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0018] Figure 1 An application scenario diagram of the service request scheduling method provided in an embodiment of the present disclosure;
[0019] Figure 2 Schematic diagram of the process of the service request scheduling method provided in the embodiment of the present disclosure Figure 1 ;
[0020] Figure 3 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;
[0021] Figure 4 A schematic diagram of a service request scheduling system provided by an embodiment of the present disclosure;
[0022] Figure 5 Schematic diagram of the process of the service request scheduling method provided in the embodiment of the present disclosure Figure 2 ;
[0023] Figure 6 for Figure 5A flowchart of a specific implementation method of step S204 in the illustrated embodiment;
[0024] Figure 7 for Figure 6 A flowchart of a specific implementation method of step S2042 in the illustrated embodiment;
[0025] Figure 8 A schematic diagram of the structure of a member group provided in an embodiment of the present disclosure;
[0026] Fig. 9 for Figure 5 A flowchart of a specific implementation method of step S210 in the illustrated embodiment;
[0027] Fig.10 A structural block diagram of a service request scheduling device provided in an embodiment of the present disclosure;
[0028] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;
[0029] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0032] The application scenarios of the embodiments of the present disclosure are explained below:
[0033] Figure 1An application scenario diagram of the service request scheduling method provided in the embodiment of the present disclosure. The service request scheduling method provided in the embodiment of the present disclosure can be applied to the application scenarios of cloud computing and cloud services. The execution subject of this embodiment can be a server for providing model services or processing model service requests, or other electronic devices that perform similar functions. Among them, in some embodiments, the above-mentioned server or electronic device can implement the service request scheduling method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Furthermore, in some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communications, cloud databases, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, among which cloud services can be interactive processing services for terminal devices to call.
[0034] refer to Figure 1 As shown in the figure, the execution subject of the solution provided by this embodiment is, for example, the task scheduling server shown in the figure. The task scheduling server receives the call request from the terminal device or other business server, and after identification and processing, it sends a task request to the local or externally deployed model service, such as a general task model, a vertical task model, etc., and obtains the corresponding execution result. After that, the execution result is returned, for example, sent back to the terminal device, so as to realize the execution of program functions based on the model service on the terminal device side, such as image and text generation, and intelligent voice assistant activation.
[0035] In the prior art, the task scheduling server usually only has the function of generating and forwarding tasks, that is, after identifying the call request as the corresponding processing task, it is sent to the task queue of the corresponding model service for queue execution. However, in actual application, when there are too many tasks to be processed in the task queue corresponding to the model service, or when the server where the model service is deployed works at high load for a long time, it will cause the problem of high response delay of the model service, thereby affecting the user experience on the terminal device side.
[0036] The embodiment of the present disclosure provides a service request scheduling method to solve the above problem.
[0037] refer to Figure 2 , Figure 2 Schematic diagram of the process of the service request scheduling method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a server or other electronic devices. The service request scheduling method includes:
[0038] Step S101: In response to a call request for a target function, at least one processing task is generated, where the processing task is used to call a model service corresponding to the target function to process target data.
[0039] refer to Figure 1 As shown in the schematic diagram of the application scenario, in this embodiment, the provided method is introduced with a task scheduling server (hereinafter referred to as the server) as the execution subject. By way of example, during the running of the application, after starting the target function that needs to be implemented based on the model service, the terminal device sends a call request for the target function to the server. Specifically, for example, the terminal device loads a data table in the application, and then, in response to the user operation, runs the "content summary" function implemented based on artificial intelligence technology to analyze a row, a column or an item of target data in the data table, and expects to obtain a content summary of the above target data. More specifically, for example, if the target data is a user comment, then based on the above target function, it is expected to obtain a tendency conclusion of the user comment, such as a positive comment or a negative comment.
[0040] Furthermore, the realization of the above-mentioned target function needs to be realized based on the model service deployed on the cloud server. Therefore, the terminal device sends a call request for the target function to the server. The specific format of the call request is determined according to the communication protocol between the two, and no specific restrictions are made here. After that, after receiving the call request, the server processes and identifies the call request, and can generate at least one processing task. Among them, the processing task is used to call the model service corresponding to the target function to process the target data, so as to realize the above-mentioned target function. In one possible implementation, the model service called by the processing task can realize the above-mentioned target function by directly processing the target data, such as calling a general large language model to process the text and realize the function of summarizing the text content. In another possible implementation, the model service called by the processing task indirectly realizes the target function, such as realizing a functional step of the target function. Specifically, for example, calling the reasoning model data, reasoning the text, and making an intention judgment, and then further calling other model services, such as model services in vertical fields, through other processing tasks, to finally realize the target function. Therefore, in the steps of this embodiment, one or more processing tasks will be generated in response to the call request of the target function. The above processing tasks may be executed in parallel or in series, depending on specific needs.
[0041] Furthermore, according to the specific implementation, the terminal device may also synchronously send the above target data, or the storage location information of the above target data, so that the above target data can be acquired and processed by the model service.
[0042] Step S102: According to the scenario information corresponding to the call request, determine the target task queue from multiple task queues, and send the processing task to the corresponding target task queue, wherein the scenario information represents the application scenario when running the target function, and the target task queue has a call priority corresponding to the scenario information.
[0043] Exemplarily, after generating the processing task, the server will further obtain the scenario information corresponding to the call request, wherein the scenario information represents the application scenario when the target function is running, such as intelligent dialogue, text-to-image generation, text processing, etc. Correspondingly, according to different application scenarios, there are different task queues, which are used to receive processing tasks and execute them in sequence, thereby realizing the orderly call of model services. Furthermore, when the application scenarios are different, the corresponding task queues have different call priorities. For example, in application scenarios such as intelligent dialogue, due to the high real-time requirements for service response and the need for a shorter response delay, it corresponds to a higher call priority, so that the call request in this scenario can be responded to and executed first; and for application scenarios of image generation such as text-to-image and image style transfer, since a certain degree of delay can be tolerated, it corresponds to a lower call priority, so that the call request in this scenario can be responded to and executed later, thereby realizing the orderly scheduling of model services.
[0044] Afterwards, according to the scenario information corresponding to the call request, a target task queue that matches it is determined from the task queue maintained by the server (the service request scheduling system within the server), that is, the task queue has a call priority, and the call priority of the target task queue matches the application scenario. Afterwards, the processing task is sent to the corresponding target task queue, and in subsequent steps, the task queue is queued and processed, and finally the execution of the processing task is realized. Among them, when there are more than two processing tasks, each processing task is sent to the target task queue separately, and the sending order can be random or based on the generation order of the processing tasks, without restriction.
[0045] Specifically, in a possible implementation, the call request includes target scenario information corresponding to the target application scenario, such as Figure 3 As shown, the specific implementation of step S102 includes:
[0046] Step S1021: Obtain target scenario information by receiving a call request, and process the call request according to the target scenario information to generate at least one processing step for realizing a target function in a target application scenario.
[0047] Step S1022: Generate processing tasks corresponding to each processing step according to the model configuration information, where the model configuration information is used to characterize the vertical domain model currently deployed in the system.
[0048] Exemplarily, in the call request received by the server, the target scene information corresponding to the call request is included, for example, the target scene information is included in a certain header field of the call request. After receiving the call request, the server obtains the target scene information by reading the content of the header field, and then determines the corresponding processing steps according to the target scene information. Among them, exemplarily, the target scene information may include a scene identifier, and the server determines the corresponding processing steps by obtaining the functional description information of the target function contained in the call request and combining the scene identifier. That is, for the same functional description information, when the application scenario (scenario identifier) is different, the server can generate different processing steps, for example, semantic extraction (functional description information) in the video generation scenario, and semantic extraction in the text image scenario, because different prompt words need to be generated and different model services need to be called to realize the corresponding function, so the corresponding processing steps are different. In this embodiment, the processing steps are generated by combining the target scene information with the call request, which can make the processing steps generated based on the call request more accurate, thereby improving the accuracy of the subsequent model service call.
[0049] Afterwards, after the processing task is generated, the vertical domain model deployed by the current service request scheduling system is determined according to the preset model configuration information, that is, the private domain model, to match the above processing steps respectively. Since the processing steps generated through the above process are task description information after field refinement, it is usually necessary to combine the vertical domain model to realize the final function. After obtaining the currently deployed vertical domain model according to the model configuration information, the vertical domain model that matches it can be selected according to the processing steps, thereby generating the corresponding processing task (used to call the vertical domain model) to achieve matching of the processing tasks.
[0050] Step S103: Based on the calling priority of the target task queue, call the target task queue, execute the processing tasks in the target task queue, and obtain the execution result of the target function returned by the corresponding model service.
[0051] Exemplarily, after sending the processing task to the corresponding target task queue, the corresponding consumer thread calls each task queue based on the calling priority, thereby realizing the calling of the target task queue. When calling the target task queue, the above-mentioned processing task in the target task queue is executed, so as to realize the purpose of sending a service request to the corresponding model service. After receiving the service request, the model service responds to it, and processes the target data with its own capabilities to obtain the processing result. Afterwards, according to the role of the model service in realizing the target function, the processing result can be the execution result of the target function or a part of the execution result, or, after one step of processing the processing result, the execution result of the target function is obtained. Finally, the execution result is returned to the terminal device, so that the terminal device side completes the execution of the target function.
[0052] Among them, in one possible implementation, the calling priority can be a corresponding priority identifier. When calling the target task queue according to the calling priority, the server calls the task queue with a priority identifier representing a high priority according to the priority identifier, and processes the processing tasks therein, and processes them in order from high to low according to the priority identifier, until all task queues are called and processed. In another possible implementation, the calling priority can correspond to the queue calling probability of the response, for example, calling priority I corresponds to a queue calling probability of 10%; calling priority II corresponds to a queue calling probability of 5%, and so on. According to the calling priority of the target task queue, the corresponding queue calling probability is determined, including determining the queue calling probability corresponding to the target task queue according to the calling priority of the target task queue. Afterwards, based on the queue calling probability corresponding to the calling priority, each task queue is randomly called until the target task queue is hit, the target task queue is called, and the processing tasks therein are executed.
[0053] Furthermore, in a possible implementation, a service request scheduling system is run in the server, and a task request processing unit and a task consumption unit are deployed inside or outside the service request scheduling system. The above-mentioned corresponding steps are executed by the task request processing unit and the task consumption unit respectively deployed inside or outside the service request scheduling system, wherein the task request processing unit and the task consumption unit can be services maintained in the memory, corresponding to independent threads or processes.
[0054] Figure 4 A schematic diagram of a service request scheduling system provided by an embodiment of the present disclosure, such as Figure 4As shown, after the terminal device makes a call request to the server (service request scheduling system), it is received by the task request processing unit and processed accordingly. Specifically, on the one hand, the task request processing unit obtains the scene information from the call request, and obtains the priority strategy by communicating with the priority configuration unit. The priority strategy represents the mapping relationship between the scene information and the call priority. On the other hand, the task request processing unit communicates with the quota management unit to obtain the traffic quota management strategy, and decides whether to respond to the call request based on the traffic quota management strategy (if the traffic quota is exceeded, no response is made). On the other hand, the task request processing unit generates a processing task based on the target scene information and sends it to the multidimensional priority queue. After that, the task consumption unit obtains and calls the target priority queue from the multidimensional priority queue, and obtains the processing task from it, that is, the task is dequeued. After that, the task consumption unit consumes the processing task to realize the call of various deployed model services, such as the vertical domain model M1, the vertical domain model M2, the general task large model, the external large model, etc. shown in the figure. At the same time, the task consumption unit will obtain the traffic status of the model service by accessing the traffic control unit to avoid the problem of excessive traffic load. Finally, the execution results generated by the model service can be cached in the task result cache area before being returned to the terminal device. At the same time, the execution results are fed back through the traffic control unit to update the traffic quota.
[0055] In the steps of this embodiment, at least one processing task is generated in response to a call request for a target function, and the processing task is used to call the model service corresponding to the target function to process the target data; according to the scenario information corresponding to the call request, the target task queue is determined from multiple task queues, and the processing task is sent to the corresponding target task queue, wherein the scenario information represents the application scenario when the target function is running, and the target task queue has a call priority corresponding to the scenario information; based on the call priority of the target task queue, the target task queue is called, and the processing task in the target task queue is executed to obtain the execution result of the target function returned by the corresponding model service. By obtaining the scenario information corresponding to the call request, and sending the processing task to the corresponding target task queue for processing based on the scenario information, the matching between the use scenario of the call request and the call priority of the task queue is achieved, so that the call request in different application scenarios can be processed with the corresponding priority, thereby improving the overall processing efficiency and response speed of the model service system, thereby improving the user experience on the terminal device side.
[0056] refer to Figure 5 , Figure 5 Schematic diagram of the process of the service request scheduling method provided in the embodiment of the present disclosure Figure 2 In this embodiment Figure 2On the basis of the illustrated embodiment, steps S101-S103 are further refined. For example, the call request includes a target account identifier and target scenario information. A service request scheduling system is running in the server. A task request processing unit and a task consumption unit are deployed inside or outside the service request scheduling system. The service request scheduling method includes:
[0057] Step S201: The task request processing unit receives a call request and obtains a target account identifier and target scenario information.
[0058] For example, reference Figure 4 In the schematic diagram of the service request scheduling system shown in FIG. 1 , after receiving the call request sent by the terminal device, the task request processing unit obtains the target account identifier and the target scenario information from the call request. Then, in a possible implementation, the task request processing unit first determines whether to respond to the call request based on the target account identifier. Specifically, optionally, after obtaining the target account identifier, the following is further included:
[0059] Step S202: The task request processing unit obtains the corresponding available task quota according to the target account identifier. If the available task quota is greater than the corresponding quota threshold, step S203 is executed.
[0060] Exemplarily, the task request processing unit obtains the current available task quota of the account corresponding to the target account identifier according to the target account identifier. If the available task quota is less than the corresponding quota threshold, for example, when the available task quota is less than 1, it means that the target user represents the corresponding account and has no authority to request the model service. In this case, the task request processing unit directly ignores the call request or returns the corresponding prompt information. This situation is not repeated; and if the available task quota is greater than the corresponding quota threshold, for example, when the available task quota is greater than 0, step S203 is executed to respond to it. In the steps of this embodiment, the request authority of the call request is determined by the available task quota, thereby realizing the authority management of the call request for different accounts and improving the flexibility of the service request scheduling system.
[0061] Step S203: The task request processing unit obtains the account level corresponding to the call request according to the target account identifier.
[0062] Step S204: The task request processing unit determines a target task queue from multiple task queues according to the scenario information and account level corresponding to the call request, and sends the processing task to the corresponding target task queue.
[0063] Exemplarily, there is a preset mapping relationship between the account identifier and the account level, and the mapping relationship can be configured through a configuration file or a corresponding instruction, without specific limitation. After the target account identifier is activated, the task request processing unit can determine the account level corresponding to the account according to the target account identifier and the preset mapping relationship. In subsequent steps, the task request processing unit can further implement the control of the call priority based on the account level, so that the call requests corresponding to accounts at different account levels are processed with different priorities.
[0064] For example, Figure 6 As shown, the specific implementation of step S204 includes:
[0065] Step S2041: determining a corresponding target member group according to the account level, wherein the target member group includes at least two task queues with different call priorities;
[0066] Step S2042: Determine the target call priority according to the scenario information, and obtain the target task queue with the target call priority in the target member group.
[0067] Step S2043: Send the processing task to the corresponding target task queue.
[0068] Exemplarily, a member group is a logical grouping, and there is a one-to-many mapping relationship between the account level and the member group. For example, when the account level is 1-3, it corresponds to member group A, specifically, member group A is, for example, a common member group; when the account level is 3-6, it corresponds to member group B, specifically, member group B is, for example, a paid member group; when the account level is 0, it corresponds to member group C, specifically, member group C is, for example, a test member group. Further, each member group corresponds to multiple task queues. For example, member group A corresponds to 10 task queues, namely task queues A_0, A_1, ..., A_9. Member group B corresponds to 12 task queues, namely task queues B_0, B_1, ..., A_11, and so on. Each of the above task queues corresponds to a corresponding call priority. In this embodiment, the task request processing unit first determines a corresponding target member group according to the account level, such as the above member group B, and then determines the corresponding target call priority according to the scenario information, and determines a task queue with the target call priority from the multiple task queues corresponding to the target member group, that is, the target task queue. After that, the above processing tasks can be sent to the target task queue.
[0069] Furthermore, in a possible implementation, the target member group includes at least one document member, and the document member represents a document object carrying data, such as Figure 7 As shown, the specific implementation of step S2042 includes:
[0070] Step S2042-1: Determine the target call priority according to the scene information.
[0071] Step S2042-2: Determine the target document member according to the document object to which the target data belongs.
[0072] Step S2042-3: Obtain a target task queue having a target call priority from at least two task queues corresponding to the target document member.
[0073] For example, in the steps of this embodiment, the structure of the member group is refined. Specifically, the member group contains three layers of structure, the first layer is the member group, the second layer is the document member, and the third layer is the task queue. Figure 6 In the illustrated embodiment, an additional structural layer of document members is provided. Figure 8 A schematic diagram of a member group structure provided in an embodiment of the present disclosure is shown below in conjunction with Figure 8 , the process of determining the target task queue is introduced. By way of example, first, according to the account level, a corresponding target member group is determined from multiple member groups (such as member group A, member group B, and member group C), such as member group A shown in the figure. Then, on the one hand, according to the scenario information, the target call priority is determined. The specific implementation process has been introduced in the previous embodiment and will not be repeated here. On the other hand, according to the document object to which the target data belongs, the target document member is determined, such as document member R1 shown in the figure. Then, according to the above target call priority, a target task queue with a target call priority is obtained from at least two task queues corresponding to document member R1, such as task queue A_R1_1.
[0074] In this embodiment, the target data is the data to be processed corresponding to the call request, such as a column of data in a table document, more specifically, 1,000 user comments. The document object to which the target data belongs is the table document that records the target data. Since the model service usually needs to target the document object and occupy a document member for processing when processing the target data, the document member can be understood as a resource provided by the model service provider (such as a cloud computing service provider), and different document members have different numbers of task queues, such as document member R1, which has 10 task queues, and document member R2, which has 100 task queues. Therefore, when processing the target data based on document member R1 and processing the target data based on document member R2, there will be different response speeds, thereby realizing differentiated allocation of model service resources.
[0075] Of course, it is understandable that when the carrier of the target data is different, the document object will also change accordingly. For example, it can be a picture document, a mixed picture and text document, etc. The above document objects can all be matched and processed through document members.
[0076] Step S205: the task consumption unit obtains the target member group where the target task queue is located, wherein different member groups correspond to different member group calling probabilities.
[0077] Step S206: The task consumption unit determines the corresponding queue calling probability according to the calling priority of the target task queue.
[0078] Step S207: The task consumption unit obtains a comprehensive calling probability according to the queue calling probability and the member group calling probability.
[0079] Step S208: The task consumption unit randomly calls each task queue based on the comprehensive calling probability corresponding to each task queue, and calls the target task queue when hitting the target task queue.
[0080] Exemplarily, after step S204 is executed, the steps executed by the task request processing unit are completed, that is, the generation process of the processing task is completed. After that, the processing task is consumed by the task consumption unit, that is, the task consumption unit needs to obtain the processing task from the target task queue and execute it. Specifically, the task consumption unit first needs to obtain the target member group where the target task queue is located, where different member groups correspond to different member group call probabilities. For example, member group A corresponds to a 50% member group call probability, member group B corresponds to a 30% member group call probability, and member group C corresponds to a 20% member group call probability. The task consumption unit obtains each member group including the target member group based on the member group call probability of each member group by random selection. After that, the task consumption unit obtains the corresponding queue call probability according to the call priority of the target task queue. The implementation process has been introduced before and will not be repeated. After that, the comprehensive call probability is obtained according to the product of the queue call probability and the member group call probability. For example, member group A corresponds to a member group call probability of 50%. According to the call priority, the queue call probability of the target task queue in member group A is 8%, and the corresponding comprehensive call probability is 4%. Afterwards, the task consumption unit makes a random call according to the comprehensive call probability until it hits the target task queue and calls it.
[0081] In another possible implementation, the task consumption unit first randomly calls the member group according to the member group call probability corresponding to each member group. After hitting the target member group, it then makes a random call based on the queue call probability of each task queue in the target member group (including the queue call probability of the target task queue) until the target task queue is hit and the call is made.
[0082] In the steps of this embodiment, since the queue call probability corresponding to each task queue matches the scenario information, and the member group call probability of the target member group matches the account level, the comprehensive call probability obtained according to the above probability takes into account the influence of the account level and the application scenario factors, so that the processing tasks in the called task queue can respond based on the delay requirements and service levels determined by the application scenario and account level, so that the processing tasks can be triggered reasonably and orderly, avoiding the problem of delayed response of important processing tasks and improving the user experience.
[0083] Step S209: the task consumption unit obtains the processing task at the current sorting position based on the sorting of the processing tasks in the target task queue each time the target task queue is called.
[0084] Step S210: When the task consumption unit obtains the target processing task, it executes the target processing task, calls the corresponding target model service, and obtains the inference result returned by the target model service.
[0085] Exemplarily, after determining the target task queue, the task consumption unit reads and executes the processing tasks in sequence based on the sorting of each processing task in the target task queue, so as to call the corresponding model service to process the target data. It should be noted that there may be multiple processing tasks in the target task queue, including processing tasks corresponding to different call requests sent by the same terminal device, and processing tasks corresponding to different call requests sent by different terminal devices. Each time the thread corresponding to the task consumption unit calls the target task queue, it obtains the processing task in the current sorting position and executes it, then releases the processing resources of the target task queue, and re-selects randomly until the next call to the target task queue, repeating the above steps. Until the task consumption unit obtains the target processing task corresponding to the call request referred to in this embodiment and executes it, the corresponding target model service will be called to obtain the inference result returned by the target model service.
[0086] For example, Fig. 9 As shown, the specific implementation of step S210 includes:
[0087] Step S2101: In response to the execution of the target processing task, detecting the current real-time traffic and / or flow limiting status of the target model service;
[0088] Step S2102: Call the target model service according to the current real-time traffic and / or flow limiting status of the target model service to obtain the inference result returned by the target model service.
[0089] Exemplarily, during the execution of step S201, when the task consumption unit obtains a processing task (including a target processing task), it will respond to the execution of the target processing task and detect the current real-time traffic and / or flow limiting state of the target model service through the traffic control unit. If the real-time traffic is greater than or equal to the traffic threshold, or the flow limiting state is in a flow-limited state, the target processing task will not be executed, thereby achieving protection and flow limiting of the model service called by the target processing service, avoiding jamming and high latency caused by high load. On the contrary, if the real-time traffic is less than the traffic threshold, or the flow limiting state is in a non-flow limiting state, or the real-time traffic is less than the traffic threshold and the flow limiting state is in a non-flow limiting state, the target model service can be executed normally to obtain the inference result returned by the target model service.
[0090] Furthermore, in a possible implementation, the method further includes:
[0091] According to the calling frequency of each described task queue, and / or, the service load corresponding to the member group where the task queue is located, the calling priority corresponding to the described task queue is adjusted, wherein, the adjustment increment of the described task queue is inversely proportional to the described calling frequency and directly proportional to the described service load. Exemplarily, the calling frequency is the frequency at which the task queue is called, and the higher the calling frequency corresponding to the task queue is, the more times the task queue is called per unit time, and the specific implementation mode of calling the task queue is introduced in detail in the subsequent embodiment steps. And the service load is the sum of the task loads of the task queues corresponding to the member groups. When the task loads of each task queue corresponding to a member group are large, the service load of the member group is also large, and vice versa. When the calling frequency of the task queue is too low, and / or, the service load corresponding to the member group where the task queue is located is too large, it will affect the overall operation efficiency of the system. In this case, adjusting the calling priority corresponding to the corresponding task queue according to the calling frequency and service load can play a role in improving the overall operation efficiency of the system. Among them, the adjustment increase of the task queue is inversely proportional to the calling frequency and directly proportional to the service load, that is, the lower the calling frequency of the task queue, the greater the increase in the calling priority of the task queue; and the higher the service load of the member group where the task queue is located, the smaller the increase in the calling priority of the task queue, or the lower the calling priority. In a possible implementation, the calling frequency and service load can be used separately to adjust the calling priority of the task queue, or they can be used together to adjust the calling priority of the task queue, wherein the adjustment amount corresponding to the calling frequency and service load can be set according to experience and actual needs, and there is no restriction this time.
[0092] Corresponding to the service request scheduling method of the above embodiment, Fig.10 A structural block diagram of a service request scheduling device provided in an embodiment of the present disclosure. The method described in the above embodiment can be executed by the service request scheduling device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. The electronic device can include but is not limited to a mobile terminal with big data processing capabilities, and a fixed terminal with big data processing capabilities such as a desktop computer and a supercomputer.
[0093] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig.10 , the service request scheduling device 3 comprises:
[0094] A response module 31, configured to generate at least one processing task in response to a call request for a target function, wherein the processing task is used to call a model service corresponding to the target function to process target data;
[0095] The processing module 32 is used to determine a target task queue from multiple task queues according to the scenario information corresponding to the call request, and send the processing task to the corresponding target task queue, wherein the scenario information represents the application scenario when the target function is run, and the target task queue has a call priority corresponding to the scenario information;
[0096] The calling module 33 is used to call the target task queue based on the calling priority of the target task queue, and execute the processing tasks in the target task queue to obtain the execution result of the target function returned by the corresponding model service.
[0097] According to one or more embodiments of the present disclosure, the call request includes target scenario information corresponding to the target application scenario, and the response module 31 is specifically used to: obtain the target scenario information by receiving the call request, and process the call request according to the target scenario information, and generate at least one processing step for realizing the target function in the target application scenario; generate processing tasks corresponding to each processing step according to the model configuration information, and the model configuration information is used to characterize the vertical domain model currently deployed in the system.
[0098] According to one or more embodiments of the present disclosure, the call request includes a target account identifier, and the response module 31 is specifically used to: obtain the target account identifier by receiving the call request; obtain the corresponding available task quota according to the target account identifier; and generate at least one processing task when the available task quota is greater than the corresponding quota threshold.
[0099] According to one or more embodiments of the present disclosure, the response module 31 is further used to: obtain the account level corresponding to the call request; the processing module 32 is specifically used to: determine the target task queue from multiple task queues according to the scenario information and account level corresponding to the call request.
[0100] According to one or more embodiments of the present disclosure, when the processing module 32 determines the target task queue from multiple task queues based on the scenario information and account level corresponding to the call request, it is specifically used to: determine the corresponding target member group based on the account level, the target member group including at least two task queues with different call priorities; determine the target call priority based on the scenario information, and obtain the target task queue with the target call priority within the target member group.
[0101] According to one or more embodiments of the present disclosure, the target member group includes at least one document member, and the document member represents a document object that carries data; the processing module 32 is also used to: determine the target document member according to the document object to which the target data belongs; obtain at least two task queues corresponding to the target document member; when the processing module 32 obtains the target task queue with the target call priority in the target member group, it is specifically used to: obtain the target task queue with the target call priority in the at least two task queues corresponding to the target document member.
[0102] According to one or more embodiments of the present disclosure, the calling module 33 is specifically used to: determine the corresponding queue calling probability according to the calling priority of the target task queue; perform random calling based on the queue calling probability until the target task queue is hit, and then call the target task queue.
[0103] According to one or more embodiments of the present disclosure, the calling module 33 is also used to: obtain the target member group where the target task queue is located, wherein different member groups correspond to different member group calling probabilities; obtain a comprehensive calling probability based on the queue calling probability and the member group calling probability; the calling module 33 makes a random call based on the queue calling probability until the target task queue is hit, and when the target task queue is called, it is specifically used to: make a random call to each task queue based on the comprehensive calling probability corresponding to each task queue, until the target task queue is hit, and then the target task queue is called.
[0104] According to one or more embodiments of the present disclosure, when the calling module 33 executes the processing task in the target task queue and obtains the execution result of the target function returned by the corresponding model service, it is specifically used to: based on the sorting of each processing task in the target task queue, obtain the processing task in the current sorting position after the target task queue is called each time; in response to the execution of the target processing task, call the corresponding target model service to obtain the inference result returned by the target model service.
[0105] According to one or more embodiments of the present disclosure, when the calling module 33 calls the corresponding target model service in response to the execution of the target processing task and obtains the inference result returned by the target model service, it is specifically used to: detect the current real-time traffic and / or flow limiting status of the target model service in response to the execution of the target processing task; call the target model service based on the current real-time traffic and / or flow limiting status of the target model service to obtain the inference result returned by the target model service.
[0106] The response module 31, the processing module 32 and the calling module 33 are connected in sequence. The service request scheduling 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.
[0107] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the electronic device 4 includes:
[0108] A processor 41, and a memory 42 communicatively connected to the processor 41;
[0109] The memory 42 stores computer executable instructions;
[0110] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 9 The service request scheduling method in the illustrated embodiment.
[0111] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .
[0112] For related instructions, please refer to Figure 2-Figure 9 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.
[0113] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 9 The service request scheduling method provided by any one of the corresponding embodiments.
[0114] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 9 The service request scheduling method provided by any one of the corresponding embodiments.
[0115] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.
[0116] refer to Fig.12, which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0117] like Fig.12 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0118] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0119] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0120] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0121] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0122] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.
[0123] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0125] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.
[0126] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0127] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0128] In a first aspect, according to one or more embodiments of the present disclosure, a service request scheduling method is provided, including:
[0129] In response to a call request for a target function, at least one processing task is generated, and the processing task is used to call the model service corresponding to the target function to process target data; according to the scenario information corresponding to the call request, a target task queue is determined from multiple task queues, and the processing task is sent to the corresponding target task queue, wherein the scenario information represents an application scenario when running the target function, and the target task queue has a calling priority corresponding to the scenario information; based on the calling priority of the target task queue, the target task queue is called, and the processing task in the target task queue is executed to obtain the execution result of the target function returned by the corresponding model service.
[0130] According to one or more embodiments of the present disclosure, the call request includes target scenario information corresponding to the target application scenario, and the response to the call request for the target function generates at least one processing task, including: obtaining the target scenario information by receiving the call request, and processing the call request according to the target scenario information, and generating at least one processing step for implementing the target function in the target application scenario; generating processing tasks corresponding to each of the processing steps according to the model configuration information, and the model configuration information is used to characterize the vertical domain model currently deployed in the system.
[0131] According to one or more embodiments of the present disclosure, the call request includes a target account identifier, and the at least one processing task is generated in response to the call request for the target function, including: obtaining the target account identifier by receiving the call request; obtaining a corresponding available task amount according to the target account identifier; and generating the at least one processing task when the available task amount is greater than a corresponding amount threshold.
[0132] According to one or more embodiments of the present disclosure, the method further includes: obtaining an account level corresponding to the call request; determining a target task queue from a plurality of task queues based on the scenario information corresponding to the call request, including: determining a target task queue from a plurality of task queues based on the scenario information and the account level corresponding to the call request.
[0133] According to one or more embodiments of the present disclosure, determining a target task queue from multiple task queues based on the scenario information and the account level corresponding to the call request includes: determining a corresponding target member group based on the account level, the target member group containing at least two task queues with different call priorities; determining a target call priority based on the scenario information, and obtaining a target task queue with the target call priority within the target member group.
[0134] According to one or more embodiments of the present disclosure, the target member group includes at least one document member, and the document member represents a document object that carries data; the method also includes: determining the target document member according to the document object to which the target data belongs; obtaining at least two task queues corresponding to the target document member; obtaining the target task queue with the target call priority within the target member group, including: obtaining the target task queue with the target call priority among the at least two task queues corresponding to the target document member.
[0135] According to one or more embodiments of the present disclosure, calling the target task queue based on the calling priority of the target task queue includes: determining a corresponding queue calling probability based on the calling priority of the target task queue; and making a random call based on the queue calling probability until the target task queue is hit, and then calling the target task queue.
[0136] According to one or more embodiments of the present disclosure, the method also includes: obtaining the target member group where the target task queue is located, wherein different member groups correspond to different member group call probabilities; obtaining a comprehensive call probability based on the queue call probability and the member group call probability; the random call based on the queue call probability until the target task queue is hit, calling the target task queue, including: based on the comprehensive call probability corresponding to each task queue, randomly calling each task queue until the target task queue is hit, calling the target task queue.
[0137] According to one or more embodiments of the present disclosure, executing the processing tasks in the target task queue to obtain the execution result of the target function returned by the corresponding model service includes: based on the sorting of each of the processing tasks in the target task queue, obtaining the processing task at the current sorting position after the target task queue is called each time; in response to the execution of the target processing task, calling the corresponding target model service to obtain the inference result returned by the target model service.
[0138] According to one or more embodiments of the present disclosure, in response to the execution of the target processing task, the corresponding target model service is called to obtain the inference result returned by the target model service, including: in response to the execution of the target processing task, detecting the current real-time traffic and / or flow limiting status of the target model service; based on the current real-time traffic and / or flow limiting status of the target model service, calling the target model service to obtain the inference result returned by the target model service.
[0139] In a second aspect, according to one or more embodiments of the present disclosure, a service request scheduling device is provided, including:
[0140] A response module, configured to generate at least one processing task in response to a call request for a target function, wherein the processing task is used to call a model service corresponding to the target function to process target data;
[0141] a processing module, configured to determine a target task queue from a plurality of task queues according to scenario information corresponding to the call request, and send the processing task to the corresponding target task queue, wherein the scenario information represents an application scenario when the target function is run, and the target task queue has a call priority corresponding to the scenario information;
[0142] The calling module is used to call the target task queue based on the calling priority of the target task queue, and execute the processing tasks in the target task queue to obtain the execution result of the target function returned by the corresponding model service.
[0143] According to one or more embodiments of the present disclosure, the call request includes target scenario information corresponding to the target application scenario, and the response module is specifically used to: obtain the target scenario information by receiving the call request, and process the call request according to the target scenario information, and generate at least one processing step for realizing the target function in the target application scenario; generate processing tasks corresponding to each of the processing steps according to the model configuration information, and the model configuration information is used to characterize the vertical domain model currently deployed in the system.
[0144] According to one or more embodiments of the present disclosure, the call request includes a target account identifier, and the response module is specifically used to: obtain the target account identifier by receiving the call request; obtain the corresponding available task quota according to the target account identifier; and generate the at least one processing task when the available task quota is greater than the corresponding quota threshold.
[0145] According to one or more embodiments of the present disclosure, the response module is further used to: obtain the account level corresponding to the call request; the processing module is specifically used to: determine the target task queue from multiple task queues based on the scenario information and account level corresponding to the call request.
[0146] According to one or more embodiments of the present disclosure, when the processing module determines the target task queue from multiple task queues based on the scenario information and account level corresponding to the call request, it is specifically used to: determine the corresponding target member group based on the account level, the target member group containing at least two task queues with different call priorities; determine the target call priority based on the scenario information, and obtain the target task queue with the target call priority within the target member group.
[0147] According to one or more embodiments of the present disclosure, the target member group includes at least one document member, and the document member represents a document object that carries data; the processing module is also used to: determine the target document member according to the document object to which the target data belongs; obtain at least two task queues corresponding to the target document member; when the processing module obtains the target task queue with the target call priority within the target member group, it is specifically used to: obtain the target task queue with the target call priority from the at least two task queues corresponding to the target document member.
[0148] According to one or more embodiments of the present disclosure, the calling module is specifically used to: determine the corresponding queue calling probability according to the calling priority of the target task queue; perform random calling based on the queue calling probability until the target task queue is hit, and then call the target task queue.
[0149] According to one or more embodiments of the present disclosure, the calling module is also used to: obtain the target member group where the target task queue is located, wherein different member groups correspond to different member group calling probabilities; obtain a comprehensive calling probability based on the queue calling probability and the member group calling probability; the calling module makes a random call based on the queue calling probability until the target task queue is hit, and then calls the target task queue, specifically for: based on the comprehensive calling probability corresponding to each task queue, randomly calls each task queue until the target task queue is hit, and then calls the target task queue.
[0150] According to one or more embodiments of the present disclosure, when the calling module executes the processing task in the target task queue and obtains the execution result of the target function returned by the corresponding model service, it is specifically used to: based on the sorting of each of the processing tasks in the target task queue, obtain the processing task in the current sorting position after the target task queue is called each time; in response to the execution of the target processing task, call the corresponding target model service to obtain the inference result returned by the target model service.
[0151] According to one or more embodiments of the present disclosure, when the calling module calls the corresponding target model service in response to the execution of the target processing task and obtains the inference result returned by the target model service, it is specifically used to: detect the current real-time traffic and / or flow limiting status of the target model service in response to the execution of the target processing task; call the target model service according to the current real-time traffic and / or flow limiting status of the target model service to obtain the inference result returned by the target model service.
[0152] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;
[0153] The memory stores computer-executable instructions;
[0154] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the service request scheduling method as described in the first aspect and various possible designs of the first aspect.
[0155] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the service request scheduling method described in the first aspect and various possible designs of the first aspect is implemented.
[0156] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the service request scheduling method described in the first aspect and various possible designs of the first aspect.
[0157] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0158] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0159] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A service request scheduling method, characterized in that: include: In response to a call request for a target function, generating at least one processing task, the processing task being used to call a model service corresponding to the target function to process target data; Determine a target task queue from a plurality of task queues according to the scenario information corresponding to the call request, and send the processing task to the corresponding target task queue, wherein the scenario information represents an application scenario when the target function is run, and the target task queue has a call priority corresponding to the scenario information; Based on the calling priority of the target task queue, the target task queue is called, and the processing tasks in the target task queue are executed to obtain the execution result of the target function returned by the corresponding model service.
2. The method according to claim 1, characterized in that The call request includes target scenario information corresponding to the target application scenario, and the generating of at least one processing task in response to the call request for the target function includes: By receiving the call request, obtaining the target scenario information, and processing the call request according to the target scenario information, generating at least one processing step for realizing the target function in the target application scenario; According to the model configuration information, a processing task corresponding to each of the processing steps is generated, and the model configuration information is used to characterize the vertical domain model currently deployed in the system.
3. The method according to claim 1, characterized in that The calling request includes a target account identifier, and the generating of at least one processing task in response to the calling request for the target function includes: Obtaining the target account identifier by receiving the call request; According to the target account identifier, obtain the corresponding available task quota; When the available task quota is greater than the corresponding quota threshold, the at least one processing task is generated.
4. The method according to claim 1, characterized in that: The method further comprises: Obtaining the account level corresponding to the call request; The step of determining a target task queue from a plurality of task queues according to the scenario information corresponding to the call request includes: According to the scenario information and the account level corresponding to the call request, a target task queue is determined from multiple task queues.
5. The method according to claim 4, characterized in that The step of determining a target task queue from a plurality of task queues according to the scenario information and the account level corresponding to the call request includes: Determine a corresponding target member group according to the account level, wherein the target member group includes at least two task queues with different call priorities; According to the scenario information, a target call priority is determined, and within the target member group, a target task queue having the target call priority is obtained.
6. The method according to claim 5, characterized in that The target member group includes at least one document member, and the document member represents a document object carrying data; the method further includes: Determine a target document member according to the document object to which the target data belongs; Obtain at least two task queues corresponding to the target document member; The step of obtaining a target task queue having the target call priority within the target member group includes: From the at least two task queues corresponding to the target document member, a target task queue having the target call priority is obtained.
7. The method according to claim 1, characterized in that The calling of the target task queue based on the calling priority of the target task queue includes: Determine the corresponding queue calling probability according to the calling priority of the target task queue; A random call is made based on the queue calling probability until the target task queue is hit, and then the target task queue is called.
8. The method according to claim 7, characterized in that The method further comprises: Obtaining a target member group where the target task queue is located, wherein different member groups correspond to different member group call probabilities; Obtaining a comprehensive calling probability according to the queue calling probability and the member group calling probability; The randomly calling based on the queue calling probability until the target task queue is hit, calling the target task queue, comprises: Based on the comprehensive calling probability corresponding to each task queue, each task queue is randomly called until the target task queue is hit, and then the target task queue is called.
9. The method according to claim 1, characterized in that: The executing the processing task in the target task queue to obtain the execution result of the target function returned by the corresponding model service includes: Based on the sorting of the processing tasks in the target task queue, after the target task queue is called each time, obtaining the processing task at the current sorting position; In response to the execution of the target processing task, the corresponding target model service is called to obtain the inference result returned by the target model service.
10. The method according to claim 9, characterized in that In response to the execution of the target processing task, calling the corresponding target model service to obtain the inference result returned by the target model service includes: In response to the execution of the target processing task, detecting the current real-time traffic and / or flow limiting status of the target model service; According to the current real-time traffic and / or flow limiting status of the target model service, the target model service is called to obtain the inference result returned by the target model service.
11. The method according to claim 1, characterized in that: The method further comprises: The calling priority corresponding to the task queue is adjusted according to the calling frequency of each task queue and / or the service load corresponding to the member group where the task queue is located, wherein the adjustment increase of the task queue is inversely proportional to the calling frequency and directly proportional to the service load.
12. A service request scheduling device, characterized in that: include: A response module, configured to generate at least one processing task in response to a call request for a target function, wherein the processing task is used to call a model service corresponding to the target function to process target data; a processing module, configured to determine a target task queue from a plurality of task queues according to scenario information corresponding to the call request, and send the processing task to the corresponding target task queue, wherein the scenario information represents an application scenario when the target function is run, and the target task queue has a call priority corresponding to the scenario information; The calling module is used to call the target task queue based on the calling priority of the target task queue, and execute the processing tasks in the target task queue to obtain the execution result of the target function returned by the corresponding model service.
13. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the service request scheduling method according to any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the service request scheduling method according to any one of claims 1 to 11 is implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the service request scheduling method according to any one of claims 1 to 11 is implemented.
Citation Information
Cited By
Service providing method and device, electronic equipment and storage medium
CN120856780A
Emotion analysis method and device, wearable equipment and storage medium
CN121167153A
Data processing method and device for high-concurrency scene of charging service, equipment, storage medium and program product
CN121388228A
Service request scheduling method and apparatus, and electronic device and storage medium
WO2026016454A1
