Inference method, inference system, inference device, equipment and storage medium

Abstract

The invention provides a reasoning method, a reasoning system, a reasoning device, equipment and a storage medium, and relates to the field of data processing, in particular to the technical fields of artificial intelligence, big data, large models and the like. According to the specific implementation scheme, a first task request is received through a first scheduling service in a plurality of scheduling services, and the first task request is a request which is sent by a first reasoning service in a plurality of reasoning services and at least carries a model identifier and current idle reasoning resources of the first reasoning service; determining a target reasoning request matched with the first task request based on a model identifier and idle reasoning resources carried by the first task request; and sending the target reasoning request to a first reasoning service through the first scheduling service.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to technical fields such as artificial intelligence, big data, and big models. Background Art

[0002] With the rapid development of artificial intelligence technology, large models have become a research hotspot in the field of artificial intelligence (AI). They can extract knowledge from massive data and perform well in a variety of tasks. In order to apply large models to actual scenarios and meet the needs of massive users, it is usually necessary to deploy large model reasoning services in a clustered manner. However, the existing clustered deployment method is difficult to reasonably schedule resources based on the characteristics of reasoning itself. Summary of the invention

[0003] The present disclosure provides an inference method, an inference system, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, there is provided a reasoning method, the method comprising:

[0005] Receiving a first task request through a first scheduling service among the multiple scheduling services, wherein the first task request is a request sent by a first inference service among the multiple inference services and carries at least a model identifier and current idle inference resources of the first inference service;

[0006] Determine, based on the model identifier and the idle reasoning resources carried by the first task request, a target reasoning request matching the first task request;

[0007] According to another aspect of the present disclosure, there is provided a reasoning system, comprising:

[0008] A first scheduling service is used to receive a first task request, wherein the first task request carries at least a model identifier and a current idle inference resource of the first inference service; based on the model identifier and the idle inference resource carried by the first task request, determine a target inference request that matches the first task request, and send the target inference request;

[0009] The first reasoning service is used to send a first task request; it is also used to obtain a target reasoning request that matches the first task request; respond to the target reasoning request to perform generative reasoning to obtain at least part of the reasoning result of the target reasoning request.

[0010] According to another aspect of the present disclosure, there is provided an inference device, comprising:

[0011] An acquisition unit, configured to receive a first task request through a first scheduling service among multiple scheduling services, wherein the first task request is a request sent by a first inference service among multiple inference services and carries at least a model identifier and currently idle inference resources of the first inference service;

[0012] A scheduling unit, configured to determine a target reasoning request matching the first task request based on a model identifier and idle reasoning resources carried by the first task request;

[0013] A sending unit is used to send the target reasoning request to the first reasoning service through the first scheduling service.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] In this way, the disclosed solution can obtain the current load situation of the first reasoning service (such as idle reasoning resources) based on the first task request actively sent by the first reasoning service, and then enable the scheduling service to determine the target reasoning request that matches the load situation, and send the target reasoning request to the first reasoning service for reasoning. Compared with the traditional polling strategy, the disclosed solution fully combines the characteristics of the large model reasoning service, effectively realizes the reasonable scheduling of resources, avoids idle resources due to load imbalance, and at the same time, ensures that the reasoning requests are processed in a timely manner, thereby improving resource utilization and ensuring system stability.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a diagram showing the scheduling results of an inference server cluster in an example;

[0024] Figure 2 This is a schematic flow chart of the reasoning method according to an embodiment of the present application. Figure 1 ;

[0025] Figure 3 This is a schematic flow chart of the reasoning method according to an embodiment of the present application. Figure 2 ;

[0026] Figure 4 is a schematic diagram of the structure of an inference system according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a scenario in a specific example of an inference system according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of the structure of an inference device according to an embodiment of the present application;

[0029] Figure 7 It is a block diagram of an electronic device used to implement the reasoning method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0032] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0033] With the rapid development of artificial intelligence technology, large models have become an important research direction in the field of artificial intelligence (AI). They can learn rich knowledge representations from massive data and then show excellent performance in various tasks. However, to apply these powerful large models to actual scenarios, they need to be service-oriented, such as providing services to the outside world through application programming interfaces (APIs), so that other applications can easily call on the capabilities of large models.

[0034] In actual applications, in order to meet the needs of a large number of users, the reasoning service of large models is usually deployed in a cluster. This cluster deployment faces two challenges: (1) improving the resource utilization of the system; (2) ensuring the stability of the system. Among them, the improvement of system resource utilization can avoid resource waste, and thus can respond to more concurrent requests from users more quickly; and the stability of the system is the basic quality of a reliable commercial system. To meet the above challenges, an efficient load balancing algorithm is needed to schedule cluster resources. However, the existing cluster load balancing components are usually general-purpose, and it is difficult to perform reasonable resource scheduling based on the characteristics of large model reasoning. This is because the existing load balancing components use general load scheduling strategies such as polling and minimum number of connections, and cannot perceive the actual load of the backend reasoning cluster, so it is easy to cause load imbalance in the reasoning cluster, resulting in a waste of computing resources.

[0035] For example, suppose the backend inference server cluster has two instances, such as Figure 1 As shown in the figure, they are inference service 1 and inference service 2. The gateway responsible for load balancing uses a polling strategy to distribute traffic. The number of concurrent requests is 2. Now three requests need to be sent. The generation length of the model corresponding to each of these three requests is 128, 512, and 128 respectively. At this time, the request process is:

[0036] (1) The gateway (a unified entry point for accessing the service-oriented cluster, acting as a reverse proxy and supporting a common load balancing algorithm) sends request 1 to inference service 1 and request 2 to inference service 2. Since request 1 is generated in a shorter length, inference service 1 will complete inference earlier.

[0037] (2) After receiving the inference result of request 1, the target object sends request 3. At this time, the gateway may send request 3 to inference service 2; inference service 2 is still processing inference request 2, while inference service 1 is idle.

[0038] From the above example, we can see that if request 3 should be sent to the idle inference service 1, but the polling strategy causes the gateway to send request 3 to the inference service 2, which causes the load imbalance of the inference server cluster.

[0039] Based on this, the disclosed solution proposes an inference method that can fully combine the characteristics of large model inference for graphics card use, actively perceive the load and hardware capacity of a single model inference service at the backend, and collaboratively optimize to keep the load of the inference server cluster balanced, thereby better improving the overall system resource utilization and ensuring system stability.

[0040] Specifically, Figure 2 This is a schematic flow chart of the reasoning method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0041] Further, the method at least includes at least part of the following contents. Here, it should be noted that the disclosed solution can be specifically applied to the scheduling server or the scheduling server cluster side; further, the disclosed solution can be integrated into the scheduling service component (or load balancing component), at this time, the scheduling service component can be integrated into any device, for example, integrated into a gateway, etc., so that the device integrated with the scheduling service component has the scheduling capability described in the disclosed solution, thereby efficiently completing the reasoning task.

[0042] Specifically, Figure 2 As shown, including:

[0043] Step S201: receiving a first task request through a first scheduling service among multiple scheduling services.

[0044] Here, the first task request is a request sent by a first reasoning service among multiple reasoning services, and carries at least a model identifier and current idle reasoning resources of the first reasoning service.

[0045] Here, it should be noted that the inference service can be specifically understood as a computing service that can use a pre-trained machine learning model to process input data and output prediction or decision results. It is a core component of artificial intelligence (AI) and machine learning (ML) applications and can be executed in an inference server. Specifically, the inference service can specifically refer to a machine learning model deployed on a server or server cluster, which can receive data, process it, and return inference results.

[0046] Correspondingly, the scheduling service can also be specifically understood as being executed on a scheduling server or a scheduling server cluster and being responsible for scheduling.

[0047] That is to say, in this example, the first reasoning service actively sends a first task request to the first scheduling service, and carries its own idle reasoning resources and model identifier to actively request to execute the reasoning task that it is capable of.

[0048] Furthermore, in one example, in order to meet multiple concurrency requirements, a gateway may be set up. At this time, the first reasoning service may send a first task request carrying at least a model identifier and the current idle reasoning resources of the first reasoning service to the first scheduling service via the gateway.

[0049] Furthermore, in one example, the model identifier carried by the first task request can be specifically understood as a unique identifier (also referred to as a model identifier) ​​of the model used by the first reasoning service, for example, can be recorded as model_id.

[0050] Further, in one example, the idle inference resources include, but are not limited to, load conditions and remaining capacity of hardware.

[0051] Step S202: Based on the model identifier and idle reasoning resources carried by the first task request, determine a target reasoning request that matches the first task request.

[0052] Here, the target reasoning request carries at least the target reasoning task to be executed and the model identifier of the target model required to execute the target reasoning task (ie, the unique identifier of the target model).

[0053] That is to say, through the first scheduling service, and based on the model identifier and idle reasoning resources carried by the first task request, a target reasoning request matching the first task request is determined. In other words, the first scheduling service determines the target reasoning request that the first reasoning service can handle based on the model identifier and idle reasoning resources.

[0054] Here, in one example, the target model may be specifically a large model, and further, may be specifically a large language model, or may be other models, which is not limited in the present disclosure.

[0055] Step S203: sending the target reasoning request to the first reasoning service through the first scheduling service, so that the first reasoning service executes the target reasoning task carried by the target reasoning request.

[0056] In this way, the disclosed solution can obtain the current load situation of the first reasoning service (such as idle reasoning resources) based on the first task request actively sent by the first reasoning service, and then enable the scheduling service to determine the target reasoning request that matches the load situation, and send the target reasoning request to the first reasoning service for reasoning. Compared with the traditional polling strategy, the disclosed solution fully combines the characteristics of the large model reasoning service, effectively realizes the reasonable scheduling of resources, avoids idle resources due to load imbalance, and at the same time, ensures that the reasoning requests are processed in a timely manner, thereby improving resource utilization and ensuring system stability.

[0057] Furthermore, since the scheduling service and reasoning service in the disclosed solution can be deployed in a clustered manner, whether a failure occurs in the scheduling service or the reasoning service, it will not affect the normal function of the overall system, so that the system has a certain fault tolerance and high availability, thereby further ensuring the reliability and stability of the system operation.

[0058] Further, in a specific example, the target reasoning request can be obtained in the following manner; specifically, the above-described determination of the target reasoning request matching the first task request based on the model identifier and the idle reasoning resources carried by the first task request (for example, step S202) specifically includes:

[0059] Through the first scheduling service, a target inference request that matches the model identifier in the first task request and for which the idle inference resources meet the inference requirements is selected from a preset request queue of the target database.

[0060] Here, the preset request queue stores a plurality of inference requests, and each inference request carries at least an inference task and a model identifier of a target model required to be used in the inference task.

[0061] That is to say, after receiving the first task request, the first scheduling service will select, from the preset request queue, an inference request that matches the model identifier carried by the first task request and whose idle inference resources meet the inference requirements of the inference task, as the target inference request, and then send the target inference request to the first inference service.

[0062] In this way, the disclosed solution can intelligently select inference requests that match the current actual load situation from the target database based on the current actual load situation of the inference service. In this way, reasonable allocation of resources is achieved, and resource idleness due to load imbalance is effectively avoided, thereby improving the utilization rate of computing resources.

[0063] Figure 3 This is a schematic flow chart of the reasoning method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 2 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0064] Further, the method includes at least part of the following contents. Figure 3 As shown, including:

[0065] Step S301: receiving a first task request through a first scheduling service among multiple scheduling services.

[0066] Here, the first task request is a request sent by a first reasoning service among multiple reasoning services, and carries at least a model identifier and current idle reasoning resources of the first reasoning service.

[0067] Here, for the related contents such as scheduling service, reasoning service, etc., please refer to the above description, which will not be repeated here.

[0068] Step S302: Determine, through the first scheduling service, the target reasoning resource required for the reasoning task based on the characteristic information of the reasoning task carried in the reasoning request.

[0069] For instance, in one example, the first scheduling service may determine the target reasoning resources required for each reasoning request in the preset request queue, thereby facilitating subsequent matching; or, in another example, the first scheduling service determines the target reasoning resources required for the reasoning request with the earliest timing one by one based on the first-in-first-out output timing of the preset request queue, thereby performing matching.

[0070] For example, in one example, the characteristic information of the reasoning task carried by the reasoning request may specifically be the input information and the required output requirements of the reasoning task.

[0071] Here, the execution order of step S301 and step S302 can be interchanged. Alternatively, in one example, the first scheduling service can predetermine the target reasoning resources required for each reasoning request, thereby laying a foundation for subsequent rapid matching to obtain the target reasoning request.

[0072] Step S303: through the first scheduling service, select a target inference request that matches the model identifier in the first task request and whose target inference resource is not greater than the idle inference resource from the multiple inference requests included in the preset request queue.

[0073] Step S304: Send the target reasoning request to the first reasoning service through the first scheduling service.

[0074] That is to say, in one example, after the first scheduling service receives the first task request, it can first determine the target reasoning resources required for the reasoning request based on the input information and output requirements of the reasoning task carried by the reasoning request in the preset request queue (e.g., Redis request queue) of the target database (e.g., Redis database); secondly, the first scheduling service, based on the model identifier carried by the first task request and the current idle reasoning resources of the first reasoning service, selects from the Redis request queue an reasoning request (i.e., a target reasoning request) that matches the model identifier in the first task request and whose target reasoning resources are not greater than the current idle reasoning resources of the first reasoning service; finally, the first scheduling service sends the selected target reasoning request to the first reasoning service so that the reasoning task carried in the target reasoning request is executed through the first reasoning service.

[0075] In this way, the disclosed scheme can utilize the characteristic information of the inference task carried by the inference request in the preset request queue to determine the target inference resources required for the inference task, so that the first scheduling service can select the inference request (that is, the target inference request) whose model identifier and target inference resource meet the requirements from multiple inference requests according to the model identifier in the first task request and the current idle inference resources of the first inference service, and send it to the first inference service through the first scheduling service. In this way, the first scheduling service provides the first inference service with inference tasks that meet its requirements, thereby effectively realizing the reasonable allocation of resources, avoiding resource idleness due to load imbalance, and thus improving the utilization rate of computing resources.

[0076] In a specific example of the disclosed solution, a preset request queue may also be constructed in the following manner; specifically, before determining a target reasoning request matching the first task request, for example, before step S102 described above, or before step S302, the method further includes:

[0077] A target inference request is obtained through a second scheduling service among the multiple scheduling services, and the target inference request is stored in the preset request queue through the second scheduling service.

[0078] Here, in an example, the second scheduling service is different from the first scheduling service.

[0079] That is, in this example, the scheduling service for storing the target reasoning request and the scheduling service for the reasoning service assigned to the target reasoning request may be different scheduling services.

[0080] In addition, it should be noted that, in actual applications, the second scheduling service and the first scheduling service may also be the same scheduling service.

[0081] For example, in one example, the target object distributes the target reasoning request to the second scheduling service among the multiple scheduling services through a gateway (for example, the gateway is provided with a distribution strategy); after receiving the target reasoning request, the second scheduling service at least writes the request ID (unique identifier, which can be recorded as user_id) carried by the target reasoning request to the Redis request queue of the Redis database for temporary storage. Alternatively, in order to facilitate subsequent matching, the second scheduling service may also write the request ID and the model identifier of the model required for the reasoning task carried by the target reasoning request into the Redis request queue. Alternatively, in another example, in order to achieve subsequent rapid matching success, the second scheduling service may also write the request ID, the model identifier of the model required for the reasoning task carried by the target reasoning request, and the target reasoning resources required for the target reasoning request into the Redis request queue.

[0082] In this way, the disclosed solution can timely store the acquired target inference request into the preset request queue via the second scheduling service. In this way, the inference request is managed with the help of the queue mechanism, laying the foundation for subsequent more efficient resource allocation and service load balancing.

[0083] In a specific example of the present disclosure, after the first inference service receives the target inference request sent by the first scheduling service and executes the corresponding inference task to obtain the corresponding inference result, the method further includes: obtaining at least part of the inference result for the target inference request through the second scheduling service; further, sending at least part of the inference result for the target inference request through the second scheduling service.

[0084] Here, at least part of the inference results are at least part of the results generated after the first inference service executes the target inference request.

[0085] That is to say, in this example, the scheduling service that stores the target reasoning request, that is, the second scheduling service, will also be responsible for feeding back the reasoning result of the target reasoning request. For example, it is responsible for feeding back the reasoning result of the target reasoning request to the target object that initiates the target reasoning request, thereby ensuring a quick response to the reasoning request of the target object.

[0086] In this way, the disclosed solution can utilize different scheduling services to achieve different scheduling purposes, thereby improving response efficiency while ensuring improved resource utilization and system stability, thereby effectively improving user experience.

[0087] Further, in a specific example, the above-described obtaining at least part of the inference results for the target reasoning request through the second scheduling service specifically includes: querying the preset result queue through the second scheduling service whether there are at least part of the inference results corresponding to the target reasoning request; further, if it is determined that there are, obtaining at least part of the inference results corresponding to the target reasoning request from the preset result queue.

[0088] In this example, the preset result queue stores the inference results returned by at least some of the multiple inference services.

[0089] For example, in one example, after the first inference service executes the target inference request and generates at least part of the inference result of the target inference request, the first inference service sends at least part of the inference result of the target inference request to the first scheduling service through the gateway; accordingly, the first scheduling service stores at least part of the received inference result of the target inference request in a preset result queue (for example, the Redis result queue of the Redis database), so that the second scheduling service can obtain the inference result from the Redis result queue.

[0090] It is understandable that the preset result queue not only stores at least part of the reasoning results of the target reasoning request, but also can store the reasoning results of other reasoning requests. Furthermore, in actual applications, the preset result queue stores the request ID and its corresponding reasoning result; at this time, the second scheduling service can query whether there is a reasoning result corresponding to the request ID of the target reasoning request from the preset result queue according to the request ID of the target reasoning request. If so, the second scheduling service directly obtains the reasoning result of the target reasoning request and sends the reasoning result of the target reasoning request through the gateway, for example, to the target object that initiated the target reasoning request.

[0091] In this way, the disclosed solution can utilize different scheduling services to achieve asynchronous processing and rapid response, thereby effectively improving the utilization of system resources and ensuring the stability of the system while improving response efficiency, thereby significantly improving the user experience.

[0092] The disclosed solution provides an inference system, such as Figure 4 As shown, including:

[0093] The first scheduling service 401 is used to receive a first task request, wherein the first task request carries at least a model identifier and a current idle inference resource of the first inference service; based on the model identifier and the idle inference resource carried by the first task request, determine a target inference request that matches the first task request, and send the target inference request;

[0094] The first reasoning service 402 is used to send a first task request; and is also used to obtain a target reasoning request that matches the first task request; and respond to the target reasoning request to perform generative reasoning to obtain at least part of the reasoning result of the target reasoning request.

[0095] In this way, the disclosed solution can obtain the current load situation of the first reasoning service (such as idle reasoning resources) based on the first task request actively sent by the first reasoning service, and then enable the scheduling service to determine the target reasoning request that matches the load situation, and send the target reasoning request to the first reasoning service for reasoning. Compared with the traditional polling strategy, the disclosed solution fully combines the characteristics of the large model reasoning service, effectively realizes the reasonable scheduling of resources, avoids idle resources due to load imbalance, and at the same time, ensures that the reasoning requests are processed in a timely manner, thereby improving resource utilization and ensuring system stability.

[0096] Furthermore, since the scheduling service and reasoning service in the disclosed solution can be deployed in a clustered manner, whether a failure occurs in the scheduling service or the reasoning service, it will not affect the normal function of the overall system, so that the system has a certain fault tolerance and high availability, thereby further ensuring the reliability and stability of the system operation.

[0097] In a specific example of the disclosed solution, the reasoning system further includes:

[0098] A second scheduling service is used to obtain a target reasoning request and store the target reasoning request in a preset request queue;

[0099] A database is used to store preset request queues.

[0100] In this way, the disclosed solution can timely store the acquired target inference request into the preset request queue via the second scheduling service. In this way, the inference request is managed with the help of the queue mechanism, laying the foundation for subsequent more efficient resource allocation and service load balancing.

[0101] In a specific example of the disclosed solution, the second scheduling service is further used to obtain at least the reasoning result corresponding to the target reasoning request, and send at least part of the reasoning result for the target reasoning request. In this way, different scheduling services can be used to achieve different scheduling purposes, thereby ensuring the utilization of resources and the stability of the system while improving the response efficiency, thereby effectively improving the user experience.

[0102] It should be noted that for the relevant contents of the first scheduling service, the second scheduling service, and the first reasoning service, please refer to the above description and will not be repeated here.

[0103] The following is a detailed description of the disclosed solution with specific examples; the disclosed solution proposes a system for load balancing for large model service deployment clusters, which combines the characteristics of large model reasoning services (such as graphics card usage characteristics), and realizes load optimization through the collaboration of load balancing components (i.e., the scheduling service mentioned above) and reasoning server clusters. The core idea is to enable the load balancing component to actively perceive the load status of the backend reasoning service, such as the current remaining hardware resources (such as the size of the graphics card), the remaining empty slots that can be inferred, etc., so as to achieve end-to-end collaborative optimization, better improve the utilization of overall system resources and ensure stability.

[0104] Furthermore, in one example, the load balancing component distributes the user request to the matching reasoning service for processing by judging the relationship between the resources required by the user request (i.e., corresponding to the above reasoning request) and the current idle reasoning resources of the reasoning service. Compared with the traditional polling strategy, the disclosed solution can perceive the load of the backend reasoning service, and logically, the user request is pushed without perception, and the reasoning service actively "pulls" the user request from the load balancing component. In this way, the user request can always be executed on the appropriate reasoning service, effectively avoiding idle resources caused by load imbalance.

[0105] Furthermore, before introducing the implementation steps of the disclosed solution, some terms used in this example are explained:

[0106] (1) Gateway: Traffic entry; specifically, a unified entry for accessing service-oriented clusters, acting as a reverse proxy.

[0107] (2) Scheduling server cluster: used for load balancing processing, which can also be understood as the load balancing component of this example, mainly including multiple scheduling services, for example, which can be recorded as scheduling service 1, scheduling service 2, ..., scheduling service n.

[0108] (3) Remote dictionary server (Redis): used to store user requests and inference results, mainly used for data transfer.

[0109] (4) Large model inference server cluster: contains multiple inference services (for example, can be recorded as inference service 1, inference service 2, ..., inference service n), mainly used to perform specific inference tasks.

[0110] It should be noted that before executing the core steps of the disclosed solution, the following preparatory work needs to be done:

[0111] Specifically, Figure 5 As shown, the core steps of the disclosed solution include:

[0112] Step S500: Preliminary work: Set a unique identifier (which can be recorded as model_id) of the model in the inference server cluster, so that when the subsequent inference service pulls user requests, only requests with the unique identifier model_id are pulled.

[0113] Step S501: Scheduling service to respond to user request.

[0114] Step S5011: the target object sends the target user request to the gateway, so that the target user request is distributed to the scheduling service of the scheduling server cluster through the gateway, for example, to the scheduling service 2 (corresponding to the second scheduling service described above).

[0115] Step S5012: the scheduling service 2 writes the received target user request into the Redis request queue (corresponding to the above preset request queue) of the Redis database for temporary storage.

[0116] Here, the target user request may specifically refer to requesting the target model to perform the specified reasoning task. At this time, the target user request may specifically carry the unique identifier of the requested target model and the required reasoning task.

[0117] For example, in one example, the scheduling service 2 may write the request id (eg, may be recorded as user_id) of the target user request and the unique identifier model_id of the target model required for the target user request into the Redis request queue for temporary storage.

[0118] Step S502: The scheduling service responds to the inference request of the inference service.

[0119] Step S5021: The inference service 1 of the inference server cluster (corresponding to the first inference service described above) sends a task pull request (used to obtain user requests, corresponding to the first task request described above) to the gateway, so as to send the task pull request to the scheduling service of the scheduling server cluster through the gateway, for example, to the scheduling service 1 (corresponding to the first scheduling service described above).

[0120] Here, the task pull request may specifically carry a unique identifier of the model used by the reasoning service (also referred to as a model identifier, which may be recorded as model_id) and the current idle reasoning resources of the reasoning service 1.

[0121] Step S5022: Scheduling service 1 determines the user request that matches the task pull request from the Redis request queue based on the model identifier and idle inference resources carried by the task pull request, such as determining the target user request that matches the task pull request, and pops the target user request out of the queue, and then sends it to inference service 1 through the gateway.

[0122] For example, in one example, the scheduling service 1 can determine the target inference resources required for the user requests contained in the Redis request queue, and select from the user requests contained in the Redis request queue the target user requests that match the model identifier in the task pull request and whose required target inference resources are no greater than the idle inference resources.

[0123] For example, in one example, the target reasoning resource required for the user request is obtained based on feature information (such as input information and required output requirements) of the reasoning task carried by the user request.

[0124] Step S5023: After receiving the target user request, the inference service 1 performs generative reasoning to obtain at least part of the inference result of the target user request, and sends at least part of the inference result of the target user request to the scheduling service 1 through the gateway, so that at least part of the inference result of the target user request is written into the Redis result queue for temporary storage through the scheduling service 1.

[0125] Step S5024: Scheduling service 2 queries the Redis result queue whether there is at least part of the inference result corresponding to the target user request. Further, if it is determined that there is, at least part of the inference result corresponding to the target user request is obtained from the Redis result queue (for example, at least part of the inference result corresponding to the target user request is popped out of the queue), and returned to the target object in streaming or whole sentence mode.

[0126] In this way, the disclosed solution can fully combine the characteristics of large model reasoning for graphics card usage, actively perceive the load and hardware capacity of a single model reasoning service at the back end, thereby achieving end-to-end collaborative optimization and better improving the overall system resource utilization and system stability.

[0127] In summary, the system architecture of the reasoning system used in the disclosed solution has the following advantages:

[0128] First, the inference service actively pulls user requests from the scheduling service, and when pulling user requests, it will actively report the load status of the current inference service (such as idle inference resources, etc.). At this time, the scheduling service, as a load balancing component, can perceive the load status of the backend inference service, and use this as a basis to select user requests that match the inference service, thereby achieving real and effective load balancing for the inference server cluster.

[0129] Second, the scheduling service can also adopt a clustered deployment solution. When one or more scheduling services fail, the inference service can make a judgment based on the communication delay between the current inference service and the scheduling service, and then reconnect to the normal scheduling service;

[0130] Third, the inference service can also adopt a clustered deployment solution. When one or more inference services fail and cannot receive user requests, it will not affect the normal function of the overall system, thereby ensuring the reliability and stability of the system.

[0131] The disclosed solution provides an inference device, such as Figure 6 As shown, including:

[0132] An acquisition unit 601 is configured to receive a first task request through a first scheduling service among multiple scheduling services, wherein the first task request is a request sent by a first inference service among multiple inference services and carries at least a model identifier and current idle inference resources of the first inference service;

[0133] The scheduling unit 602 is used to determine a target reasoning request matching the first task request based on the model identifier and the idle reasoning resources carried by the first task request;

[0134] The sending unit 603 is used to send the target reasoning request to the first reasoning service through the first scheduling service.

[0135] In a specific example of the disclosed solution, the scheduling unit is specifically used to:

[0136] Selecting, through a first scheduling service, from a preset request queue of a target database, a target inference request that matches the model identifier in the first task request and for which the idle inference resource meets the inference requirement;

[0137] The preset request queue stores a plurality of inference requests, and each inference request carries at least an inference task and a model identifier of a model required to be used in the inference task.

[0138] In a specific example of the disclosed solution, the scheduling unit is specifically used to:

[0139] Determine the target reasoning resource required for the reasoning task based on the characteristic information of the reasoning task carried by the reasoning request;

[0140] A target inference request that matches the model identifier in the first task request and whose target inference resource is not greater than the idle inference resource is selected from the multiple inference requests included in the preset request queue.

[0141] In a specific example of the disclosed solution, the acquisition unit is further configured to acquire the target inference request through a second scheduling service among the multiple scheduling services; wherein the second scheduling service is different from the first scheduling service;

[0142] The scheduling unit is further configured to store the target inference request in the preset request queue through the second scheduling service.

[0143] In a specific example of the disclosed solution, the acquisition unit is further used to acquire at least part of the inference results for the target inference request through the second scheduling service; wherein at least part of the inference results are generated after the first inference service executes the target inference request;

[0144] The sending unit is further configured to send at least part of the inference result for the target inference request through the second scheduling service.

[0145] In a specific example of the disclosed solution, the acquisition unit is specifically used to:

[0146] Through the second scheduling service, querying a preset result queue whether there are at least some inference results corresponding to the target inference request; wherein the preset result queue stores inference results returned by at least some of the multiple inference services;

[0147] If it is determined that there is, at least part of the reasoning results corresponding to the target reasoning request is obtained from the preset result queue.

[0148] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0149] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0150] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0151] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0152] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0153] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0154] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as reasoning methods. For example, in some embodiments, the reasoning method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the reasoning method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the reasoning method in any other appropriate manner (e.g., by means of firmware).

[0155] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0159] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0160] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0162] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A reasoning method, the method comprising: Receiving a first task request through a first scheduling service among the multiple scheduling services, wherein the first task request is a request sent by a first inference service among the multiple inference services and carries at least a model identifier and current idle inference resources of the first inference service; Determine, based on the model identifier and the idle reasoning resources carried by the first task request, a target reasoning request matching the first task request; The target inference request is sent to a first inference service through the first scheduling service.

2. The method according to claim 1, wherein: The determining, based on the model identifier and the idle reasoning resources carried by the first task request, a target reasoning request matching the first task request includes: Selecting, through a first scheduling service, from a preset request queue of a target database, a target inference request that matches the model identifier in the first task request and for which the idle inference resource meets the inference requirement; The preset request queue stores a plurality of inference requests, and each inference request carries at least an inference task and a model identifier of a model required to be used in the inference task.

3. The method according to claim 2, wherein: The selecting, from a preset request queue of the target database, a target reasoning request that matches the model identifier in the first task request and for which the idle reasoning resources meet the reasoning requirements comprises: Determine the target reasoning resource required for the reasoning task based on the characteristic information of the reasoning task carried by the reasoning request; A target inference request that matches the model identifier in the first task request and whose target inference resource is not greater than the idle inference resource is selected from the multiple inference requests included in the preset request queue.

4. The method according to claim 2 or 3, further comprising: Obtaining a target inference request through a second scheduling service among the multiple scheduling services; wherein the second scheduling service is different from the first scheduling service; The target inference request is stored in the preset request queue through the second scheduling service.

5. The method according to claim 4, further comprising: Obtaining at least part of the inference results for the target inference request through the second scheduling service; wherein at least part of the inference results are generated after the first inference service executes the target inference request; At least part of the inference result for the target inference request is sent through the second scheduling service.

6. The method according to claim 5, wherein: The obtaining, through the second scheduling service, at least part of the inference result for the target inference request includes: Through the second scheduling service, querying a preset result queue whether there are at least some inference results corresponding to the target inference request; wherein the preset result queue stores inference results returned by at least some of the multiple inference services; If it is determined that there is, at least part of the reasoning results corresponding to the target reasoning request is obtained from the preset result queue.

7. An inference system comprising: A first scheduling service is used to receive a first task request, wherein the first task request carries at least a model identifier and a current idle inference resource of the first inference service; based on the model identifier and the idle inference resource carried by the first task request, determine a target inference request that matches the first task request, and send the target inference request; The first reasoning service is used to send a first task request; it is also used to obtain a target reasoning request that matches the first task request; respond to the target reasoning request to perform generative reasoning to obtain at least part of the reasoning result of the target reasoning request.

8. The reasoning system according to claim 7, further comprising: A second scheduling service is used to obtain a target reasoning request and store the target reasoning request in a preset request queue; A database is used to store preset request queues.

9. The inference system according to claim 8, wherein: The second scheduling service is further used to obtain at least part of the reasoning results corresponding to the target reasoning request, and send at least part of the reasoning results for the target reasoning request.

10. An inference device, comprising: An acquisition unit, configured to receive a first task request through a first scheduling service among multiple scheduling services, wherein the first task request is a request sent by a first inference service among multiple inference services and carries at least a model identifier and currently idle inference resources of the first inference service; A scheduling unit, configured to determine a target reasoning request matching the first task request based on a model identifier and idle reasoning resources carried by the first task request; A sending unit is used to send the target reasoning request to the first reasoning service through the first scheduling service.

11. The device according to claim 10, wherein: The scheduling unit is specifically used for: Selecting, through a first scheduling service, from a preset request queue of a target database, a target inference request that matches the model identifier in the first task request and for which the idle inference resource meets the inference requirement; The preset request queue stores a plurality of inference requests, and each inference request carries at least an inference task and a model identifier of a model required to be used in the inference task.

12. The device according to claim 11, wherein The scheduling unit is specifically used for: Determine the target reasoning resource required for the reasoning task based on the characteristic information of the reasoning task carried by the reasoning request; A target inference request that matches the model identifier in the first task request and whose target inference resource is not greater than the idle inference resource is selected from the multiple inference requests included in the preset request queue.

13. The device according to claim 11 or 12, wherein: The acquisition unit is further configured to acquire the target inference request through a second scheduling service among the multiple scheduling services; wherein the second scheduling service is different from the first scheduling service; The scheduling unit is further configured to store the target inference request in the preset request queue through the second scheduling service.

14. The device according to claim 13, wherein: The acquisition unit is further configured to acquire at least part of the inference results for the target inference request through the second scheduling service; wherein at least part of the inference results are generated after the first inference service executes the target inference request; The sending unit is further configured to send at least part of the inference result for the target inference request through the second scheduling service.

15. The device according to claim 14, wherein: The acquisition unit is specifically used for: Through the second scheduling service, querying a preset result queue whether there are at least some inference results corresponding to the target inference request; wherein the preset result queue stores inference results returned by at least some of the multiple inference services; If it is determined that there is, at least part of the reasoning results corresponding to the target reasoning request is obtained from the preset result queue.

16. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

18. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images