Model inference scheduling method, device, system, medium, product and equipment

Abstract

The application discloses a model inference scheduling method, device, system, medium, product and equipment. The method comprises the following steps: receiving a first scheduling request message from an inference request end, wherein the first scheduling request message is used for indicating at least one region and respective inference request quantities of the at least one region; determining a first target service end from a to-be-scheduled service end based on the first scheduling request message; and returning a first scheduling response message to the inference request end, wherein the first scheduling response message carries first information of the first target service end, so that the scheduling end can select a suitable first target service end for the inference request end to complete an inference task according to the demand of the inference request end for different regions and the inference request quantities of the different regions, and the inference request end can also complete regional batch scheduling of the inference task for one or more regions by sending the first scheduling request message only once, so as to reduce scheduling delay in the regional batch scheduling scenario.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a scheduling method, apparatus, system, medium, product and equipment for model reasoning. Background Technology

[0002] In existing technologies, current model inference deployment schemes typically centralize AI (Artificial Intelligence) requests onto a single inference server cluster, which then acts as the inference executor for scheduling and computation. This aligns with the characteristics of centralized and cloud-based computing resources in AI inference scenarios within the internet industry.

[0003] However, as industries such as communication networks begin to undergo intelligent transformation and upgrading, a new scenario for AI inference deployment has emerged, in which the inference requester and the inference executor are distributed in various regions. However, some AI inference requesters have certain requirements for the location of the inference executor (for example, requiring the inference executor to be deployed nearby relative to the requester). Existing scheduling schemes are difficult to meet the deployment needs of these new AI inference scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application propose a scheduling method, apparatus, system, medium, product, and equipment for model reasoning.

[0005] This application provides a scheduling method for model inference, applied to a scheduling terminal, the method comprising:

[0006] Receive a first scheduling request message from the inference request end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region.

[0007] Based on the first scheduling request message, the first target server is determined from the servers to be scheduled;

[0008] A first scheduling response message is returned to the inference requesting end, wherein the first scheduling response message carries the first information of the first target server.

[0009] Furthermore, determining the first target server from the servers to be scheduled based on the first scheduling request message includes:

[0010] From the servers to be scheduled, determine the first batch of scheduling servers corresponding to each of the at least one region;

[0011] Based on the number of inference requests, a second batch of scheduling servers corresponding to each region is selected from the first batch of scheduling servers corresponding to each region.

[0012] All second batch scheduling servers are identified as the first target server.

[0013] Furthermore, the first scheduling request message carries at least one combination of regional parameters associated with each of the at least one region. Each combination of regional parameters includes first parameter information of the associated region and the number of inference requests. The information of the servers to be scheduled is stored in a pre-configured KV database. Determining the first batch of scheduling servers corresponding to each of the at least one region from the servers to be scheduled includes:

[0014] Construct an array corresponding to each combination of regional parameters;

[0015] For each combination of regional parameters, a query is performed in the KV database based on the array corresponding to the combination of regional parameters to determine all the servers to be scheduled that meet the requirements indicated by the combination of regional parameters, which are then used as the first batch of scheduled servers corresponding to the region associated with the combination of regional parameters.

[0016] Furthermore, the first scheduling response message is used to instruct the inference request terminal to determine whether the first information meets the requirements indicated by the first scheduling request message, and to send back a first confirmation scheduling message when it is determined that the requirements are met.

[0017] Furthermore, the method also includes:

[0018] After identifying the first target server, set the status of the first target server to "temporarily unavailable".

[0019] In response to the first confirmation scheduling message, the status of the first target server is changed to the scheduled status, and a first scheduling success response message is sent to the inference request client. The first scheduling success response message is used to instruct the inference request client to initiate a first inference request to the first target server corresponding to the first scheduling request message. The first inference request carries the inference task that needs to be undertaken by the first target server.

[0020] Furthermore, the method also includes:

[0021] Receive a second scheduling request message from the inference request terminal, wherein the second scheduling request message carries second parameter information of the inference request terminal;

[0022] Based on the second parameter information, a second target server is determined from the servers to be scheduled;

[0023] The second scheduling response message is returned to the inference requesting end. The second scheduling response message carries the second information of the second target server. The second scheduling response message is used to instruct the inference requesting end to determine whether the second information meets the requirements indicated by the second scheduling request message, and to send back a second confirmation scheduling message when it is determined that the requirements are met.

[0024] Furthermore, the second parameter information includes: the geographical location of the inference request terminal, or the geographical location of the inference request terminal, and the device type and / or data center information;

[0025] The step of determining the second target server from the servers to be scheduled based on the second parameter information includes:

[0026] Based on the geographical location, the corresponding first affinity scheduling server is found from the servers to be scheduled;

[0027] If the number of the first affinity scheduling servers is greater than 1, a load balancing algorithm is used to filter the first affinity scheduling servers to obtain the second affinity scheduling server as the second target server.

[0028] If the number of the first affinity scheduling servers is equal to 1, the first affinity scheduling server is used as the second target server.

[0029] Furthermore, the method also includes:

[0030] After identifying the second target server, set the status of the second target server to temporarily unavailable;

[0031] In response to the second confirmation scheduling message, the status of the second target server is changed to the scheduled status, and a second scheduling success response message is sent to the inference request client. The second scheduling success response message is used to instruct the inference request client to initiate a second inference request to the second target server corresponding to the second scheduling request message. The second inference request carries the inference task that needs to be undertaken by the second target server.

[0032] Furthermore, the first scheduling response message is also used to instruct the inference requesting end to cache the address information of the first target server locally after determining that the first information meets the requirements indicated by the first scheduling request message, so as to use it for the subsequent first inference task corresponding to the first scheduling request message.

[0033] The second scheduling response message is also used to instruct the inference requesting end to cache the address information of the second target server locally after determining that the second information meets the requirements indicated by the second scheduling request message, so as to use it for the subsequent second inference task corresponding to the second scheduling request message.

[0034] Furthermore, the method also includes:

[0035] Receive a subscription message sent by the inference request terminal, wherein the subscription message is used to subscribe to the status of at least one server, and the at least one server is at least one of the first target server and / or the second target server;

[0036] In response to the subscription message, a corresponding load threshold is configured for each of the at least one server.

[0037] If it is detected that the load reported by any of the at least one server exceeds the corresponding load threshold, a load status change message of the any server is fed back to the inference request client, wherein the load status change message is used to instruct the inference request client to delete the address information of the any server cached locally.

[0038] Furthermore, in cases where at least a portion of the first inference task needs to be undertaken by any of the servers, the load status change message is also used to instruct the inference requesting end to stop the first inference task that needs to be undertaken by any of the servers when the address information of any of the servers is deleted, and to convert the first inference task that needs to be undertaken by any of the servers into corresponding scheduling request information to be added to the next sent first scheduling request message.

[0039] And / or,

[0040] In cases where at least a portion of the second inference task needs to be undertaken by either of the servers, the load status change message is further used to instruct the inference requesting end to stop the second inference task that needs to be undertaken by either of the servers when the address information of either of the servers is deleted, and to convert the second inference task that needs to be undertaken by either of the servers into corresponding scheduling request information to be added to the next second scheduling request message sent.

[0041] This application also provides a model inference scheduling method, applied to the inference request end, the method comprising:

[0042] Send a first scheduling request message to the scheduling end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region;

[0043] The system receives a first scheduling response message returned by the scheduling terminal, wherein the first scheduling response message carries first information of a first target server, and the first target server is determined by the scheduling terminal from the server to be scheduled based on the first scheduling request message.

[0044] This application also provides a model inference scheduling device, applied to a scheduling terminal, the device comprising:

[0045] The first scheduling request message receiving module is used to receive a first scheduling request message from the inference request end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region.

[0046] The first target server determination module is used to determine the first target server from the servers to be scheduled based on the first scheduling request message;

[0047] The first scheduling response message sending module is used to return a first scheduling response message to the inference request end, wherein the first scheduling response message carries the first information of the first target server.

[0048] This application also provides a model inference scheduling device, applied to an inference request end, the device comprising:

[0049] The first scheduling request message sending module is used to send a first scheduling request message to the scheduling end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region.

[0050] The first scheduling response message receiving module is used to receive the first scheduling response message returned by the scheduling end, wherein the first scheduling response message carries the first information of the first target server, and the first target server is determined by the scheduling end from the server to be scheduled based on the first scheduling request message.

[0051] This application also provides a scheduling system for model inference, including:

[0052] An inference request terminal, wherein the inference request terminal is configured to execute the method described above applied to the inference request terminal, or the inference request terminal is equipped with the means described above applied to the inference request terminal; and,

[0053] The scheduling terminal is used to execute any of the methods described above applied to the scheduling terminal, or the scheduling terminal is equipped with the devices described above applied to the scheduling terminal.

[0054] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0055] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0056] This application also provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0057] In summary, the embodiments of this application have at least the following beneficial effects:

[0058] In this embodiment, by receiving a first scheduling request message from the inference request end, wherein the first scheduling request message indicates at least one region and the number of inference requests for each region; based on the first scheduling request message, a first target server is determined from the servers to be scheduled; and a first scheduling response message is returned to the inference request end, wherein the first scheduling response message carries first information of the first target server, thereby enabling the scheduling end to select a suitable first target server for the inference request end to complete the inference task according to the inference request end's requirements regarding different regions and the number of inference requests. In addition, it is also possible to enable the inference request end to complete the regional batch scheduling of inference tasks for one or more regions by sending only the first scheduling request message once, thereby significantly reducing the scheduling latency in regional batch scheduling scenarios. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the scheduling method for model inference provided in an embodiment of this application;

[0060] Figure 2 This is a schematic diagram of the registration of the scheduling server provided in an embodiment of this application;

[0061] Figure 3 This is a schematic diagram of information interaction for scheduling model inference provided in an embodiment of this application;

[0062] Figure 4 This is a schematic diagram illustrating the filtering of the second batch of scheduling servers provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram of information interaction for scheduling model inference provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of information interaction for scheduling model inference provided in an embodiment of this application;

[0065] Figure 7 This is a flowchart illustrating the scheduling method for model inference provided in an embodiment of this application;

[0066] Figure 8 This is a schematic diagram of the structure of the scheduling device for model inference provided in the embodiments of this application;

[0067] Figure 9 This is a schematic diagram of the structure of the scheduling device for model inference provided in the embodiments of this application;

[0068] Figure 10 This is a schematic diagram of the scheduling system for model inference provided in the embodiments of this application;

[0069] Figure 11 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0071] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments."

[0072] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0073] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0074] The following explains some terms and concepts used in the embodiments of this application:

[0075] Key-value (KV) databases are a type of NoSQL database system that stores and retrieves data in the form of simple key-value pairs. In a KV database, each data item consists of a unique key and an associated value, similar to a dictionary or mapping (such as a Map in Java) data structure in programming languages.

[0076] See Figure 1 The diagram illustrates a flowchart of a model inference scheduling method provided in an embodiment of this application. This method is applied to a scheduling terminal and includes steps S101-S103, as detailed below:

[0077] S101, the scheduling terminal receives a first scheduling request message from the inference request terminal, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region.

[0078] Optionally, when the scheduler needs to execute a batch of inference tasks, it generates and sends a first scheduling request message based on the inference tasks. The first scheduling request message indicates at least one region, which may refer to the region where the inference task executor (i.e., the server) required by the inference task is located. Furthermore, the inference task may also require the inference task executor in each region to provide computing power sufficient to meet the number of inference requests corresponding to that region. Therefore, in this embodiment, the first scheduling request message is equivalent to integrating a batch scheduling request for one or more regions.

[0079] S102, the scheduling terminal determines the first target server from the servers to be scheduled based on the first scheduling request message;

[0080] Optionally, the scheduler can, based on the first scheduling request message, select a first target server from the pre-registered servers to be scheduled that meets the requirements of at least one region and the number of inference requests for each region, to ensure that the selected first target server can provide computing power to each region to meet the corresponding number of inference requests. The first scheduling request message may include: location information of at least one region, and inference request information for at least one region (at least indicating the number of inference requests).

[0081] In addition, it should be noted that, see Figure 2 A service instance to be scheduled is a service instance used for inference services. When a service instance to be scheduled is pre-registered with the scheduler, it can register via a registration message. That is, the scheduler receives a registration message from the service instance to be scheduled and, upon successful registration, returns a registration success message to the service instance to be scheduled (e.g., a "201 response" to indicate successful registration). In response to the registration message, the scheduler marks the status of the service instance to be scheduled as available, so that the service instance marked as available can be discovered by any inference request client. The registration message can include at least one of the following:

[0082] The co-location device type of the server to be scheduled can be used to represent the device type of the hardware entity deployed on the server to be scheduled. This co-location device type can be configured as follows: when the value is 0, it means that the server to be scheduled is set up separately in the hardware and software all-in-one mode and is not co-located with other types of devices; when the value is not 0, it takes the standard category name of the co-located device, such as: compute server, UPF (User Plane Function), CU (Centralized Unit), etc.

[0083] The geographical location of the server to be dispatched can include three types of geographical location fields: first, a location system based on administrative divisions, including fields such as country, province, city, and street; second, directly encoding the device location according to rules, with only one field: location code; and third, information such as the device's latitude and longitude, and TAC (Tracking Area Code).

[0084] The data center information of the server to be scheduled can be used to indicate the data center information where the hardware of the server to be scheduled is located. This data center information can be configured as follows: when the value is 0, it means that the hardware is placed outdoors; when the value is not 0, the corresponding fields include the data center name and the location number of the hardware in the data center.

[0085] Accordingly, the first scheduling request message may carry at least one of the following:

[0086] Device type of the inference requester: This indicates the device type to which the inference requester belongs. The value is the standard category name of the device to which it belongs, such as: Compute Server, UPF, CU, etc.

[0087] The geographic location of the inference request end can include three types of geographic location fields: first, a location system based on administrative divisions, including fields such as country, province, city, and street; second, directly encoding the device location according to rules, with only one field: location code; and third, information such as the device's latitude and longitude, and TAC.

[0088] The data center information of the inference request client can be used to indicate the data center information where the hardware of the inference request client is located. This data center information can be configured as follows: when the value is 0, it means that the hardware is placed outdoors; when the value is not 0, the corresponding fields include the data center name and the location number of the hardware in the data center.

[0089] S103, the scheduling terminal returns a first scheduling response message to the inference request terminal, wherein the first scheduling response message carries the first information of the first target server.

[0090] Optionally, the first information may carry information such as the service instance name, service instance number, service instance address, priority, load and / or capacity of the first target server.

[0091] The following, combined with Figure 3 The various embodiments of this application will be further explained below.

[0092] In an optional embodiment, determining the first target server from the servers to be scheduled based on the first scheduling request message includes:

[0093] From the servers to be scheduled, determine the first batch of scheduling servers corresponding to each of the at least one region;

[0094] Based on the number of inference requests, a second batch of scheduling servers corresponding to each region is selected from the first batch of scheduling servers corresponding to each region.

[0095] All second batch scheduling servers are identified as the first target server.

[0096] In one example, the first batch of scheduling servers can be any of the servers to be scheduled that are at a distance from the corresponding region that meets a preset distance requirement. This preset distance requirement may include a distance less than a preset distance threshold. It should be understood that the first batch of scheduling servers corresponding to different regions may be partially the same or completely different. For example, the first batch of scheduling server A corresponding to region A and the first batch of scheduling server B corresponding to region B may have one or more identical servers to be scheduled. That is, both the first batch of scheduling servers A and B contain one or more of these identical servers to be scheduled. Of course, it is not difficult to understand that the first batch of scheduling servers corresponding to any two regions (such as the first batch of scheduling servers A and B) may also be completely different.

[0097] In one example, based on the number of inference requests, selecting a second batch scheduling server for each region from the first batch scheduling servers for each region can include: using a load balancing algorithm to filter the first batch scheduling servers for each region to select the second batch scheduling server for each region. Therefore, this embodiment can optimize resource usage and improve system response speed and availability among all the first batch scheduling servers that meet the requirements of each region through a load balancing algorithm.

[0098] In one example, the second batch scheduling server corresponding to each selected region may include: all first batch scheduling servers in the first batch scheduling server corresponding to that region whose computing power is sufficient to meet the computing power required for the number of inference requests in that region.

[0099] In an optional embodiment, the first scheduling request message carries at least one combination of regional parameters associated with each of the at least one region. Each combination of regional parameters includes first parameter information of the associated region and the number of inference requests. The information of the servers to be scheduled is stored in a pre-configured key-value database. Determining the first batch of scheduling servers corresponding to each of the at least one region from the servers to be scheduled includes:

[0100] Construct an array corresponding to each combination of regional parameters;

[0101] For each combination of regional parameters, a query is performed in the KV database based on the array corresponding to the combination of regional parameters to determine all the servers to be scheduled that meet the requirements indicated by the combination of regional parameters, which are then used as the first batch of scheduled servers corresponding to the region associated with the combination of regional parameters.

[0102] In one example, the requirement indicated by the combination of regional parameters is determined by the first parameter information included in the combination and the number of inference requests. In this case, meeting the requirement may mean that the distance between the geographical location of the scheduling server and the associated region meets a preset distance requirement, and the maximum computing power it can provide is higher than the computing power required for the number of inference requests corresponding to the associated region. Here, the goal is to find all scheduling servers that can simultaneously meet the requirements of that region and its number of inference requests for each region.

[0103] At this point, further, the second batch scheduling server corresponding to each region selected can refer to: among all the first batch scheduling servers that meet the requirements indicated by the region parameter combination associated with that region, the second batch scheduling server is selected according to a preset rule, wherein the preset rule includes that the minimum total computing power provided by the selected second batch scheduling server is not less than the computing power required by the number of inference requests corresponding to that region. For ease of understanding, combined with Figure 4 To further explain this embodiment, firstly, for each combination of regional parameters, such as regional parameter combination m, the first batch of scheduled servers (i.e., the instances selected in the first stage) corresponding to the region associated with regional parameter combination m are determined. Then, based on the number of inference requests in the region associated with regional parameter combination m, a second stage of screening is performed on the instances selected in the first stage to select servers with a minimum total computing power not less than the computing power required for the number of inference requests corresponding to that region, which are then used as the second batch of scheduled servers corresponding to that region. Here, since the servers to be scheduled selected in the first stage in this embodiment only initially screen out all servers that simultaneously meet the requirements of the region and the number of inference requests, while the computing power corresponding to the number of inference requests in that region is not actually that high, this embodiment can perform a second stage of screening to avoid wasting computing power.

[0104] Optionally, after filtering out the second batch of scheduling servers corresponding to each region, the elements corresponding to the filtered second batch of scheduling servers can be renumbered in the array, and / or the filtered second batch of scheduling servers can be stored in the array to store the information of the filtered second batch of scheduling servers locally.

[0105] For example, the scheduler can generate information related to the server to be scheduled based on the registration message and store it locally (e.g., in a KV database) in a key-value manner, as shown in Table 1.

[0106]

[0107]

[0108] Table 1

[0109] Understandably, some existing technologies analyze the server's location solely based on its IP (Internet Protocol) address. However, this method is only suitable for scenarios with standardized address naming, and in many current scenarios, it's difficult to obtain accurate location information from IP addresses. Furthermore, when a device is in motion, its IP address may be assigned from an address pool by an SMF (Session Management Function) or P-GW (Packet Gateway). Some intranet devices are only configured with private IP addresses, making it difficult to determine the server's location based solely on its IP address. Accordingly, this embodiment, by constructing Table 1 above, can solve the problem of insufficient accuracy in obtaining the server's location based solely on its IP address in some existing technologies.

[0110] For example, the information carried in the first scheduling request message can be constructed in a key-value manner, as shown in Table 2.

[0111] key value Requested service name Assign values ​​according to the situation. The number of instances requested for scheduling (i.e., the number of inference requests). Assign values ​​according to the situation. The requested scheduling request includes a combination of geographic location parameters A (country / province / city / street, number of instances). Assign values ​​according to the situation. The requested scheduling requires a combination of geographic location parameters, B (location code list, number of instances). Assign values ​​according to the situation. The requested scheduling location parameter combination C (latitude and longitude range, number of instances) Assign values ​​according to the situation. The request schedules the geographical location parameter combination D (TAC list, number of instances). Assign values ​​according to the situation. The requested scheduling request includes the geographical location parameter combination E (list of data center names, number of instances). Assign values ​​according to the situation.

[0112] Table 2

[0113] For example, the first parameter information for each region may include the geographic location, device type, and / or data center information of the inference request terminal corresponding to that region.

[0114] In an optional embodiment, the first scheduling response message is used to instruct the inference request terminal to determine whether the first information meets the requirements indicated by the first scheduling request message, and to send back a first confirmation scheduling message when it is determined that the requirements are met.

[0115] It should be noted that, in this embodiment, the process of determining whether the first information meets the requirements indicated by the first scheduling request message, performed by the inference request end, can specifically use any of the embodiments in this disclosure where the scheduling end determines the first target server based on the first scheduling request message. This allows the inference request end to determine whether the first information (the indicated first target server) meets the requirements according to the first scheduling request message, which will not be elaborated further here. This embodiment enables secondary verification at the inference request end to ensure that the determined first target server meets the requirements indicated by the first scheduling request message, thereby improving scheduling accuracy.

[0116] Optionally, if it is determined that the first information does not meet the requirements indicated by the first scheduling request message, the inference request end may also send a first scheduling result verification message to the scheduling end, so that the scheduling end can determine a new first target server based on the first scheduling request message and the first scheduling result verification message.

[0117] In an optional embodiment, the method further includes:

[0118] After identifying the first target server, set the status of the first target server to "temporarily unavailable".

[0119] In response to the first confirmation scheduling message, the status of the first target server is changed to the scheduled status, and a first scheduling success response message is sent to the inference request client. The first scheduling success response message is used to instruct the inference request client to initiate a first inference request to the first target server corresponding to the first scheduling request message. The first inference request carries the inference task that needs to be undertaken by the first target server.

[0120] In one example, the first scheduling success response message may include a 200 OK scheduling success response message for the first target server.

[0121] In an optional embodiment, see Figure 5 The method further includes:

[0122] Receive a second scheduling request message from the inference request terminal, wherein the second scheduling request message carries second parameter information of the inference request terminal;

[0123] Based on the second parameter information, a second target server is determined from the servers to be scheduled;

[0124] The second scheduling response message is returned to the inference requesting end. The second scheduling response message carries the second information of the second target server. The second scheduling response message is used to instruct the inference requesting end to determine whether the second information meets the requirements indicated by the second scheduling request message, and to send back a second confirmation scheduling message when it is determined that the requirements are met.

[0125] Optionally, the second information may carry information such as the service instance name, service instance number, service instance address, priority, load and / or capacity of the second target server.

[0126] For example, the information carried in the second scheduling request message (e.g., the second parameter information) can be constructed in a key-value manner, as shown in Table 3.

[0127] key value Requested service name Assign values ​​according to the situation. Requester (i.e., the inference requester) device type Assign values ​​according to the situation. Requester's geographical location / country Assign values ​​according to the situation. Requester's geographical location / province Assign values ​​according to the situation. Requester's geographical location / city Assign values ​​according to the situation. Requester's geographical location / street Assign values ​​according to the situation. Requester's geolocation / location code Assign values ​​according to the situation. Requester's geographical location / latitude and longitude Assign values ​​according to the situation. Requester's geographic location / TAC Assign values ​​according to the situation. Requester's data center information / data center name Assign values ​​according to the situation. Requester's data center information / location number of hardware within the data center Assign values ​​according to the situation.

[0128] Table 3

[0129] In an optional embodiment, the second parameter information includes: the geographical location of the inference request terminal, or the geographical location of the inference request terminal, and the device type and / or data center information;

[0130] The step of determining the second target server from the servers to be scheduled based on the second parameter information includes:

[0131] Based on the geographical location, the corresponding first affinity scheduling server is found from the servers to be scheduled;

[0132] If the number of the first affinity scheduling servers is greater than 1, a load balancing algorithm is used to filter the first affinity scheduling servers to obtain the second affinity scheduling server as the second target server.

[0133] If the number of the first affinity scheduling servers is equal to 1, the first affinity scheduling server is used as the second target server.

[0134] In this embodiment, the scheduler can perform local affinity scheduling through a second scheduling request message to select a second target server from the servers to be scheduled, thereby improving the efficiency of task execution in the distributed system, especially in scenarios involving data locality and reducing network latency.

[0135] In an optional embodiment, the method further includes:

[0136] After identifying the second target server, set the status of the second target server to temporarily unavailable;

[0137] In response to the second confirmation scheduling message, the status of the second target server is changed to the scheduled status, and a second scheduling success response message is sent to the inference request client. The second scheduling success response message is used to instruct the inference request client to initiate a second inference request to the second target server corresponding to the second scheduling request message. The second inference request carries the inference task that needs to be undertaken by the second target server.

[0138] In one example, the second scheduling success response message contains a 200 OK scheduling success response message for the second target server.

[0139] The following combination Figure 6 The embodiments of this application will be further explained and illustrated below.

[0140] In an optional embodiment, the first scheduling response message is further used to instruct the inference requesting end to cache the address information of the first target server locally after determining that the first information meets the requirements indicated by the first scheduling request message, so as to use it for the subsequent first inference task corresponding to the first scheduling request message.

[0141] The second scheduling response message is also used to instruct the inference requesting end to cache the address information of the second target server locally after determining that the second information meets the requirements indicated by the second scheduling request message, so as to use it for the subsequent second inference task corresponding to the second scheduling request message.

[0142] It should be noted that, in this embodiment, caching the address information locally for subsequent inference tasks means that when the inference requesting end needs to initiate an inference task corresponding to the first scheduling request message and / or the second scheduling request message again, it can directly initiate the first inference request and / or the second inference request to the corresponding target server based on the address information cached locally, thereby instructing the first target server and / or the second target server that received the first inference request and / or the second inference request to undertake the corresponding inference task.

[0143] In this embodiment, after completing a local affinity scheduling (i.e., determining the second target server in response to the second scheduling request message) and / or regional batch scheduling (i.e., determining the second target server in response to the first scheduling request message), the inference request client will, according to its business logic, choose to cache the address information of the first target server and / or the second target server locally, so that when the same inference request is initiated again (i.e., the inference request initiated to the corresponding target server corresponding to the corresponding scheduling request message is equivalent to the subsequent inference task corresponding to the corresponding scheduling request message), it can directly initiate the inference request to the corresponding target server (i.e., use the cached address information locally for the subsequent inference task corresponding to the corresponding scheduling request message), so as to avoid the latency caused by the scheduling process or the creation of the server.

[0144] In an optional embodiment, the method further includes:

[0145] Receive a subscription message sent by the inference request terminal, wherein the subscription message is used to subscribe to the status of at least one server, and the at least one server is at least one of the first target server and / or the second target server;

[0146] In response to the subscription message, a corresponding load threshold is configured for each of the at least one server.

[0147] If it is detected that the load reported by any of the at least one server exceeds the corresponding load threshold, a load status change message of the any server is fed back to the inference request client, wherein the load status change message is used to instruct the inference request client to delete the address information of the any server cached locally.

[0148] It should be noted that, in this embodiment, at least one server refers to any first target server and / or any second target server.

[0149] In this embodiment, when the inference requesting end caches the address information of the first target server and / or the second target server locally for subsequent inference tasks, it can further consider the load status of each server and delete the address information of any overloaded first target server and / or second target server. This is to prevent some first target servers and / or some second target servers from being called by related inference tasks after their load reaches the limit (because the initiation of such inference tasks does not require scheduling processing by the scheduler, so it does not consider whether the computing power of the server is occupied by other inference tasks). This avoids the inference task needing to queue in the corresponding server, thus further improving scheduling efficiency.

[0150] In an optional embodiment, if at least a portion of the first inference task needs to be undertaken by any of the servers, the load status change message is further used to instruct the inference requesting end to stop the first inference task that needs to be undertaken by any of the servers when the address information of any of the servers is deleted, and to convert the first inference task that needs to be undertaken by any of the servers into corresponding scheduling request information to be added to the next sent first scheduling request message.

[0151] And / or,

[0152] In cases where at least a portion of the second inference task needs to be undertaken by either of the servers, the load status change message is further used to instruct the inference requesting end to stop the second inference task that needs to be undertaken by either of the servers when the address information of either of the servers is deleted, and to convert the second inference task that needs to be undertaken by either of the servers into corresponding scheduling request information to be added to the next second scheduling request message sent.

[0153] It is understood that, in this embodiment, at least a portion of the first inference task refers to a part or all of the first inference tasks corresponding to the first scheduling request message. At least a portion of the second inference task refers to a part or all of the second inference tasks corresponding to the second scheduling request message.

[0154] In this embodiment, after the inference requester deletes the address information cached locally, a lazy scheduling strategy can be adopted. Instead of immediately sending the corresponding first scheduling request message and / or second scheduling request message to the scheduler based on the first inference task and / or second inference task to be subsequently undertaken by either of the servers, the first scheduling request message and / or second scheduling request message to be subsequently undertaken by either of the servers will be converted into corresponding scheduling request information and added to the first scheduling request message and / or second scheduling request message when the first scheduling request message and / or second scheduling request message need to be sent normally next time. This allows for a one-time rescheduling of the accumulated first inference tasks and / or second inference tasks to be subsequently undertaken by either of the servers, thereby reducing the scheduler's load and avoiding frequent wake-ups of the scheduler.

[0155] It should be noted that, in this embodiment, after the scheduling terminal receives the first scheduling request message and / or the second scheduling request message to be sent next, its processing flow for the received scheduling request message can refer to any embodiment of the first scheduling request message and / or the second scheduling request message in this disclosure, and will not be repeated here.

[0156] See Figure 7 The diagram illustrates a flowchart of a model inference scheduling method provided in an embodiment of this application. This method is applied to the inference request end and includes steps S701-S702, as detailed below:

[0157] S701, a first scheduling request message is sent to the scheduling end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region.

[0158] S702, receive the first scheduling response message returned by the scheduling terminal, wherein the first scheduling response message carries the first information of the first target server, and the first target server is determined by the scheduling terminal from the servers to be scheduled based on the first scheduling request message.

[0159] In an optional embodiment, determining the first target server from the servers to be scheduled includes:

[0160] From the servers to be scheduled, determine the first batch of scheduling servers corresponding to each of the at least one region;

[0161] Based on the number of inference requests, a second batch of scheduling servers corresponding to each region is selected from the first batch of scheduling servers corresponding to each region.

[0162] All second batch scheduling servers are identified as the first target server.

[0163] In an optional embodiment, the first scheduling request message carries at least one combination of regional parameters associated with each of the at least one region. Each combination of regional parameters includes first parameter information of the associated region and the number of inference requests. The information of the servers to be scheduled is stored in a pre-configured key-value database. Determining the first batch of scheduling servers corresponding to each of the at least one region from the servers to be scheduled includes:

[0164] Construct an array corresponding to each combination of regional parameters;

[0165] For each combination of regional parameters, a query is performed in the KV database based on the array corresponding to the combination of regional parameters to determine all the servers to be scheduled that meet the requirements indicated by the combination of regional parameters, which are then used as the first batch of scheduled servers corresponding to the region associated with the combination of regional parameters.

[0166] In an optional embodiment, the method further includes:

[0167] The inference request terminal responds to the first scheduling response message, determines whether the first information meets the requirements indicated by the first scheduling request message, and sends a first confirmation scheduling message back to the scheduling terminal when it determines that the requirements are met.

[0168] In an optional embodiment, the first confirmation scheduling message is further used to instruct the scheduling end to set the status of the first target server to a temporarily unselectable state after determining the first target server, and, in response to the first confirmation scheduling message, to change the status of the first target server to a scheduled state and send a first scheduling success response message to the inference request end.

[0169] The method further includes:

[0170] In response to the first scheduling success response message, the inference requesting end initiates a first inference request corresponding to the first scheduling request message to the first target server. The first inference request carries the inference task that needs to be undertaken by the first target server.

[0171] In an optional embodiment, the method further includes:

[0172] Send a second scheduling request message to the scheduling terminal, wherein the second scheduling request message carries the second parameter information of the inference request terminal;

[0173] The system receives a second scheduling response message returned by the scheduling terminal, wherein the second scheduling response message carries second information of the second target server, and the second target server is determined by the scheduling terminal from the server to be scheduled based on the second parameter information;

[0174] Determine whether the second information meets the requirements indicated by the second scheduling request message, and if it is determined that the requirements are met, send back a second confirmation scheduling message.

[0175] In an optional embodiment, the second parameter information includes: the geographical location of the inference request terminal, or the geographical location of the inference request terminal, and the device type and / or data center information;

[0176] Determining the second target server from the servers to be scheduled based on the second parameter information includes:

[0177] Based on the geographical location, the corresponding first affinity scheduling server is found from the servers to be scheduled;

[0178] If the number of the first affinity scheduling servers is greater than 1, a load balancing algorithm is used to filter the first affinity scheduling servers to obtain the second affinity scheduling server as the second target server.

[0179] If the number of the first affinity scheduling servers is equal to 1, the first affinity scheduling server is used as the second target server.

[0180] In an optional embodiment, the second confirmation scheduling message is further used to instruct the scheduling end to set the status of the second target server to temporarily unselectable after determining the second target server, and to change the status of the second target server to a scheduled status, and send a second scheduling success response message to the inference request end;

[0181] The method further includes:

[0182] In response to the second scheduling success response message, the inference requesting end initiates a second inference request corresponding to the second scheduling request message to the second target server. The second inference request carries the inference task that needs to be undertaken by the second target server.

[0183] In an optional embodiment, the method further includes:

[0184] In response to the first scheduling response message, the inference requesting end caches the address information of the first target server locally after determining that the first information meets the requirements indicated by the first scheduling request message, so as to use it for the subsequent first inference task corresponding to the first scheduling request message.

[0185] The inference requesting end responds to the second scheduling response message. After determining that the second information meets the requirements indicated by the second scheduling request message, it caches the address information of the second target server locally for subsequent second inference tasks corresponding to the second scheduling request message.

[0186] In an optional embodiment, the method further includes:

[0187] The inference requester sends a subscription message to the scheduler, wherein the subscription message is used to subscribe to the status of at least one server, the at least one server being at least one of the first target server and / or the second target server, and the subscription message is also used to instruct the scheduler to configure a corresponding load threshold for each of the at least one server.

[0188] The inference request terminal receives a load status change message from any of the at least one server sent by the scheduling terminal, wherein the load status change message is generated by the scheduling terminal when it detects that the load reported by any of the at least one server exceeds the corresponding load threshold.

[0189] In response to receiving the load status change message, the inference request client deletes the address information of any of the servers cached locally.

[0190] In an optional embodiment,

[0191] In cases where at least a portion of the first inference task needs to be undertaken by any of the servers, the inference requesting end responds to the load state change message by stopping the first inference task that needs to be undertaken by any of the servers when the address information of any of the servers is deleted, and converts the first inference task that needs to be undertaken by any of the servers into corresponding scheduling request information to be added to the next first scheduling request message sent.

[0192] And / or,

[0193] In cases where at least a portion of the second inference task needs to be undertaken by either of the servers, the inference requesting end responds to the load state change message by stopping the second inference task that needs to be undertaken by either of the servers when the address information of either of the servers is deleted, and converts the second inference task that needs to be undertaken by either of the servers into corresponding scheduling request information to be added to the next second scheduling request message sent.

[0194] Accordingly, this application also provides a model inference scheduling device that can implement all processes of any model inference scheduling method applied to the scheduling end provided in the above embodiments.

[0195] See Figure 8 The diagram illustrates the structure of a model inference scheduling device provided in an embodiment of this application. This device is applied to a scheduling terminal and includes:

[0196] The first scheduling request message receiving module 801 is used to receive a first scheduling request message from the inference request end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region.

[0197] The first target server determination module 802 is used to determine the first target server from the servers to be scheduled based on the first scheduling request message;

[0198] The first scheduling response message sending module 803 is used to return a first scheduling response message to the inference request end, wherein the first scheduling response message carries the first information of the first target server.

[0199] Accordingly, this application also provides a model inference scheduling device that can implement all processes of any model inference scheduling method applied to the inference request end provided in the above embodiments.

[0200] See Figure 9 The diagram illustrates the structure of a model inference scheduling device provided in an embodiment of this application. This device is applied to the inference request end and includes:

[0201] The first scheduling request message sending module 901 is used to send a first scheduling request message to the scheduling end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region.

[0202] The first scheduling response message receiving module 902 is used to receive the first scheduling response message returned by the scheduling end, wherein the first scheduling response message carries the first information of the first target server, and the first target server is determined by the scheduling end from the server to be scheduled based on the first scheduling request message.

[0203] Accordingly, embodiments of this application also provide a scheduling system for model inference, see [link to relevant documentation]. Figure 10 This document illustrates a schematic diagram of the scheduling system for model inference provided in an embodiment of this application, including:

[0204] An inference request terminal 1001 is configured to execute the method described above for the inference request terminal, or the inference request terminal is equipped with the aforementioned means for the inference request terminal; and,

[0205] The scheduling terminal 1002 is used to execute any of the methods described above applied to the scheduling terminal, or the scheduling terminal is equipped with the devices described above applied to the scheduling terminal.

[0206] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0207] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0208] This application also provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0209] See Figure 11 The computer device in this embodiment includes a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101, such as a model inference scheduler. When the processor 1101 executes the computer program, it implements the steps in the various model inference scheduling method embodiments described above, for example... Figure 1 Steps S101-S103 shown, or, Figure 7 The steps S701-S702 are shown.

[0210] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 1102 and executed by the processor 1101 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0211] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 1101 and a memory 1102. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0212] The processor 1101 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 1101 can be any conventional processor. The processor 1101 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0213] The memory 1102 can be used to store the computer programs and / or modules. The processor 1101 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 1102 and calling the data stored in the memory 1102. The memory 1102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0214] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 1101, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0215] In summary, the embodiments of this application have at least the following beneficial effects:

[0216] In this embodiment, by receiving a first scheduling request message from the inference request end, wherein the first scheduling request message indicates at least one region and the number of inference requests for each region; based on the first scheduling request message, a first target server is determined from the servers to be scheduled; and a first scheduling response message is returned to the inference request end, wherein the first scheduling response message carries first information of the first target server, thereby enabling the scheduling end to select a suitable first target server for the inference request end to complete the inference task according to the inference request end's requirements regarding different regions and the number of inference requests. In addition, it is also possible to enable the inference request end to complete the regional batch scheduling of inference tasks for one or more regions by sending only the first scheduling request message once, thereby significantly reducing the scheduling latency in regional batch scheduling scenarios.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0218] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A scheduling method for model inference, characterized in that, Applied to the scheduling end, the method includes: Receive a first scheduling request message from the inference request end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region. Based on the first scheduling request message, the first target server is determined from the servers to be scheduled; A first scheduling response message is returned to the inference requesting end, wherein the first scheduling response message carries the first information of the first target server; Receive a second scheduling request message from the inference request terminal, wherein the second scheduling request message carries second parameter information of the inference request terminal; Based on the second parameter information, a second target server is determined from the servers to be scheduled; The second scheduling response message is returned to the inference request end, wherein the second scheduling response message carries the second information of the second target server. The second scheduling response message is used to instruct the inference request end to determine whether the second information meets the requirements indicated by the second scheduling request message, and to feed back a second confirmation scheduling message when it is determined that the requirements are met. The second parameter information includes the geographical location of the inference request terminal. The step of determining the second target server from the servers to be scheduled based on the second parameter information includes: finding a corresponding first affinity scheduling server from the servers to be scheduled based on the geographical location; if the number of first affinity scheduling servers is greater than one, using a load balancing algorithm to filter the first affinity scheduling servers to obtain a second affinity scheduling server as the second target server; if the number of first affinity scheduling servers is equal to one, using the first affinity scheduling server as the second target server.

2. The scheduling method for model inference as described in claim 1, characterized in that, The step of determining the first target server from the servers to be scheduled based on the first scheduling request message includes: From the servers to be scheduled, determine the first batch of scheduling servers corresponding to each of the at least one region; Based on the number of inference requests, a second batch of scheduling servers corresponding to each region is selected from the first batch of scheduling servers corresponding to each region. All second batch scheduling servers are identified as the first target server.

3. The scheduling method for model inference as described in claim 2, characterized in that, The first scheduling request message carries at least one combination of regional parameters associated with each of the at least one region. Each combination of regional parameters includes first parameter information of the associated region and the number of inference requests. The information of the servers to be scheduled is stored in a pre-configured key-value database. Determining the first batch of scheduling servers corresponding to each of the at least one region from the servers to be scheduled includes: Construct an array corresponding to each combination of regional parameters; For each combination of regional parameters, a query is performed in the KV database based on the array corresponding to the combination of regional parameters to determine all the servers to be scheduled that meet the requirements indicated by the combination of regional parameters, which are then used as the first batch of scheduled servers corresponding to the region associated with the combination of regional parameters.

4. The scheduling method for model inference as described in claim 1, characterized in that, The first scheduling response message is used to instruct the inference request terminal to determine whether the first information meets the requirements indicated by the first scheduling request message, and to send back a first confirmation scheduling message when it is determined that the requirements are met.

5. The scheduling method for model inference as described in claim 4, characterized in that, The method further includes: After identifying the first target server, set the status of the first target server to "temporarily unavailable". In response to the first confirmation scheduling message, the status of the first target server is changed to the scheduled status, and a first scheduling success response message is sent to the inference request client. The first scheduling success response message is used to instruct the inference request client to initiate a first inference request to the first target server corresponding to the first scheduling request message. The first inference request carries the inference task that needs to be undertaken by the first target server.

6. The scheduling method for model inference as described in claim 1, characterized in that, The second parameter information also includes: equipment type and / or computer room information.

7. The scheduling method for model inference as described in claim 1, characterized in that, The method further includes: After identifying the second target server, set the status of the second target server to temporarily unavailable; In response to the second confirmation scheduling message, the status of the second target server is changed to the scheduled status, and a second scheduling success response message is sent to the inference request client. The second scheduling success response message is used to instruct the inference request client to initiate a second inference request to the second target server corresponding to the second scheduling request message. The second inference request carries the inference task that needs to be undertaken by the second target server.

8. The scheduling method for model inference as described in claim 1, characterized in that, The first scheduling response message is also used to instruct the inference requesting end to cache the address information of the first target server locally after determining that the first information meets the requirements indicated by the first scheduling request message, so as to use it for the subsequent first inference task corresponding to the first scheduling request message. The second scheduling response message is also used to instruct the inference requesting end to cache the address information of the second target server locally after determining that the second information meets the requirements indicated by the second scheduling request message, so as to use it for the subsequent second inference task corresponding to the second scheduling request message.

9. The scheduling method for model inference as described in claim 8, characterized in that, The method further includes: Receive a subscription message sent by the inference request terminal, wherein the subscription message is used to subscribe to the status of at least one server, and the at least one server is at least one of the first target server and / or the second target server; In response to the subscription message, a corresponding load threshold is configured for each of the at least one server. If it is detected that the load reported by any of the at least one server exceeds the corresponding load threshold, a load status change message of the any server is fed back to the inference request client, wherein the load status change message is used to instruct the inference request client to delete the address information of the any server cached locally.

10. The scheduling method for model inference as described in claim 9, characterized in that, In cases where at least a portion of the first inference task needs to be undertaken by any of the servers, the load status change message is further used to instruct the inference requesting end to stop the first inference task that needs to be undertaken by any of the servers when the address information of any of the servers is deleted, and to convert the first inference task that needs to be undertaken by any of the servers into corresponding scheduling request information to be added to the next first scheduling request message sent. And / or, In cases where at least a portion of the second inference task needs to be undertaken by either of the servers, the load status change message is further used to instruct the inference requesting end to stop the second inference task that needs to be undertaken by either of the servers when the address information of either of the servers is deleted, and to convert the second inference task that needs to be undertaken by either of the servers into corresponding scheduling request information to be added to the next second scheduling request message sent.

11. A scheduling method for model inference, characterized in that, Applied to the inference request end, the method includes: Send a first scheduling request message to the scheduling end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region; The system receives a first scheduling response message returned by the scheduling terminal, wherein the first scheduling response message carries first information of a first target server, and the first target server is determined by the scheduling terminal from the server to be scheduled based on the first scheduling request message; Send a second scheduling request message to the scheduling terminal, wherein the second scheduling request message carries the second parameter information of the inference request terminal; Receive a second scheduling response message returned by the scheduling terminal, wherein the second scheduling response message carries second information of the second target server; Determine whether the second information meets the requirements indicated by the second scheduling request message, and if it is determined that the requirements are met, send back a second confirmation scheduling message; Wherein, the second parameter information includes the geographical location of the inference request end, and the second target server is determined by the scheduling end from the servers to be scheduled based on the second parameter information in the following manner: based on the geographical location, a corresponding first affinity scheduling server is found from the servers to be scheduled; if the number of the first affinity scheduling servers is greater than 1, a load balancing algorithm is used to filter the first affinity scheduling servers to obtain a second affinity scheduling server as the second target server; if the number of the first affinity scheduling servers is equal to 1, the first affinity scheduling server is used as the second target server.

12. A scheduling device for model inference, characterized in that, The device, applied to a scheduling terminal, includes: The first scheduling request message receiving module is used to receive a first scheduling request message from the inference request end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region. The first target server determination module is used to determine the first target server from the servers to be scheduled based on the first scheduling request message; The first scheduling response message sending module is used to return a first scheduling response message to the inference request end, wherein the first scheduling response message carries the first information of the first target server; The device is also used for: Receive a second scheduling request message from the inference request terminal, wherein the second scheduling request message carries second parameter information of the inference request terminal; Based on the second parameter information, a second target server is determined from the servers to be scheduled; The second scheduling response message is returned to the inference request end, wherein the second scheduling response message carries the second information of the second target server. The second scheduling response message is used to instruct the inference request end to determine whether the second information meets the requirements indicated by the second scheduling request message, and to feed back a second confirmation scheduling message when it is determined that the requirements are met. The second parameter information includes the geographical location of the inference request terminal. The step of determining the second target server from the servers to be scheduled based on the second parameter information includes: finding a corresponding first affinity scheduling server from the servers to be scheduled based on the geographical location; if the number of first affinity scheduling servers is greater than one, using a load balancing algorithm to filter the first affinity scheduling servers to obtain a second affinity scheduling server as the second target server; if the number of first affinity scheduling servers is equal to one, using the first affinity scheduling server as the second target server.

13. A scheduling device for model inference, characterized in that, The device, applied to the inference request end, includes: The first scheduling request message sending module is used to send a first scheduling request message to the scheduling end, wherein the first scheduling request message is used to indicate at least one region and the number of inference requests for each region. The first scheduling response message receiving module is used to receive the first scheduling response message returned by the scheduling end, wherein the first scheduling response message carries the first information of the first target server, and the first target server is determined by the scheduling end from the server to be scheduled based on the first scheduling request message; The device is also used for: Send a second scheduling request message to the scheduling terminal, wherein the second scheduling request message carries the second parameter information of the inference request terminal; Receive a second scheduling response message returned by the scheduling terminal, wherein the second scheduling response message carries second information of the second target server; Determine whether the second information meets the requirements indicated by the second scheduling request message, and if it is determined that the requirements are met, send back a second confirmation scheduling message; Wherein, the second parameter information includes the geographical location of the inference request end, and the second target server is determined by the scheduling end from the servers to be scheduled based on the second parameter information in the following manner: based on the geographical location, a corresponding first affinity scheduling server is found from the servers to be scheduled; if the number of the first affinity scheduling servers is greater than 1, a load balancing algorithm is used to filter the first affinity scheduling servers to obtain a second affinity scheduling server as the second target server; if the number of the first affinity scheduling servers is equal to 1, the first affinity scheduling server is used as the second target server.

14. A scheduling system for model inference, characterized in that, include: The inference request terminal is used to execute the method of claim 11, or the inference request terminal is provided with the device of claim 13; as well as, A scheduling terminal, wherein the scheduling terminal is used to execute the method described in any one of claims 1-10, or the scheduling terminal is provided with the apparatus described in claim 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-11.

16. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-11.

17. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-11.

Images