Distributed reasoning operation method and device, computer device, and storage medium

Abstract

The application relates to the field of load allocation, and discloses a distributed reasoning operation method and device, computer equipment and a storage medium. The method comprises the following steps: receiving a service request forwarded by a service server; obtaining reasoning model configuration information generated by a main allocation node in an application coordination service component; the reasoning model configuration information is generated based on pre-configuration information and service state information of each reasoning server; determining a reasoning server corresponding to the service request according to the reasoning model configuration information; sending the service request to the reasoning server, and receiving a reasoning result for responding to the service request from the reasoning server; and sending the reasoning result to the service server. The reasoning server is determined through real-time obtained reasoning model configuration information, so that the distributed computing resources can be better allocated, and the flexibility of increasing or decreasing the computing resources is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of load allocation, in particular to a distributed inference operation method and device, computer equipment and storage medium. BACKGROUND

[0002] With the diffusion of deep learning algorithms, more and more application services need more computing resources to participate in the inference operation of deep learning algorithms. Due to the performance bottleneck, a single chip cannot meet the demand of large-scale inference operation. It is necessary to process by multiple computer devices to meet the demand of large-scale inference operation.

[0003] However, in order to realize the cooperative processing of multiple computer devices, it is often necessary to deeply optimize the overall service architecture (software level) to adapt to the hardware resources. Such deep optimization is not timely and lacks flexibility. SUMMARY

[0004] Therefore, it is necessary to provide a distributed inference operation method and device, computer equipment and storage medium to better allocate dispersed computing resources and improve the flexibility of increasing and decreasing computing resources.

[0005] A distributed inference operation method comprises:

[0006] receiving a service request forwarded by a service server;

[0007] obtaining inference model configuration information generated by a master allocation node in an application coordination service component; the inference model configuration information is generated based on pre-configuration information and service state information of each inference server;

[0008] determining an inference server corresponding to the service request according to the inference model configuration information;

[0009] sending the service request to the inference server and receiving an inference result for responding to the service request from the inference server;

[0010] sending the inference result to the service server.

[0011] A distributed inference operation device comprises:

[0012] a service request receiving module configured to receive a service request forwarded by a service server;

[0013] a model configuration information obtaining module configured to obtain inference model configuration information generated by a master allocation node in an application coordination service component; the inference model configuration information is generated based on pre-configuration information and service state information of each inference server;

[0014] determining an inference server corresponding to the service request according to the inference model configuration information;

[0015] obtaining an inference result module, configured to send the service request to the inference server, and receive an inference result for responding to the service request from the inference server;

[0016] sending an inference result module, configured to send the inference result to the service server.

[0017] A computer device, comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer readable instructions to implement the distributed inference operation method.

[0018] One or more readable storage media storing computer readable instructions, wherein the computer readable instructions are executed by one or more processors to cause the one or more processors to execute the distributed inference operation method.

[0019] In the distributed inference operation method, apparatus, computer device, and storage medium, the service request forwarded by the service server is received, and the application coordination service component is connected with the service server, so that the inference server is allocated to the service request. The inference model configuration information generated by the main allocation node in the application coordination service component is obtained, and the inference model configuration information is generated based on the pre-configuration information and the service state information of each inference server, so as to determine the inference server according to the inference model configuration information. The inference server corresponding to the service request is determined according to the inference model configuration information, so as to select the optimal inference server according to the weight information of each service model, and reduce the inference time. The service request is sent to the inference server, and the inference result for responding to the service request is received from the inference server, so as to complete the inference process and generate the inference result. The inference result is sent to the service server, so that the service request party can receive the inference result from the service server through the client. The inference server is determined by the inference model configuration information obtained in real time, so that the distributed computing resources can be better allocated, and the flexibility of increasing and decreasing the computing resources can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0021] Figure 1is an application environment schematic diagram of a distributed reasoning operation method in an embodiment of the present application;

[0022] Figure 2 is a flow schematic diagram of a distributed reasoning operation method in an embodiment of the present application;

[0023] Figure 3 is a structure schematic diagram of a distributed reasoning operation device in an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0026] The distributed reasoning operation method provided in the embodiment can be applied in an application environment as shown in Figure 1 , wherein the client and the server communicate. The client includes but is not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0027] In an embodiment, as shown in Figure 2 , a distributed reasoning operation method is provided. Taking the server in Figure 1 as an example, the method includes the following steps S10-S50.

[0028] S10, receiving a service request forwarded by a service server.

[0029] Understandably, the server can be configured with an application coordination service component (ZooKeeper), which can receive the service request forwarded by the service server. Here, the user sends the service request to the service server through the client, and then the service server forwards the service request to the application coordination service component.

[0030] S20, obtaining reasoning model configuration information generated by a master coordination node in the application coordination service component; the reasoning model configuration information is generated based on pre-configuration information and service state information of each reasoning server.

[0031] Understandably, after receiving a business request, the application coordination service component can retrieve inference model configuration information from the master node. Inference model configuration information is data generated by the master node based on pre-configuration information and the service status information of each inference server. Here, inference model configuration information includes, but is not limited to, pre-configuration information and the service status information of each inference server. Pre-configuration information may refer to the address information and machine weight of each inference server. The address information may refer to the IP address and / or network port of the inference server. Machine weight reflects the inference capability of the inference server. The higher the inference capability of the inference server, the higher its machine weight. Service status information reflects the current service status of the inference server and the model weight of the service model on the inference server. On the same inference server, if a service model has a higher model weight, that service model has a higher priority.

[0032] S30. Determine the inference server corresponding to the business request based on the inference model configuration information.

[0033] Understandably, each business request can be decomposed into one or more inference tasks, and each inference task corresponds to a service model. Inference servers can be assigned to each service model based on the inference model configuration information. A service model can be assigned one or more inference servers, preferably those with higher weight values. In some examples, the total weight of the inference servers is the product of the machine weight and the model weight.

[0034] S40. Send the business request to the inference server and receive the inference result from the inference server in response to the business request.

[0035] S50. Send the inference result to the business server.

[0036] Understandably, after determining the inference server, a business request can be pushed to the inference server, enabling the inference server to execute the inference tasks involved in the business request. After completing the inference task, the inference server can generate the inference result and return it to the server, which then sends the inference result to the business server, which in turn forwards the inference result to the requester of the business request.

[0037] In steps S10-S50, a business request forwarded by the business server is received. Here, the application coordination service component connects with the business server and can allocate a suitable inference server for the business request. Inference model configuration information generated by the main dispatch node in the application coordination service component is obtained. This inference model configuration information is generated based on pre-configuration information and the service status information of each inference server, to determine the inference server. The inference server corresponding to the business request is determined according to the inference model configuration information, so as to select the optimal inference server based on the weight information of each service model, reducing inference time. The business request is sent to the inference server, and the inference result used to respond to the business request is received from the inference server to complete the inference process and generate the inference result. The inference result is sent to the business server so that the business requester can receive the inference result from the business server through a client. This embodiment determines the inference server by acquiring the inference model configuration information in real time, which can better allocate distributed computing resources and improve the flexibility of adding or removing computing resources.

[0038] Optionally, before step S20, i.e. before obtaining the inference model configuration information generated by the master dispatch node in the application coordination service component, the method further includes:

[0039] S21. Monitor the working status of the main dispatch node from the dispatch node;

[0040] S22. If the working status of the master dispatch node is abnormal, the slave dispatch node shall take over the master dispatch node.

[0041] Understandably, an architecture consisting of a master node and slave nodes can be used to manage and configure the inference server. The slave node has hot standby functionality, allowing it to monitor the master node's operational status in real time. If the master node malfunctions, the slave node takes over, managing and configuring the inference server to ensure uninterrupted operation.

[0042] This embodiment sets up a slave scheduling node that can take over the master scheduling node, so that when the master scheduling node fails, the slave scheduling node can quickly take over, ensuring that the management and configuration functions of the inference server are not interrupted.

[0043] Optionally, before step S20, i.e. before obtaining the inference model configuration information generated by the master dispatch node in the application coordination service component, the method further includes:

[0044] S23. The main dispatch node obtains the pre-configuration information from the configuration center component. The pre-configuration information includes the address information and machine weight of each inference server.

[0045] S24. The main dispatch node periodically obtains the service status information of each inference server according to the address information; the service status information includes the service model running on the inference server and the model weight corresponding to the service model;

[0046] S25. The main dispatch node generates the inference model configuration information based on the pre-configuration information and the service status information of each inference server.

[0047] Understandably, the server-side configuration center component is provided. Administrators can configure pre-configuration information through this component. This pre-configuration information includes the address information and machine weights of each inference server. The address information includes the IP address and / or network port of the inference server. The machine weight reflects the inference capability of the inference server. The higher the inference capability of the inference server, the higher its machine weight. In some cases, the configuration center component can also automatically obtain the hardware information of the inference server and automatically generate machine weights for the inference server based on this hardware information.

[0048] When a new inference server comes online, the configuration center component can assign it address information and update the pre-configuration information. Similarly, when an inference server goes offline, the configuration center component marks the address information of that inference server as unavailable.

[0049] The master dispatch node periodically retrieves service status information from each inference server based on its address information. The time interval for retrieval can be configured according to actual needs, such as every 3 hours or 12 hours. The service status information includes the service models running on the inference servers and the corresponding model weights. Each service model corresponds to a model weight.

[0050] After obtaining the pre-configuration information and the service status information of each inference server, the master dispatch node can generate inference model configuration information based on the pre-configuration information and the service status information of each inference server.

[0051] In one example, the inference model configuration information can be represented as:

[0052] Inference Server 1: IP1 / port1; Machine Weight 1; Model 1: Model Weight 11; Model 2: Model Weight 12; Model 3: Model Weight 13; ...;

[0053] Inference Server 2: IP2 / port2; Machine Weight 2; Model 1: Model Weight 21; Model 3: Model Weight 23; Model 4: Model Weight 24; ...; ...

[0054] In some cases, the machine weights from the pre-configured information can also be written into the service status information. In this case, the master dispatch node can directly generate inference model configuration information based on the service status information of each inference server.

[0055] This embodiment monitors the service status information of each inference server through the master dispatch node, which can update the inference model configuration information in a timely manner and reduce the manual configuration process.

[0056] Optionally, step S30, namely determining the inference server corresponding to the service request based on the inference model configuration information, includes:

[0057] S301. Parse the inference model configuration information to generate an inverted list; obtain the service model and model selection rules associated with the business request;

[0058] S302. Obtain machine weight information associated with the service model from the inverted list;

[0059] S303. Process the machine weight information according to the model selection rules to determine the inference server corresponding to the business request.

[0060] Understandably, inference model configuration information can be parsed to generate an inverted index. The inverted index contains the weights of each inference model on each inference server. The service model and model selection rules associated with a business request can be obtained. There can be one or more service models associated with a business request. Each service model has corresponding model selection rules. These rules can be set according to the actual needs of the service model. Different service models can have the same or different model selection rules.

[0061] After obtaining the inverted index, machine weight information associated with the service model (the service model used by the business request) can be retrieved from the inverted index. Then, the machine weight information is processed according to the model selection rules to determine the inference server selected for the service model, which is also the inference server corresponding to the business request. If the business request involves only one service model, the inference server corresponding to the business request is the inference server selected for that service model; if the business request involves more than one service model, the inference server corresponding to the business request is the union of the inference servers selected for all involved service models.

[0062] This embodiment sets model selection rules for each service model, which can select the best combination of inference servers for the service model.

[0063] Optionally, step S301, namely parsing the inference model configuration information and generating an inverted list, includes:

[0064] S3011. Parse the address information, machine weight, and service status information of each inference server from the inference model configuration information; the service status information includes the service model running on the inference server and the model weight corresponding to the service model.

[0065] S3012. Generate the weight information of each service model on each inference server based on the address information, the machine weight, and the service status information, and form the inverted list.

[0066] Understandably, the address information, machine weight, service model, and corresponding model weight of each inference server are parsed from the inference model configuration information. Then, the weight information of each service model on each inference server is generated based on the address information, machine weight, service model, and model weight, forming an inverted list.

[0067] In one example, an inverted list can be represented as:

[0068] Model 1=IP1:port:weight; IP2:port:weight;…;IPn:port:weight;

[0069] Model 2=IP1:port:weight; IP2:port:weight;…;IPn:port:weight;

[0070] ...

[0071] Here, each IP address represents an inference server, and n is the total number of inference servers. Service models have different weights on different inference servers. If a service model is not present on a particular inference server, its weight is 0.

[0072] This embodiment can be used to generate an inverted index, which can quickly find the weight information of a certain service model on each inference server.

[0073] Optionally, step S303, namely, processing the machine weight information according to the model selection rules to determine the inference server corresponding to the business request, includes:

[0074] S3031. Sort the inference servers associated with the service model according to the machine weight information;

[0075] S3032. Determine the specified number of inference servers that are ranked first as the inference server corresponding to the business request.

[0076] Understandably, inference servers associated with a service model can be sorted based on machine weight information, such as sorting by weight value from largest to smallest. Then, a specified number of inference servers ranked first are selected as the inference servers corresponding to the business request. This specified number can be set according to actual needs. Different service models may have different specified numbers. Furthermore, the number of inference servers used for the same service model may differ depending on the service status of the inference servers.

[0077] In one example, there are two service models associated with the business request: Service Model 1 and Service Model 2. Service Model 1 has a specified number of occurrences of 3, and Service Model 2 has a specified number of occurrences of 4. The inference server corresponding to the business request can then be represented as:

[0078] Service Model 1: Inference Server 11, Inference Server 12, Inference Server 13;

[0079] Service Model 2: Inference Server 21, Inference Server 22, Inference Server 23, Inference Server 24.

[0080] It should be noted that if a certain inference server can run both service model 1 and service model 2 at the same time, then the inference server may be selected by both service model 1 and service model 2.

[0081] This embodiment can quickly select the server corresponding to the business request.

[0082] Optionally, after step S20, i.e. after obtaining the inference model configuration information generated by the master dispatch node in the application coordination service component, the method further includes:

[0083] S201. Monitor the service status of each inference server through the main dispatch node and generate service status change information;

[0084] S202. If the service status change information is not empty, the main dispatch node updates the inference model configuration information according to the service status change information.

[0085] Understandably, the master dispatch node can monitor the service status of each inference server and generate service status change information. If the service status of any inference server changes, the service status change information will be non-empty. If the service status of all inference servers remains unchanged, the service status change information will be empty. If the service status change information is non-empty, the master dispatch node updates the inference model configuration information based on the service status change information.

[0086] In this embodiment, the master dispatch node monitors the service status of each inference server. When the inference server is horizontally scaled up or taken offline, the master dispatch node can quickly detect it and automatically update the inference model configuration information without manual configuration, which greatly improves configuration efficiency.

[0087] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0088] In one embodiment, a distributed inference computing device is provided, which corresponds one-to-one with the distributed inference computing methods described in the above embodiments. For example... Figure 3 As shown, the distributed inference computing device includes a service request receiving module 10, a model configuration information acquisition module 20, an inference server determination module 30, an inference result acquisition module 40, and an inference result sending module 50. Detailed descriptions of each functional module are as follows:

[0089] The service request receiving module 10 is used to receive service requests forwarded by the service server.

[0090] The model configuration information acquisition module 20 is used to acquire inference model configuration information generated by the main dispatch node in the application coordination service component; the inference model configuration information is generated based on pre-configuration information and service status information of each inference server;

[0091] The inference server determination module 30 is used to determine the inference server corresponding to the business request based on the inference model configuration information.

[0092] The inference result acquisition module 40 is used to send the business request to the inference server and receive the inference result from the inference server in response to the business request;

[0093] The inference result sending module 50 is used to send the inference result to the business server.

[0094] Optionally, the distributed inference computing device further includes:

[0095] The master node status monitoring module is used to monitor the working status of the master dispatch node by means of the dispatch node;

[0096] The takeover module is used to take over the main dispatch node from the slave dispatch node if the working status of the main dispatch node is abnormal.

[0097] Optionally, the distributed inference computing device further includes an inference model configuration information generation module, which includes:

[0098] A pre-configuration information acquisition unit is used for the main dispatch node to acquire the pre-configuration information from the configuration center component. The pre-configuration information includes the address information and machine weight of each inference server.

[0099] The service status information acquisition unit is used by the main dispatch node to periodically acquire the service status information of each inference server based on the address information; the service status information includes the service model running on the inference server and the model weights corresponding to the service model;

[0100] A unit for generating inference model configuration information is used by the main dispatch node to generate the inference model configuration information based on the pre-configuration information and the service status information of each inference server.

[0101] Optionally, the inference server module 30 includes:

[0102] A unit for generating an inverted index is used to parse the inference model configuration information and generate an inverted index; and to obtain the service model and model selection rules associated with the business request.

[0103] The weight information acquisition unit is used to acquire machine weight information associated with the service model from the inverted list;

[0104] The inference server unit is used to process the machine weight information according to the model selection rules and determine the inference server corresponding to the business request.

[0105] Optionally, generating the inverted index cell includes:

[0106] The parsing information unit is used to parse the address information, machine weight, and service status information of each inference server from the inference model configuration information; the service status information includes the service model running on the inference server and the model weight corresponding to the service model.

[0107] An inverted index unit is formed to generate weight information of each service model on each inference server based on the address information, the machine weight, and the service status information, thus forming the inverted index.

[0108] Optionally, determining the inference server unit includes:

[0109] A sorting unit is used to sort the inference servers associated with the service model according to the machine weight information;

[0110] The inference server unit is used to determine a specified number of inference servers that are sorted first as the inference server corresponding to the business request.

[0111] Optionally, the distributed inference computing device further includes:

[0112] The service status monitoring unit is used to monitor the service status of each inference server through the main dispatch node and generate service status change information.

[0113] The inference model configuration information update unit is used so that if the service status change information is not empty, the main dispatch node updates the inference model configuration information according to the service status change information.

[0114] Specific limitations regarding the distributed inference computing device can be found in the limitations of the distributed inference computing method described above, and will not be repeated here. Each module in the aforementioned distributed inference computing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0115] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data involved in distributed inference computation methods. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a distributed inference computation method is implemented. The readable storage medium provided in this embodiment includes both non-volatile readable storage media and volatile readable storage media.

[0116] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0117] Receive business requests forwarded by the business server;

[0118] Obtain the inference model configuration information generated by the master dispatch node in the application coordination service component; the inference model configuration information is generated based on pre-configuration information and service status information of each inference server;

[0119] The inference server corresponding to the business request is determined based on the inference model configuration information.

[0120] The service request is sent to the inference server, and the inference result is received from the inference server in response to the service request.

[0121] The inference result is sent to the business server.

[0122] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0123] Receive business requests forwarded by the business server;

[0124] Obtain the inference model configuration information generated by the master dispatch node in the application coordination service component; the inference model configuration information is generated based on pre-configuration information and service status information of each inference server;

[0125] The inference server corresponding to the business request is determined based on the inference model configuration information.

[0126] The service request is sent to the inference server, and the inference result is received from the inference server in response to the service request.

[0127] The inference result is sent to the business server.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0130] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A distributed reasoning computation method, characterized in that, include: Receive business requests forwarded by the business server; Retrieve the inference model configuration information generated by the master dispatch node in the application coordination service component; The inference model configuration information is generated based on pre-configuration information and the service status information of each inference server; The inference server corresponding to the business request is determined based on the inference model configuration information. The service request is sent to the inference server, and the inference result is received from the inference server in response to the service request. The inference result is sent to the business server; Prior to obtaining the inference model configuration information generated by the master dispatch node in the application coordination service component, the method further includes: The main dispatch node obtains the pre-configuration information from the configuration center component, and the pre-configuration information includes the address information and machine weight of each inference server; The master dispatch node periodically obtains the service status information of each inference server based on the address information; the service status information includes the service model running on the inference server and the model weights corresponding to the service model. The main dispatch node generates the inference model configuration information based on the pre-configuration information and the service status information of each inference server. The step of determining the inference server corresponding to the service request based on the inference model configuration information includes: Parse the inference model configuration information to generate an inverted list; obtain the service model and model selection rules associated with the business request; Obtain machine weight information associated with the service model from the inverted list; The machine weight information is processed according to the model selection rules to determine the inference server corresponding to the business request.

2. The distributed reasoning computation method as described in claim 1, characterized in that, Before obtaining the inference model configuration information generated by the master dispatch node in the application coordination service component, the process also includes: The working status of the main dispatch node is monitored from the dispatch node; If the working status of the master dispatch node is abnormal, the slave dispatch node will take over the master dispatch node.

3. The distributed reasoning computation method as described in claim 1, characterized in that, The process of parsing the inference model configuration information and generating an inverted list includes: The address information, machine weight, and service status information of each inference server are parsed from the inference model configuration information; the service status information includes the service model running on the inference server and the model weight corresponding to the service model. The weight information of each service model on each inference server is generated based on the address information, the machine weight, and the service status information, forming the inverted list.

4. The distributed reasoning computation method as described in claim 1, characterized in that, The step of processing the machine weight information according to the model selection rules to determine the inference server corresponding to the business request includes: The inference servers associated with the service model are sorted according to the machine weight information; The specified number of inference servers that are ranked first are determined as the inference servers corresponding to the business request.

5. The distributed reasoning computation method as described in claim 1, characterized in that, After obtaining the inference model configuration information generated by the master dispatch node in the application coordination service component, the process further includes: The main dispatch node monitors the service status of each inference server and generates service status change information. If the service status change information is not empty, the main dispatch node updates the inference model configuration information according to the service status change information.

6. A distributed inference computing device, characterized in that, include: The business request receiving module is used to receive business requests forwarded by the business server. The module for obtaining model configuration information is used to obtain inference model configuration information generated by the master dispatch node in the application coordination service component; the inference model configuration information is generated based on pre-configuration information and service status information of each inference server; The inference server determination module is used to determine the inference server corresponding to the business request based on the inference model configuration information. The inference result acquisition module is used to send the business request to the inference server and receive the inference result from the inference server in response to the business request; A module for sending inference results is used to send the inference results to the business server; The distributed inference computing device further includes an inference model configuration information generation module, which comprises: A pre-configuration information acquisition unit is used for the main dispatch node to acquire the pre-configuration information from the configuration center component. The pre-configuration information includes the address information and machine weight of each inference server. The service status information acquisition unit is used by the main dispatch node to periodically acquire the service status information of each inference server based on the address information; the service status information includes the service model running on the inference server and the model weights corresponding to the service model; A unit for generating inference model configuration information is used by the main dispatch node to generate the inference model configuration information based on the pre-configuration information and the service status information of each inference server. The determination inference server module includes: A unit for generating an inverted index is used to parse the inference model configuration information and generate an inverted index; and to obtain the service model and model selection rules associated with the business request. The weight information acquisition unit is used to acquire machine weight information associated with the service model from the inverted list; The inference server unit is used to process the machine weight information according to the model selection rules and determine the inference server corresponding to the business request.

7. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the distributed inference operation method as described in any one of claims 1 to 5.

8. One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the distributed inference operation method as described in any one of claims 1 to 5.

Images