Service scheduling method and device, scheduling server and storage medium

Abstract

The invention belongs to the technical field of service scheduling, and discloses a service scheduling method and device, a scheduling server and a storage medium. The method comprises the following steps: when a target questioning request sent by a client is received, obtaining a current load state of each reasoning node of a reasoning server; determining a target node from the reasoning nodes according to the current load state; and scheduling the target question request to the target node, so that the target node responds to the target question request. Through the method, the request time delay of the user can be effectively shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of service scheduling, and in particular to a service scheduling method, device, scheduling server and storage medium. Background Art

[0002] Since the client cannot identify bad nodes or high-load nodes on the server, if only traditional load balancing algorithms such as Round Robin and Least Connections are used to send user question requests to the server, a large number of user requests will be blocked in the queue, which will lead to an increase in user request latency. Summary of the invention

[0003] The main purpose of the present invention is to provide a service scheduling method, device, scheduling server and storage medium, aiming to solve the technical problem in the prior art that a large number of user requests are blocked in the queue due to the client's inability to determine the bad nodes or high-load nodes in the server, causing the user request delay to increase.

[0004] To achieve the above object, the present invention provides a service scheduling method, which comprises the following steps:

[0005] When receiving a target query request sent by the client, obtain the current load status of each inference node of the inference server;

[0006] Determining a target node from each inference node according to the current load state;

[0007] The target question request is dispatched to the target node, so that the target node responds to the target question request.

[0008] Optionally, the service scheduling method further includes:

[0009] Determining target answer information of the target question request from a target storage area, wherein the target storage area stores answer information output by the inference node for the question request;

[0010] The target answer information is sent to the client.

[0011] Optionally, determining target answer information of the target question request from the target storage area includes:

[0012] Perform sentence splitting on the target question request to obtain target split sentences;

[0013] Determine multiple solution information of the target split sentence from the target storage area;

[0014] The multiple pieces of answer information are integrated to obtain target answer information.

[0015] Optionally, determining target answer information of the target question request from the target storage area includes:

[0016] Perform sentence splitting on the target question request to obtain target split sentences;

[0017] Determine a first split statement and a second split statement in the target split statement;

[0018] Determining first answer information of the first split statement from the target storage area;

[0019] Determine second answer information output by the target node for the second split statement;

[0020] The first answer information and the second answer information are integrated to obtain target answer information.

[0021] Optionally, determining the target node from each inference node according to the current load state includes:

[0022] Determining an initial target node from each inference node according to the current load state;

[0023] After sending the request information to the initial target node, receiving the return information of the initial target node;

[0024] A target node among the initial target nodes is determined according to the return time of the return information.

[0025] Optionally, after dispatching the target question request to the target node, the method further includes:

[0026] generating scheduling information of the target node;

[0027] The scheduling information is forwarded to other scheduling servers.

[0028] Optionally, the service scheduling method further includes:

[0029] When the inference server is in a congested state, determining a low priority question request from multiple target question requests;

[0030] When the low-priority question request is in the first state, discarding the low-priority question request;

[0031] When the low priority question request is in the second state, the low priority question request is inserted into the queuing queue of the target node.

[0032] Optionally, determining a low-priority question request from a plurality of target question requests includes:

[0033] Determining the business conditions of the multiple target question requests according to the identifiers of the target question requests;

[0034] A low priority question request is determined from the multiple question requests according to the business situation.

[0035] Optionally, the service scheduling method further includes:

[0036] Based on the inference service IP list, the abnormal nodes in the inference server are determined through a health check mechanism, and the abnormal nodes are removed from the inference server, wherein the abnormal nodes are inference nodes in an unhealthy state.

[0037] Optionally, the service scheduling method further includes:

[0038] When the online and offline information of the inference node is received, the inference node of the inference server is updated according to the online and offline information.

[0039] Optionally, the service scheduling method further includes:

[0040] The reasoning node in the overload state is regarded as an overload node;

[0041] The queuing queue of the overloaded node is determined, and the question request corresponding to the queuing queue is dispatched to the idle reasoning node of the reasoning server.

[0042] Optionally, the service scheduling method further includes:

[0043] Determine the queue time of the queued question request in the queued state;

[0044] When the queuing time is greater than a preset queuing time, the queuing question request is dispatched to an idle reasoning node of the reasoning server.

[0045] In addition, to achieve the above object, the present invention also proposes a service scheduling device, the service scheduling device comprising:

[0046] The acquisition module is used to obtain the current load status of each inference node of the inference server when receiving the target question request sent by the client;

[0047] A determination module, used to determine a target node from each inference node according to the current load state;

[0048] The scheduling module is used to schedule the target question request to the target node, so that the target node responds to the target question request.

[0049] Optionally, the service scheduling device further includes a sending module; wherein,

[0050] The sending module is used to determine the target answer information of the target question request from the target storage area, wherein the target storage area stores the answer information output by the inference node for the question request;

[0051] The sending module is further used to send the target answer information to the client.

[0052] Optionally, the sending module includes a splitting unit, a determining unit and an integrating unit; wherein,

[0053] The splitting unit is used to perform sentence splitting on the target question request to obtain target split sentences;

[0054] The determining unit is used to determine multiple pieces of solution information of the target split sentence from the target storage area;

[0055] The integration unit is used to integrate the multiple pieces of answer information to obtain target answer information.

[0056] Optionally, the sending module includes a splitting unit, a determining unit and an integrating unit; wherein,

[0057] The splitting unit is used to perform sentence splitting on the target question request to obtain target split sentences;

[0058] The determining unit is used to determine a first split statement and a second split statement in the target split statement;

[0059] The determining unit is further used to determine the first answer information of the first split statement from the target storage area;

[0060] The determining unit is further configured to determine second answer information output by the target node for the second split statement;

[0061] The integration unit is used to integrate the first answer information and the second answer information to obtain target answer information.

[0062] Optionally, the scheduling module includes a node determination unit and a receiving unit; wherein,

[0063] The node determination unit is used to determine an initial target node from each inference node according to the current load state;

[0064] The receiving unit is used to receive return information from the initial target node after sending request information to the initial target node;

[0065] The node determination unit is used to determine the target node in the initial target node according to the return time of the return information.

[0066] Optionally, the scheduling module further includes a generating unit and a forwarding unit; wherein,

[0067] The generating unit is used to generate the scheduling information of the target node;

[0068] The forwarding unit is used to forward the scheduling information to other scheduling servers.

[0069] In addition, to achieve the above-mentioned purpose, the present invention also proposes a service scheduling device, which includes: a memory, a processor, and a service scheduling program stored in the memory and executable on the processor, and the service scheduling program is configured to implement the steps of the service scheduling method described above.

[0070] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a service scheduling program is stored, and when the service scheduling program is executed by a processor, the steps of the service scheduling method described above are implemented.

[0071] The service scheduling method, device, scheduling server and storage medium proposed in the present invention obtain the current load status of each inference node of the inference server when receiving the target question request sent by the client; determine the target node from each inference node according to the current load status; and schedule the target question request to the target node so that the target node responds to the target question request. In the above manner, when a user initiates a question request, the target node can be determined according to the current load status of each inference node in the inference server, and then the question request can be scheduled to the target node, thereby ensuring that the question request can be responded to in a timely manner, thereby effectively shortening the user's request delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a structural diagram of a service scheduling device of a hardware operating environment involved in an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of a flow chart of a first embodiment of a service scheduling method of the present invention;

[0074] Figure 3 This is a functional schematic diagram of a scheduling server in the first embodiment of the service scheduling method of the present invention;

[0075] Figure 4 A schematic diagram of a flow chart of a second embodiment of a service scheduling method of the present invention;

[0076] Figure 5 This is a structural block diagram of the first embodiment of the service scheduling device of the present invention.

[0077] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0078] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0079] Reference Figure 1 , Figure 1 A schematic diagram of the structure of a service scheduling device in a hardware operating environment involved in an embodiment of the present invention.

[0080] like Figure 1 As shown, the service scheduling device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0081] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the service scheduling device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0082] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a service scheduler.

[0083] exist Figure 1In the service scheduling device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the service scheduling device of the present invention can be set in the service scheduling device, and the service scheduling device calls the service scheduling program stored in the memory 1005 through the processor 1001, and executes the service scheduling method provided by the embodiment of the present invention.

[0084] Based on the above hardware structure, an embodiment of the service scheduling method of the present invention is proposed.

[0085] Reference Figure 2 , Figure 2 The figure is a flow chart of a first embodiment of a service scheduling method of the present invention.

[0086] In this embodiment, the service scheduling method includes the following steps:

[0087] Step S10: when receiving a target question request sent by the client, the current load status of each inference node of the inference server is obtained.

[0088] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a mobile phone, a tablet computer, a personal computer, etc., or an electronic device or a scheduling server that can realize the above functions. The scheduling server is taken as an example to illustrate this embodiment and the following embodiments.

[0089] It should be noted that the scheduling server can be a proxy layer added between the client and the inference server, and the scheduling server can play the role of proxy and scheduling; the current load state includes a high load state and a low load state.

[0090] It should be noted that the scheduling server obtains information about all inference nodes through the service registration center, such as the current load status, health status, online and offline information, etc. of the inference nodes. Currently, ETCD is selected as the service registration center.

[0091] In the specific implementation, the scheduling server can support mixed deployment of internal and external networks, as well as local and cloud. The scheduling server can achieve integrated deployment.

[0092] Step S20: determining a target node from each inference node according to the current load state.

[0093] It should be noted that when the current load state of the inference node is a low load state, the inference node can be used as a target node.

[0094] Further, determining the target node from each inference node according to the current load state includes:

[0095] Determining an initial target node from each inference node according to the current load state;

[0096] After sending the request information to the initial target node, receiving the return information of the initial target node;

[0097] A target node among the initial target nodes is determined according to the return time of the return information.

[0098] It should be noted that the initial target node refers to the inference node whose current load state in the inference server is a low load state; the return time refers to the time when the initial target node sends the return information to the scheduling server.

[0099] In the specific implementation, after determining the initial target node in the inference server, the scheduling server will send a request message to the initial target node. After receiving the request message, the initial target node will send a return message to the scheduling server. Since the initial target node includes at least two inference nodes, the scheduling server will receive at least two return messages. The return time of each initial target node sending return information to the scheduling server is different. The initial target node corresponding to the return information first received by the scheduling server can be used as the target node.

[0100] In this embodiment, the target node in the initial target node can be quickly determined according to the return time, so that the target question request can be quickly dispatched to the target node, thereby effectively improving the response efficiency of the reasoning server.

[0101] Step S30: dispatching the target question request to the target node, so that the target node responds to the target question request.

[0102] It should be noted that after the target question request is dispatched to the target node, the target node will calculate the answer information of the target question request according to the target question request.

[0103] Furthermore, after dispatching the target question request to the target node, the method further includes:

[0104] generating scheduling information of the target node;

[0105] The scheduling information is forwarded to other scheduling servers.

[0106] It should be noted that the scheduling information refers to the inference node that has been scheduled by the scheduling server to answer the question at the current moment.

[0107] In the specific implementation, forwarding scheduling information through distributed locks can achieve global control.

[0108] It should be noted that there may be multiple scheduling servers between the client and the inference server, so when one of the scheduling servers determines to schedule the target question request to the target node, it needs to generate the scheduling information of the target node, and then the scheduling server forwards the scheduling information to other scheduling servers. Other scheduling servers can promptly learn that a new question request has been scheduled to the target node, and other scheduling servers will not continue to schedule the question request to the target node.

[0109] In this embodiment, after the target question request is dispatched to the target node, scheduling information is generated, and then the scheduling information is forwarded to other scheduling servers, so that other scheduling servers can timely understand the inference nodes that are scheduled with the question request at the current moment, preventing the same inference node from receiving question requests dispatched by multiple scheduling servers at the same time, thereby effectively ensuring the load balancing of the inference nodes.

[0110] Furthermore, the service scheduling method further includes:

[0111] When the inference server is in a congested state, determining a low priority question request from multiple target question requests;

[0112] When the low-priority question request is in the first state, discarding the low-priority question request;

[0113] When the low priority question request is in the second state, the low priority question request is inserted into the queuing queue of the target node.

[0114] It should be noted that the first state refers to a state that needs to be abandoned, and the second state refers to a state that does not need to be abandoned; when the low-priority question request is in the first state, the low-priority question request can be abandoned, that is, the low-priority question request is not scheduled to the target node; when the low-priority question request is in the second state, the low-priority question request can be inserted into the queue of the target node and wait for delayed processing by the target node.

[0115] In a specific implementation, the priority situation can be determined according to the business situation of the target question request, and then the low-priority question request among the multiple target question requests can be determined according to the priority situation. For example, when the business situation of the target question request is an online business, the target question request is a high-priority question request, and when the business situation of the target question request is an offline business, the target question request is a low-priority question request. The priority situation can also be determined according to the search scenario of the target question request, and then the low-priority question request among the multiple target question requests can be determined according to the priority situation. For example, the search scenario includes map search, translation search and question search, and the priority of question search is higher than that of translation search, and the priority of question search is higher than that of map search, and the priority of map search is higher than that of translation search. Among the multiple target question requests, the question request belonging to translation search can be regarded as a low-priority question request, and among the multiple target question requests, the question request belonging to translation search or map search can be regarded as a low-priority question request.

[0116] In this embodiment, multiple target question requests are prioritized to determine low-priority question requests, and then it is determined whether the low-priority question requests need to be discarded according to their status, so that the congestion state of the inference server can be alleviated.

[0117] Further, determining a low priority question request from a plurality of target question requests includes:

[0118] Determining the business conditions of the multiple target question requests according to the identifiers of the target question requests;

[0119] A low priority question request is determined from the multiple question requests according to the business situation.

[0120] It should be noted that the business situation includes online business and offline business. Since offline business is delayed and insensitive, the target question request belonging to offline business can be regarded as a low-priority question request.

[0121] Furthermore, the service scheduling method further includes:

[0122] Based on the inference service IP list, the abnormal nodes in the inference server are determined through a health check mechanism, and the abnormal nodes are removed from the inference server, wherein the abnormal nodes are inference nodes in an unhealthy state.

[0123] It should be noted that the scheduling server can obtain the inference service IP list from the service registration center; the inference service IP list records the IP address of each inference node in the inference server. The scheduling server regularly detects the inference node through the health check mechanism based on the IP address, so as to determine the abnormal nodes in the inference server that are in an unhealthy state. Among them, the nodes in an unhealthy state may have a physical power outage, network disconnection, or hardware failure, or there may be network delays, abnormal process exits, or the process cannot process requests.

[0124] In this embodiment, the health check mechanism is used to regularly detect the inference nodes to discover abnormal nodes, so that the abnormal nodes can be removed from the inference server in time, thereby preventing the query request from being unresponsive for a long time due to the existence of abnormal nodes in the inference server.

[0125] Furthermore, the service scheduling method further includes:

[0126] When the online and offline information of the inference node is received, the inference node of the inference server is updated according to the online and offline information.

[0127] It should be noted that when the inference node registers and goes online in the service registration center, or when the inference node initiates an offline process in the service registration center, the service registration center will send the online and offline information of the inference node to the scheduling server. When the scheduling server receives the online and offline information, the scheduling server will update the inference node directly connected to it. For example, when the online and offline information is online, the scheduling server will expand the capacity of the inference server, that is, connect to the online inference node. When the online and offline information is offline, the scheduling server will reduce the capacity of the inference server, that is, cancel the connection with the offline inference node.

[0128] Furthermore, the service scheduling method further includes:

[0129] The reasoning node in the overload state is regarded as an overload node;

[0130] The queuing queue of the overloaded node is determined, and the question request corresponding to the queuing queue is dispatched to the idle reasoning node of the reasoning server.

[0131] It should be noted that the queuing queue of the overloaded node refers to the queue formed by the query requests waiting to be processed by the overloaded node; the idle inference node refers to the inference node in the inference server that is in an idle state.

[0132] In a specific implementation, when a certain inference node is in an overloaded state, the recheck mechanism of the scheduling server can be used to schedule the question requests in the queue of the overloaded node to the idle inference node of the inference server.

[0133] In this embodiment, the recheck mechanism of the scheduling server can be used to schedule the question request in the overloaded node to the idle reasoning node of the reasoning server, which can avoid code redundancy in the reasoning server.

[0134] Furthermore, the service scheduling method further includes:

[0135] Determine the queue time of the queued question request in the queued state;

[0136] When the queuing time is greater than a preset queuing time, the queuing question request is dispatched to an idle reasoning node of the reasoning server.

[0137] It should be noted that the queue state refers to the state of waiting in line for processing by the inference node; the preset queue time can be preset; and the queued question request refers to the question request waiting in line for processing by the inference node.

[0138] In a specific implementation, when the queuing time of a queued question request is greater than a preset queuing time, the queued question request may be dispatched to an idle inference node of the inference server.

[0139] In this embodiment, the recheck mechanism of the scheduling server can be used to schedule the queued question request to the idle reasoning node of the reasoning server, which can avoid code redundancy on the reasoning server.

[0140] In the specific implementation, Figure 3 As shown, after the scheduling server receives the target question request sent by the client, the scheduling server needs to read the information of all reasoning nodes through the service registration center, and then determine the target node from multiple reasoning nodes, and then the target node outputs the answer information for the target question request. The scheduling server can also directly determine the answer information of the target question request through the target storage area. The scheduling server needs to obtain the node status information and health status information of the reasoning node through the service registration center.

[0141] This embodiment obtains the current load status of each inference node of the inference server when receiving a target question request sent by the client; determines the target node from each inference node according to the current load status; and dispatches the target question request to the target node so that the target node responds to the target question request. In the above manner, when a user initiates a question request, the target node can be determined according to the current load status of each inference node in the inference server, and then the question request can be dispatched to the target node, thereby ensuring that the question request can be responded to in a timely manner, thereby effectively shortening the user's request delay.

[0142] refer to Figure 4 , Figure 4It is a flow chart of a second embodiment of a service scheduling method of the present invention.

[0143] Based on the above first embodiment, the service scheduling method of this embodiment further includes:

[0144] Step S401: determining target answer information of the target question request from a target storage area, wherein the target storage area stores answer information output by the inference node in response to the question request.

[0145] It should be noted that the target storage area may be a cache with multiple granularity levels; the target storage area is responsible for storing the user's question request and the answer information output by the reasoning node in response to the question request.

[0146] In a specific implementation, the target storage area may store all question requests and the answer information corresponding to all question requests; when the storage capacity of the target storage area is limited, the scheduling server may select a suitable question request from all question requests, and then store the suitable question request and the answer information of the suitable question request in the target storage area, wherein the suitable question request may be a question request whose number of questions asked by the user exceeds a threshold, and the suitable question request may also be a question request belonging to hot information.

[0147] Further, the step of determining target answer information of the target question request from the target storage area includes:

[0148] Perform sentence splitting on the target question request to obtain target split sentences;

[0149] Determine multiple solution information of the target split sentence from the target storage area;

[0150] The multiple pieces of answer information are integrated to obtain target answer information.

[0151] It should be noted that when there are multiple question requests in the target question request, it is necessary to split the target question request into multiple question requests, and then determine multiple answer information of the multiple question requests from the target storage area respectively, and finally the scheduling server integrates the multiple answer information to obtain the target answer information.

[0152] It can be understood that the target split sentence is a plurality of question requests obtained after sentence splitting of the target question request.

[0153] Further, the step of determining target answer information of the target question request from the target storage area includes:

[0154] Perform sentence splitting on the target question request to obtain target split sentences;

[0155] Determine a first split statement and a second split statement in the target split statement;

[0156] Determining first answer information of the first split statement from the target storage area;

[0157] Determine second answer information output by the target node for the second split statement;

[0158] The first answer information and the second answer information are integrated to obtain target answer information.

[0159] It should be noted that when there are multiple question requests in the target question request, the target question request needs to be sentence-splitting to obtain multiple question requests, and the target split sentence is the multiple question requests obtained after the target question request is sentence-splitting.

[0160] It is understandable that some of the multiple question requests of the target question request may not be stored in the target storage area, so the answer information of these question requests not stored in the target storage area cannot be directly obtained from the target storage area, and the target node needs to calculate the question requests before outputting the answer information.

[0161] It should be noted that the first split statement refers to the question request stored in the target storage area among the multiple question requests of the target question request; the second split statement refers to the question request not stored in the target storage area among the multiple question requests of the target question request.

[0162] In a specific implementation, the first answer information of the first split statement can be directly determined from the target storage area, and then the target node calculates and outputs the second answer information for the second split statement, and finally the scheduling server integrates the first answer information and the second answer information to obtain the target answer information.

[0163] Step S402: Send the target answer information to the client.

[0164] It should be noted that after the scheduling server determines the target answer information, it can be sent directly to the client.

[0165] This embodiment determines the target answer information of the target question request from the target storage area, wherein the target storage area stores the answer information output by the inference node for the question request; and sends the target answer information to the client. In the above manner, by setting a multi-granularity level cache, when the user initiates a question request, the answer information of the question request is directly determined from the target storage area, which can not only effectively improve the answer efficiency, but also effectively reduce the calculation pressure of the inference server.

[0166] In addition, an embodiment of the present invention further proposes a storage medium, on which a service scheduling program is stored. When the service scheduling program is executed by a processor, the steps of the service scheduling method described above are implemented.

[0167] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the service scheduling device of the present invention.

[0168] like Figure 5 As shown, the service scheduling device proposed in the embodiment of the present invention includes:

[0169] The acquisition module 10 is used to acquire the current load status of each inference node of the inference server when receiving a target question request sent by the client.

[0170] The determination module 20 is used to determine the target node from each inference node according to the current load state.

[0171] The scheduling module 30 is used to schedule the target question request to the target node, so that the target node responds to the target question request.

[0172] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0173] This embodiment obtains the current load status of each inference node of the inference server when receiving a target question request sent by the client; determines the target node from each inference node according to the current load status; and dispatches the target question request to the target node so that the target node responds to the target question request. In the above manner, when a user initiates a question request, the target node can be determined according to the current load status of each inference node in the inference server, and then the question request can be dispatched to the target node, thereby ensuring that the question request can be responded to in a timely manner, thereby effectively shortening the user's request delay.

[0174] In one embodiment, the service scheduling device further includes a sending module; wherein,

[0175] The sending module is used to determine the target answer information of the target question request from the target storage area, wherein the target storage area stores the answer information output by the inference node for the question request;

[0176] The sending module is further used to send the target answer information to the client.

[0177] In one embodiment, the sending module includes a splitting unit, a determining unit and an integrating unit; wherein,

[0178] The splitting unit is used to perform sentence splitting on the target question request to obtain target split sentences;

[0179] The determining unit is used to determine multiple pieces of solution information of the target split sentence from the target storage area;

[0180] The integration unit is used to integrate the multiple pieces of answer information to obtain target answer information.

[0181] In one embodiment, the sending module includes a splitting unit, a determining unit and an integrating unit; wherein,

[0182] The splitting unit is used to perform sentence splitting on the target question request to obtain target split sentences;

[0183] The determining unit is used to determine a first split statement and a second split statement in the target split statement;

[0184] The determining unit is further used to determine the first answer information of the first split statement from the target storage area;

[0185] The determining unit is further configured to determine second answer information output by the target node for the second split statement;

[0186] The integration unit is used to integrate the first answer information and the second answer information to obtain target answer information.

[0187] In one embodiment, the scheduling module includes a node determination unit and a receiving unit; wherein,

[0188] The node determination unit is used to determine an initial target node from each inference node according to the current load state;

[0189] The receiving unit is used to receive return information from the initial target node after sending request information to the initial target node;

[0190] The node determination unit is used to determine the target node in the initial target node according to the return time of the return information.

[0191] In one embodiment, the scheduling module 30 further includes a generating unit and a forwarding unit; wherein,

[0192] The generating unit is used to generate the scheduling information of the target node;

[0193] The forwarding unit is used to forward the scheduling information to other scheduling servers.

[0194] In one embodiment, the determination module 20 includes a determination unit, a discarding unit and an inserting unit; wherein,

[0195] The determining unit is used to determine a low-priority question request from a plurality of target question requests when the inference server is in a congested state;

[0196] The discarding unit is used to discard the low-priority question request when the low-priority question request is in the first state;

[0197] The inserting unit is used to insert the low priority question request into the queuing queue of the target node when the low priority question request is in the second state.

[0198] In one embodiment, the determining unit is further configured to:

[0199] Determining the business conditions of the multiple target question requests according to the identifiers of the target question requests;

[0200] A low priority question request is determined from the multiple question requests according to the business situation.

[0201] In one embodiment, the service scheduling device further includes a rejection module; wherein,

[0202] The removal module is used to determine the abnormal nodes in the inference server through a health check mechanism based on the inference service IP list, and remove the abnormal nodes from the inference server, wherein the abnormal nodes are inference nodes in an unhealthy state.

[0203] In one embodiment, the service scheduling device further includes an updating module; wherein,

[0204] The updating module is used to update the inference node of the inference server according to the online and offline information when the online and offline information of the inference node is received.

[0205] In one embodiment, the determining unit is further configured to:

[0206] The reasoning node in the overload state is regarded as an overload node;

[0207] The queuing queue of the overloaded node is determined, and the question request corresponding to the queuing queue is dispatched to the idle reasoning node of the reasoning server.

[0208] In one embodiment, the determining unit is further configured to:

[0209] Determine the queue time of the queued question request in the queued state;

[0210] When the queuing time is greater than a preset queuing time, the queuing question request is dispatched to an idle reasoning node of the reasoning server.

[0211] The present invention discloses A1, a service scheduling method, applied to a scheduling server, the service scheduling method comprising:

[0212] When receiving a target query request sent by the client, obtain the current load status of each inference node of the inference server;

[0213] Determining a target node from each inference node according to the current load state;

[0214] The target question request is dispatched to the target node, so that the target node responds to the target question request.

[0215] A2. The service scheduling method as described in A1, wherein the service scheduling method further includes:

[0216] Determining target answer information of the target question request from a target storage area, wherein the target storage area stores answer information output by the inference node for the question request;

[0217] The target answer information is sent to the client.

[0218] A3. The service scheduling method as described in A2, wherein the step of determining target answer information of the target question request from the target storage area comprises:

[0219] Perform sentence splitting on the target question request to obtain target split sentences;

[0220] Determine multiple solution information of the target split sentence from the target storage area;

[0221] The multiple pieces of answer information are integrated to obtain target answer information.

[0222] A4, as described in A2, the service scheduling method, wherein determining the target answer information of the target question request from the target storage area comprises:

[0223] Perform sentence splitting on the target question request to obtain target split sentences;

[0224] Determine a first split statement and a second split statement in the target split statement;

[0225] Determining first answer information of the first split statement from the target storage area;

[0226] Determine second answer information output by the target node for the second split statement;

[0227] The first answer information and the second answer information are integrated to obtain target answer information.

[0228] A5. The service scheduling method as described in A1, wherein determining the target node from each inference node according to the current load state comprises:

[0229] Determining an initial target node from each inference node according to the current load state;

[0230] After sending the request information to the initial target node, receiving the return information of the initial target node;

[0231] A target node among the initial target nodes is determined according to the return time of the return information.

[0232] A6. The service scheduling method as described in A1, after scheduling the target question request to the target node, further comprising:

[0233] generating scheduling information of the target node;

[0234] The scheduling information is forwarded to other scheduling servers.

[0235] A7. The service scheduling method as described in A1, wherein the service scheduling method further includes:

[0236] When the inference server is in a congested state, determining a low priority question request from multiple target question requests;

[0237] When the low-priority question request is in the first state, discarding the low-priority question request;

[0238] When the low priority question request is in the second state, the low priority question request is inserted into the queuing queue of the target node.

[0239] A8. The service scheduling method as described in A7, wherein determining a low-priority question request from a plurality of target question requests comprises:

[0240] Determining the business conditions of the multiple target question requests according to the identifiers of the target question requests;

[0241] A low priority question request is determined from the multiple question requests according to the business situation.

[0242] A9. The service scheduling method as described in A1, wherein the service scheduling method further includes:

[0243] Based on the inference service IP list, the abnormal nodes in the inference server are determined through a health check mechanism, and the abnormal nodes are removed from the inference server, wherein the abnormal nodes are inference nodes in an unhealthy state.

[0244] A10. The service scheduling method as described in A1, wherein the service scheduling method further includes:

[0245] When the online and offline information of the inference node is received, the inference node of the inference server is updated according to the online and offline information.

[0246] A11. The service scheduling method as described in A1, wherein the service scheduling method further includes:

[0247] The reasoning node in the overload state is regarded as an overload node;

[0248] The queuing queue of the overloaded node is determined, and the question request corresponding to the queuing queue is dispatched to the idle reasoning node of the reasoning server.

[0249] A12. The service scheduling method as described in A1, wherein the service scheduling method further includes:

[0250] Determine the queue time of the queued question request in the queued state;

[0251] When the queuing time is greater than a preset queuing time, the queuing question request is dispatched to an idle reasoning node of the reasoning server.

[0252] The present invention also discloses B13, a service scheduling device, the service scheduling device comprising:

[0253] The acquisition module is used to obtain the current load status of each inference node of the inference server when receiving the target question request sent by the client;

[0254] A determination module, used to determine a target node from each inference node according to the current load state;

[0255] The scheduling module is used to schedule the target question request to the target node, so that the target node responds to the target question request.

[0256] B14. The service scheduling device as described in B13, further comprising a sending module; wherein:

[0257] The sending module is used to determine the target answer information of the target question request from the target storage area, wherein the target storage area stores the answer information output by the inference node for the question request;

[0258] The sending module is further used to send the target answer information to the client.

[0259] B15. The service scheduling device as described in B14, wherein the sending module includes a splitting unit, a determining unit and an integrating unit; wherein,

[0260] The splitting unit is used to perform sentence splitting on the target question request to obtain target split sentences;

[0261] The determining unit is used to determine multiple pieces of solution information of the target split sentence from the target storage area;

[0262] The integration unit is used to integrate the multiple pieces of answer information to obtain target answer information.

[0263] B16. The service scheduling device as described in B14, wherein the sending module includes a splitting unit, a determining unit and an integrating unit; wherein,

[0264] The splitting unit is used to perform sentence splitting on the target question request to obtain target split sentences;

[0265] The determining unit is used to determine a first split statement and a second split statement in the target split statement;

[0266] The determining unit is further used to determine the first answer information of the first split statement from the target storage area;

[0267] The determining unit is further configured to determine second answer information output by the target node for the second split statement;

[0268] The integration unit is used to integrate the first answer information and the second answer information to obtain target answer information.

[0269] B17. The service scheduling device as described in B13, wherein the scheduling module includes a node determination unit and a receiving unit; wherein,

[0270] The node determination unit is used to determine an initial target node from each inference node according to the current load state;

[0271] The receiving unit is used to receive return information from the initial target node after sending request information to the initial target node;

[0272] The node determination unit is used to determine the target node in the initial target node according to the return time of the return information.

[0273] B18. The service scheduling device as described in B13, wherein the scheduling module further includes a generating unit and a forwarding unit; wherein,

[0274] The generating unit is used to generate the scheduling information of the target node;

[0275] The forwarding unit is used to forward the scheduling information to other scheduling servers.

[0276] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0277] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the service scheduling method provided in any embodiment of the present invention, and will not be repeated here.

[0278] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0279] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0280] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0281] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A service scheduling method, characterized in that: Applied to a scheduling server, the service scheduling method includes: When receiving a target query request sent by the client, obtain the current load status of each inference node of the inference server; Determining a target node from each inference node according to the current load state; The target question request is dispatched to the target node, so that the target node responds to the target question request.

2. The method according to claim 1, characterized in that The service scheduling method further includes: Determining target answer information of the target question request from a target storage area, wherein the target storage area stores answer information output by the inference node for the question request; The target answer information is sent to the client.

3. The method according to claim 2, characterized in that The step of determining target answer information of the target question request from the target storage area includes: Perform sentence splitting on the target question request to obtain target split sentences; Determine multiple solution information of the target split sentence from the target storage area; The multiple pieces of answer information are integrated to obtain target answer information.

4. The method according to claim 2, characterized in that The step of determining target answer information of the target question request from the target storage area includes: Perform sentence splitting on the target question request to obtain target split sentences; Determine a first split statement and a second split statement in the target split statement; Determining first answer information of the first split statement from the target storage area; Determine second answer information output by the target node for the second split statement; The first answer information and the second answer information are integrated to obtain target answer information.

5. The method according to claim 1, characterized in that Determining the target node from each inference node according to the current load state includes: Determining an initial target node from each inference node according to the current load state; After sending the request information to the initial target node, receiving the return information of the initial target node; A target node among the initial target nodes is determined according to the return time of the return information.

6. The method according to claim 1, characterized in that After dispatching the target question request to the target node, the method further includes: generating scheduling information of the target node; The scheduling information is forwarded to other scheduling servers.

7. The method according to claim 1, characterized in that The service scheduling method further includes: When the inference server is in a congested state, determining a low priority question request from multiple target question requests; When the low-priority question request is in the first state, discarding the low-priority question request; When the low priority question request is in the second state, the low priority question request is inserted into the queuing queue of the target node.

8. A service scheduling device, characterized in that: The service scheduling device comprises: The acquisition module is used to obtain the current load status of each inference node of the inference server when receiving the target question request sent by the client; A determination module, used to determine a target node from each inference node according to the current load state; The scheduling module is used to schedule the target question request to the target node, so that the target node responds to the target question request.

9. A scheduling server, characterized in that: The scheduling server comprises: a memory, a processor, and a service scheduling program stored in the memory and executable on the processor, wherein the service scheduling program is configured to implement the steps of the service scheduling method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a service scheduling program, and when the service scheduling program is executed by the processor, the steps of the service scheduling method according to any one of claims 1 to 7 are implemented.

Images