Batch reasoning method, device and equipment and readable storage medium
Through batch inference method, multiple requests are merged and patterned, which solves the problems of inefficiency and resource competition of Flask and FastAPI under high concurrent requests in the prior art, and realizes more efficient model inference services.
Patent Information
- Application Number
- CN202510668549.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-22
- Publication Date
- 2025-08-15
AI Technical Summary
The existing model inference services based on Flask and FastAPI are less efficient when handling high concurrent requests, and are prone to cause resource competition problems, making it difficult to meet the efficient inference requirements of enterprise-level applications.
The batch inference method is adopted to obtain the inference requests of each requesting party, merge based on preset merging rules, form a batch request task set, and push it to the target model service for processing in accordance with synchronous, asynchronous or streaming request modes, thereby improving inference efficiency.
Through the scheduling merging mechanism, multiple inference requests are merged and processed and sent to the target model service according to the corresponding request mode, which significantly improves the inefficiency problem in the existing technology.
Smart Images
Figure CN120494106A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a batch reasoning method, apparatus, device, and readable storage medium. Background Art
[0002] With the rapid development of artificial intelligence (AI), inference services, as the key link between model training and practical application, have become crucial indicators of AI system performance, driven by their efficiency and flexibility. Python-based AI model inference services widely utilize web frameworks such as Flask and FastAPI. These frameworks offer advantages such as low development barriers, quick adoption, and a robust ecosystem, making them the preferred method for many AI engineers and researchers to deploy model inference services.
[0003] Existing inference methods typically load trained model files into a Python server, define an HTTP interface to receive client requests, parse input data, perform model inference, and return inference results. However, when handling highly concurrent requests, existing frameworks cannot effectively handle asynchronous processing and are prone to resource contention, resulting in low inference efficiency.
[0004] Therefore, how to improve the efficiency of reasoning is an urgent problem that those skilled in the art need to solve. Summary of the Invention
[0005] Based on the above problems, the present application provides a batch inference method, device, equipment and readable storage medium. Based on the scheduling merging mechanism, multiple inference requests are merged and sent to the target model service for processing according to the corresponding request pattern, thereby improving the inference efficiency.
[0006] In a first aspect, an embodiment of the present application provides a batch reasoning method, the method comprising:
[0007] Obtain inference requests sent by each requester;
[0008] Merging the inference requests based on a preset merging rule to obtain a batch request task set;
[0009] According to the request mode corresponding to the batch request task set, the batch request task set is pushed to the target model service corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes one of synchronous, asynchronous and streaming.
[0010] Optionally, obtaining the inference request sent by each requesting party includes:
[0011] Accept inference requests from various requesting parties;
[0012] The inference request is written into a request aggregation queue, and a unique corresponding identity is set for each inference request.
[0013] Optionally, merging the inference requests based on a preset merging rule to obtain a batch request task set includes:
[0014] Determine the actual number of inference requests received within a preset time window;
[0015] If the actual number is less than the preset number, all inference requests within the preset time window are merged to obtain a batch request task set.
[0016] Optionally, merging the inference requests based on a preset merging rule to obtain a batch request task set includes:
[0017] Determine the actual number of inference requests received within a preset time window;
[0018] If the actual number is greater than or equal to the preset number, the preset number of inference requests obtained first are merged to obtain a batch request task set.
[0019] Optionally, pushing the batch request task set to a target model service corresponding to the batch request task set according to a request mode corresponding to the batch request task set, so that the target model service processes the batch request task set, includes:
[0020] Determining a target model service and a request mode corresponding to the batch request task set;
[0021] The batch request task set is added to the inference request queue, and the batch request task set is pushed to the target model service according to the request mode, so that the target model service processes the batch request task set.
[0022] Optionally, the method further includes:
[0023] Obtaining the batch inference results returned by the target model service;
[0024] The batch inference results are split, and based on the identity identifier, the inference results corresponding to each inference request in the batch request task set are pushed to the corresponding requester.
[0025] Optionally, the method further includes:
[0026] Monitoring the current status of each preset model service; the target model service is one of the preset model services;
[0027] If the current status exceeds a preset threshold, an email warning is sent.
[0028] In a second aspect, an embodiment of the present application provides a batch inference device, including:
[0029] The acquisition module is used to obtain the inference requests sent by each requester;
[0030] A request aggregation module is used to merge the inference requests based on a preset merging rule to obtain a batch request task set;
[0031] An inference scheduling module is used to push the batch request task set to the target model service corresponding to the batch request task set in accordance with the request mode corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes one of synchronous, asynchronous and streaming.
[0032] In a third aspect, an embodiment of the present application provides a batch inference device, including:
[0033] memory for storing computer programs;
[0034] A processor is configured to implement the steps of the batch inference method described above when executing the computer program.
[0035] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the batch inference method described above are implemented.
[0036] It can be seen from the above technical solutions that compared with the existing technology, this application has the following advantages:
[0037] This application first obtains the inference requests sent by each requesting party. Then, based on the preset merging rules, the inference requests are merged to obtain a batch request task set. Finally, according to the request mode corresponding to the batch request task set, the batch request task set is pushed to the target model service corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes synchronous, asynchronous and streaming. In this way, based on the scheduling merging mechanism, multiple inference requests are merged and sent to the target model service for processing according to the corresponding request mode, thereby improving the inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A flow chart of a batch reasoning method provided in an embodiment of the present application;
[0039] Figure 2A schematic diagram of the overall logic provided by an embodiment of the present application;
[0040] Figure 3 A schematic diagram of an overall structure provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of the structure of a batch inference device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] As mentioned above, existing inference methods suffer from low inference efficiency. Specifically, the FastAPI and Flask frameworks used to build models support multiple concurrent request inputs. However, for model serving, the time consumption is primarily concentrated in the model inference phase. Flask does not natively support async processing. Although FastAPI introduces an async mechanism, the model inference process often involves CPU / GPU resource operations, and its async processing does not effectively improve processing efficiency and is prone to resource contention. Therefore, while existing model serving based on FastAPI and Flask is suitable for rapid verification in the early stages of development, it struggles to meet the efficient inference requirements of enterprise-level applications with high concurrency.
[0043] To address the above issues, the present application provides a batch inference method, comprising: first, obtaining inference requests sent by each requester. Then, merging the inference requests based on preset merging rules to obtain a batch request task set. Finally, according to the request mode corresponding to the batch request task set, the batch request task set is pushed to the target model service corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes one of synchronous, asynchronous, and streaming.
[0044] In this way, based on the scheduling merging mechanism, multiple inference requests are merged and sent to the target model service for processing according to the corresponding request pattern, thereby improving the inference efficiency.
[0045] It should be noted that the batch inference method, apparatus, device, and readable storage medium provided in this application can be applied in the field of data processing technology. The above is only an example and does not limit the application field of the batch inference method, apparatus, device, and readable storage medium provided in this application.
[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0047] Figure 1 A flow chart of a batch reasoning method provided in an embodiment of the present application. Figure 1 As shown, a batch reasoning method provided by an embodiment of the present application may include:
[0048] S101: Obtain inference requests sent by various requesting parties.
[0049] Figure 2 A schematic diagram of the overall logic provided by the embodiment of the present application. Figure 2 As shown, the batch inference system (device) serves as an intermediate inference scheduling layer, embedded between the requester and the model service. It is responsible for the unified reception, caching, aggregation, scheduling, and distribution of inference requests, and returns the processing results. The batch inference system is compatible with any hardware and most model services. It can be deployed together with the model service to process inference requests from only one model service, or it can be deployed independently on a single machine to simultaneously schedule and process inference requests from multiple model services. It is understandable that the batch inference system typically interacts with multiple requesters (requesters 1, 2, 3, etc.), can obtain inference requests sent by each requester, and can also feedback the corresponding inference results to each requester, that is, distribute the results.
[0050] In addition, since the methods for obtaining inference requests are different, the embodiments of the present application can illustrate a possible obtaining method.
[0051] In one case, S101: obtaining inference requests sent by various requesting parties may specifically include:
[0052] Accept inference requests from various requesting parties;
[0053] The inference request is written into a request aggregation queue, and a unique corresponding identity is set for each inference request.
[0054] In actual applications, a batch inference system (device) can receive inference requests from various requesters. After all inference requests are received, they are first written to the request aggregation queue, and a unique ID (identifier) is generated for each inference request to distinguish them.
[0055] In addition, regarding how to transmit data, the embodiment of the present application introduces an input and output adaptation module, which is compatible with various types of inference request types and request methods, various types of abnormal requests and abnormal returns, and encapsulates and adapts input and output.
[0056] Figure 3 A schematic diagram of an overall structure provided in an embodiment of the present application. Figure 3 As shown, in actual applications, in order to better implement the reasoning task, the batch reasoning system provided by the embodiment of the present application may include an acquisition module, an input and output adaptation module, a request aggregation module, an inference scheduling module, a result distribution module, and a log and monitoring module. Among them, the acquisition module can be integrated into the request aggregation module, and the input and output adaptation module can adapt to requests sent in various ways such as http, websocket, redis, etc., and uniformly process and convert them into a format that can be processed by the request aggregation module. The results received through the above-mentioned different methods are adapted, and the model reasoning results are converted into formats that are suitable for different methods and pushed. The input and output adaptation module can also intercept abnormal requests and directly return an error prompt of an incorrect format to the requester to avoid the additional pressure on the model caused by incorrect requests. In addition, exception handling can be added to the model processing process. When a single error is processed in batches, it will not affect the results of other requests in the same batch, and error results can be returned for reasoning requests that handle exceptions.
[0057] S102: Merging the inference requests based on a preset merging rule to obtain a batch request task set.
[0058] In actual applications, multiple requesters continuously send inference requests to the batch inference system. Once the inference requests are received, the request aggregation module merges the multiple inference requests based on preset merging rules to generate a batch request task set. It is understood that the preset merging rules are manually set and can be based on the time at which the inference requests are received or the number of incoming requests. Typically, a batch request task set includes multiple inference requests.
[0059] In addition, since the methods of merging inference requests are different, the embodiments of the present application can illustrate a possible merging method.
[0060] In one case, S102: merging the inference requests based on a preset merging rule to obtain a batch request task set may specifically include:
[0061] Determine the actual number of inference requests received within a preset time window;
[0062] If the actual number is less than the preset number, all inference requests within the preset time window are merged to obtain a batch request task set.
[0063] In practical applications, preset merge rules can be set based on time and quantity. For example, the preset merge rule stipulates that the calculation starts from the time the first inference request is received. If the actual number of inference requests received by the preset time window does not reach the preset number, all inference requests within the preset time window will be merged to obtain a batch request task set. In other words, let's set the preset time window to 1 second and the preset number to 128. When the batch inference system receives the first inference request, the timing begins. At 1 second, the actual number of inference requests received is 100. At this time, 100 is less than 128. The system will merge the 100 inference requests received within 1 second to obtain a batch request task set. This is then pushed to the next module (the inference scheduling module).
[0064] In addition, since the methods of merging inference requests are different, the embodiments of the present application can illustrate another possible merging method.
[0065] In another case, S102: merging the inference requests based on a preset merging rule to obtain a batch request task set may further include:
[0066] Determine the actual number of inference requests received within a preset time window;
[0067] If the actual number is greater than or equal to the preset number, the preset number of inference requests obtained first are merged to obtain a batch request task set.
[0068] In actual applications, based on the above example, the preset merging rules can be further expanded. If the actual number of inference requests received within the preset time window is greater than or equal to the preset number, the preset number of inference requests will be merged to form a batch request task set. Continuing with the above description, assume the preset time window is 1 second and the preset number of inference requests is 128. When the batch inference system receives the first inference request, the timer begins. By 0.5 seconds, the actual number of inference requests received is 128, and by 1 second, the actual number of inference requests received is 200. The system will merge the 128 inference requests received within the first 0.5 seconds to form a single batch request task set. This is then pushed to the next module (the inference scheduling module). The 129th inference request becomes the first inference request again and continues to be merged using the preset merging rules, starting from the time the 129th inference request was received.
[0069] S103: According to the request mode corresponding to the batch request task set, the batch request task set is pushed to the target model service corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes one of synchronous, asynchronous and streaming.
[0070] In actual applications, the inference scheduling module is connected to the request aggregation module, and can execute inference tasks according to the aggregated batch request task set and perform scheduling control, that is, according to the request pattern corresponding to the batch request task set and the corresponding target model service, so that the target model service can process the batch request task set and obtain batch inference results.
[0071] In addition, since the methods of pushing batch request task sets are different, the embodiments of the present application can illustrate a possible pushing method.
[0072] In one case, pushing the batch request task set to a target model service corresponding to the batch request task set according to a request mode corresponding to the batch request task set, so that the target model service processes the batch request task set, includes:
[0073] Determining a target model service and a request mode corresponding to the batch request task set;
[0074] The batch request task set is added to the inference request queue, and the batch request task set is pushed to the target model service according to the request mode, so that the target model service processes the batch request task set.
[0075] In actual applications, the inference scheduling module receives aggregated requests (batch request task sets) merged and pushed by the request aggregation module and identifies the model service and request mode (synchronous, asynchronous, streaming, etc.) corresponding to each batch request task set. Once identification is complete, the inference scheduling module adds all batch request task sets to the inference request queue and then pushes each batch request task set to its corresponding target model service based on its corresponding target model service and request mode. Furthermore, the inference scheduling module can also implement supermarket control for model processing time to ensure model processing efficiency. Furthermore, after push completion, the inference scheduling module can also clear batch request task sets from the inference request queue. For example, if the inference request queue contains batch request task sets A, B, C, and D, the inference scheduling module can choose to clear batch request task set A from the inference request queue immediately after pushing A to its corresponding target model service. At this point, the inference request queue will only contain batch request task sets B, C, and D.
[0076] In addition, since the methods of returning the inference results are different, the embodiments of the present application can illustrate a possible return method.
[0077] In one embodiment, the method further comprises:
[0078] Obtaining the batch inference results returned by the target model service;
[0079] The batch inference results are split, and based on the identity identifier, the inference results corresponding to each inference request in the batch request task set are pushed to the corresponding requester.
[0080] In actual applications, the model service processes the batch request task set and obtains batch inference results, which are then fed back to the batch inference system. The result distribution module in the system can receive the batch inference results fed back by the model service, split the batch inference results, and push the inference results corresponding to each inference request to each request source. Specifically, the batch inference results fed back by the model service are also stored in the inference request queue. The result distribution module receives the batch inference results from the inference request queue, and then splits the batch inference results to obtain the inference results and unique ID for each inference request. Furthermore, the inference results are pushed to the corresponding requester based on the unique ID. In addition, the result distribution module can also clean up the batch inference results in the inference request queue. For example, the inference request queue includes batch inference results a, b, c, and d. The result distribution module can choose to split the batch inference result a and push it to process it, and then immediately clean up the batch inference result a in the inference request queue. At this time, the inference request queue only includes batch inference results b, c, and d.
[0081] In addition, since the methods of monitoring the reasoning process are different, the embodiments of the present application can illustrate a possible monitoring method.
[0082] In one embodiment, the method further comprises:
[0083] Monitoring the current status of each preset model service; the target model service is one of the preset model services;
[0084] If the current status exceeds a preset threshold, an email warning is sent.
[0085] In practical applications, in order to better ensure the stability of system operation. The embodiment of the present application also provides a logging and monitoring module that can record the key indicators and anomalies of each module in the system, thereby realizing feedback. Specifically, the logging and monitoring module can record the number of inference requests in each batch of the request aggregation module, as well as the unique ID, request content, and request time of each inference request. For the inference scheduling module, it records the request volume of each model service (the number of batch request task sets corresponding to each target model service) and the processing time (the time spent processing a batch request task set at each task node, such as identifying the target model service and request pattern, pushing to the target model service, and waiting for the target model service to infer and return the inference result, etc.), as well as the model service and timeout time that have timed out. In addition, the logging and monitoring of the result distribution module can record the inference result, end time, and error content corresponding to each inference request. For the input and output adaptation module, it can record the specific content of each abnormal inference request, etc. In addition, the logging and monitoring module can also monitor the current status of each preset model service, such as whether the request pressure and return time are normal. If the preset threshold or average is exceeded, an email warning will be sent.
[0086] In summary, the present application first obtains the inference requests sent by each requesting party. Then, the inference requests are merged based on the preset merging rules to obtain a batch request task set. Finally, according to the request mode corresponding to the batch request task set, the batch request task set is pushed to the target model service corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes synchronous, asynchronous and streaming. In this way, based on the scheduling merging mechanism, multiple inference requests are merged and sent to the target model service for processing according to the corresponding request mode, thereby improving the inference efficiency.
[0087] Based on the batch reasoning method provided in the above embodiment, the present application also provides a batch reasoning device. The batch reasoning device is described below in conjunction with the embodiments and drawings.
[0088] Figure 4 A schematic diagram of the structure of a batch reasoning device provided in an embodiment of the present application. Figure 4 As shown, the batch inference device 400 provided in the embodiment of the present application includes:
[0089] The acquisition module 401 is used to acquire the inference request sent by each requesting party;
[0090] A request aggregation module 402 is configured to merge the inference requests based on a preset merging rule to obtain a batch request task set;
[0091] The inference scheduling module 403 is used to push the batch request task set to the target model service corresponding to the batch request task set according to the request mode corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes one of synchronous, asynchronous and streaming.
[0092] As an implementation method, regarding how to obtain the inference request sent by each requesting party, the acquisition module 401 is specifically configured to:
[0093] Accept inference requests from various requesting parties;
[0094] The inference request is written into a request aggregation queue, and a unique corresponding identity is set for each inference request.
[0095] As an implementation manner, regarding how to merge the inference requests based on a preset merging rule, the request aggregation module 402 is specifically configured to:
[0096] Determine the actual number of inference requests received within a preset time window;
[0097] If the actual number is less than the preset number, all inference requests within the preset time window are merged to obtain a batch request task set.
[0098] As another embodiment, regarding how to merge the inference requests based on a preset merging rule, the request aggregation module 402 is further configured to:
[0099] Determine the actual number of inference requests received within a preset time window;
[0100] If the actual number is greater than or equal to the preset number, the preset number of inference requests obtained first are merged to obtain a batch request task set.
[0101] As an implementation method, regarding how to push the batch request task set to the corresponding target model service, the above-mentioned reasoning scheduling module 403 is specifically used to:
[0102] Determining a target model service and a request mode corresponding to the batch request task set;
[0103] The batch request task set is added to the inference request queue, and the batch request task set is pushed to the target model service according to the request mode, so that the target model service processes the batch request task set.
[0104] As an implementation method, regarding how to implement inference result feedback, the batch inference device 400 further includes: a result distribution module;
[0105] A result distribution module, configured to obtain the batch inference results returned by the target model service;
[0106] The batch inference results are split, and based on the identity identifier, the inference results corresponding to each inference request in the batch request task set are pushed to the corresponding requester.
[0107] As an implementation method, regarding how to record the reasoning process, the batch reasoning device 400 further includes: a log and monitoring module;
[0108] A log and monitoring module, configured to monitor the current status of each preset model service; the target model service is one of the preset model services;
[0109] If the current status exceeds a preset threshold, an email warning is sent.
[0110] In summary, the present application first obtains the inference requests sent by each requesting party. Then, the inference requests are merged based on the preset merging rules to obtain a batch request task set. Finally, according to the request mode corresponding to the batch request task set, the batch request task set is pushed to the target model service corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes synchronous, asynchronous and streaming. In this way, based on the scheduling merging mechanism, multiple inference requests are merged and sent to the target model service for processing according to the corresponding request mode, thereby improving the inference efficiency.
[0111] In addition, the present application also provides a batch inference device, including: a memory for storing a computer program; and a processor for implementing the steps of the batch inference method described above when executing the computer program.
[0112] In addition, the present application also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the batch inference method described above are implemented.
[0113] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A batch inference method, characterized in that: The method comprises: Obtain inference requests sent by each requester; Merging the inference requests based on a preset merging rule to obtain a batch request task set; According to the request mode corresponding to the batch request task set, the batch request task set is pushed to the target model service corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes one of synchronous, asynchronous and streaming.
2. The method according to claim 1, characterized in that The obtaining of the inference request sent by each requesting party includes: Accept inference requests from various requesting parties; The inference request is written into a request aggregation queue, and a unique corresponding identity is set for each inference request.
3. The method according to claim 1, characterized in that The merging of the inference requests based on a preset merging rule to obtain a batch request task set includes: Determine the actual number of inference requests received within a preset time window; If the actual number is less than the preset number, all inference requests within the preset time window are merged to obtain a batch request task set.
4. The method according to claim 1, wherein The merging of the inference requests based on a preset merging rule to obtain a batch request task set includes: Determine the actual number of inference requests received within a preset time window; If the actual number is greater than or equal to the preset number, the preset number of inference requests obtained first are merged to obtain a batch request task set.
5. The method according to claim 1, wherein The step of pushing the batch request task set to a target model service corresponding to the batch request task set according to a request mode corresponding to the batch request task set, so that the target model service processes the batch request task set, includes: Determining a target model service and a request mode corresponding to the batch request task set; The batch request task set is added to the inference request queue, and the batch request task set is pushed to the target model service according to the request mode, so that the target model service processes the batch request task set.
6. The method according to claim 2, characterized in that The method further comprises: Obtaining the batch inference results returned by the target model service; The batch inference results are split, and based on the identity identifier, the inference results corresponding to each inference request in the batch request task set are pushed to the corresponding requester.
7. The method according to claim 1, characterized in that The method further comprises: Monitoring the current status of each preset model service; the target model service is one of the preset model services; If the current status exceeds a preset threshold, an email warning is sent.
8. A batch inference device, characterized in that: include: The acquisition module is used to obtain the inference requests sent by each requester; A request aggregation module is used to merge the inference requests based on a preset merging rule to obtain a batch request task set; An inference scheduling module is used to push the batch request task set to the target model service corresponding to the batch request task set in accordance with the request mode corresponding to the batch request task set, so that the target model service processes the batch request task set and obtains batch inference results; the request mode includes one of synchronous, asynchronous and streaming.
9. A batch inference device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the batch inference method according to any one of claims 1 to 7 when executing the computer program.
10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the batch inference method according to any one of claims 1 to 7.