A Streaming Batch Inference Method and System Based on Message Queues

Abstract

The present invention provides a streaming batch inference method and system based on a message queue; including: in a complete large batch data source, a data source adapter is used to stream and pull small batch data, and after multiple executions of streaming pulling, the complete large batch data source is sliced to form multiple small batch data; the multiple small batch data are put into a batch inference queue through a message queue middleware for immediate batch inference in the message queue mode to obtain multiple small batch data inference results; the multiple small batch data inference results are compressed and sent to an inference result queue through the message queue mode for consumption by a concurrent data source adapter; the concurrent data source adapter listens to the inference result queue and decompresses and saves each one to a data source database one by one, realizing the streaming pulling, concurrent inference, and batch storage of the complete large batch data source.

Description

Technical Field

[0001] The present invention relates to the technical field of concurrent inference data processing for streaming pull message queues. More specifically, the present invention relates to a streaming batch inference method and system based on a message queue. Background Art

[0002] Currently, with the explosive growth of data scale and the high increase in the complexity of data sources, the technical requirements for the speed and efficiency of data processing and information inference are getting higher and higher. Specific problems include: how to pull small batches of data from a large number of data sources and split them into multiple small batches of data, how to perform batch inference in the batch inference queue of batch inference nodes in real-time in the message queue mode, how to perform concurrent data source adaptation and compress the inference results of multiple small batches of data and send them to the inference result queue for consumption adaptation, how to monitor the inference results and save them one by one to the data source database for concurrent inference and batch storage of the complete large number of data sources. Therefore, it is necessary to propose a streaming batch inference method and system based on a message queue to at least partially solve the problems existing in the prior art. Summary of the Invention

[0003] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further described in detail in the Detailed Implementation section. The Summary of the Invention section of the present invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the protection scope of the claimed technical solution.

[0004] To at least partially solve the above problems, the present invention provides a streaming batch inference method based on a message queue, including:

[0005] S100, in a complete large number of data sources, streaming pull small batches of data through a data source adapter and perform streaming pull multiple times to split the complete large number of data sources into multiple small batches of data;

[0006] S200, put multiple small batches of data into a batch inference queue through a message queue middleware to perform batch inference in the message queue mode in real-time, and obtain inference results of multiple small batches of data;

[0007] S300, compress the inference results of multiple small batches of data and send them to the inference result queue in the message queue mode for consumption by a concurrent data source adapter;

[0008] S400, the concurrent data source adapter monitors the inference result queue and decompresses and saves each one to the data source database one by one, realizing streaming pull, concurrent inference, and batch storage of the complete large number of data sources.

[0009] Preferably, S100 includes:

[0010] S101, set the data pulling range in the complete large - batch data source; according to the data pulling range, stream - pull small - batch data through the data source adapter;

[0011] S102, divide the complete large - batch data source by performing stream - pulling through the data source adapter multiple times to form multiple small - batch data.

[0012] Preferably, S200 includes:

[0013] S201, compress each small - batch data among multiple small - batch data to form multiple messages to be inferred;

[0014] S202, through the message queue mode, perform batch inference in the batch inference queue of the batch inference node for multiple messages to be inferred through the batch inference queue model in real - time to obtain multiple small - batch data inference results;

[0015] The batch inference node includes one or more.

[0016] Preferably, S300 includes:

[0017] S301, compress multiple small - batch data inference results to form multiple small - batch data inference result messages;

[0018] S302, through the message queue mode, send multiple small - batch data inference result messages to the inference result queue for consumption by the concurrent data source adapter.

[0019] Preferably, S400 includes:

[0020] S401, the concurrent data source adapter monitors the inference result queue, receives multiple small - batch data inference result messages and decompresses them one by one to obtain multiple decompressed small - batch data inference results;

[0021] S402, save multiple decompressed small - batch data inference results to the data source database one by one to achieve streaming pull, concurrent inference, and batch - by - batch storage of the complete large - batch data source.

[0022] The present invention provides a streaming batch inference system based on a message queue, including:

[0023] The data source adapter streaming pull module, in the complete large - batch data source, stream - pull small - batch data through the data source adapter and divide the complete large - batch data source by performing stream - pulling multiple times to form multiple small - batch data;

[0024] The queue model instant inference module puts multiple small batches of data into the batch inference queue through the message queue middleware, and instantaneously performs batch inference in the message queue mode to obtain the inference results of multiple small batches of data;

[0025] The concurrent data source adaptation module compresses the inference results of multiple small batches of data and sends them to the inference result queue in the message queue mode for consumption by the concurrent data source adapter;

[0026] The inference result monitoring and warehousing module: The concurrent data source adapter monitors the inference result queue, decompresses each one one by one, and saves them to the data source database one by one, realizing the streaming pull and concurrent inference and batch warehousing of the complete large batch of data sources.

[0027] Preferably, the data source adapter streaming pull module includes:

[0028] The range setting adaptation pull unit sets the data pull range in the complete large batch of data sources; according to the data pull range, it streams and pulls small batches of data through the data source adapter;

[0029] The loop execution data source splitting unit splits the complete large batch of data sources into multiple small batches of data by performing the data source adapter streaming pull multiple times.

[0030] Preferably, the queue model instant inference module includes:

[0031] The data compression inference message unit compresses multiple small batches of data one by one to form multiple messages to be inferred;

[0032] The message queue distributed concurrent inference unit, through the message queue mode, instantaneously performs batch inference in the batch inference queue of the batch inference node on multiple messages to be inferred, and obtains the inference results of multiple small batches of data;

[0033] The batch inference node includes one or more.

[0034] Preferably, the concurrent data source adaptation module includes:

[0035] The data inference result compression unit compresses the inference results of multiple small batches of data to form multiple inference result messages of small batches of data;

[0036] The inference message queue sending unit sends multiple inference result messages of small batches of data to the inference result queue in the message queue mode for consumption by the concurrent data source adapter.

[0037] Preferably, the inference result monitoring and warehousing module includes:

[0038] The inference queue monitoring decompression unit, and the concurrent data source adapter monitors the inference result queue, receives multiple small-batch data inference result messages for decompression one by one, and obtains multiple decompressed small-batch data inference results;

[0039] The inference result batch-by-batch storage unit stores multiple decompressed small-batch data inference results into the data source database one by one, realizing the streaming pull and concurrent inference batch-by-batch storage of the complete large-batch data source.

[0040] Compared with the prior art, the present invention has at least the following beneficial effects:

[0041] A streaming batch inference method and system based on a message queue. In a complete large batch of data sources, a data source adapter is used to stream and pull small batches of data, and after multiple executions of streaming pulls, the complete large batch of data source is segmented to form multiple small batches of data. The multiple small batches of data are put into a batch inference queue through a message queue middleware for immediate batch inference in the message queue mode to obtain inference results for multiple small batches of data. The inference results for multiple small batches of data are compressed and sent to an inference result queue through the message queue mode for consumption by a concurrent data source adapter. The concurrent data source adapter listens to the inference result queue, decompresses each one by one and saves them to a data source database one by one, realizing streaming pull, concurrent inference, and batch storage of a complete large batch of data sources. The data source adapter pulls data within a specified range from the data source in a streaming manner. After pulling a part of the data, a small batch of data can be formed. The data source includes: a database or a file system. Such a pull of a complete large batch of data can be segmented into many streaming pulls of small batches of data. After the small batch of data is compressed, an inference message can be formed. Through the message queue mode, each small batch of data is sent to the batch inference queue for consumption by batch inference nodes for small batch batch inference. The batch inference nodes can be one or more, listening to the batch inference queue, receiving small batch inference messages sent from the data source adapter, decompressing to obtain small batch of data, and using a specified model for batch inference to obtain small batch inference results. The inference results are compressed to form result messages and sent to the inference result queue through the message queue mode to be consumed by the data source adapter for data storage of the small batch inference results. The data source adapter listens to the inference result queue, decompresses the message after receiving the small batch inference result to obtain the small batch inference result, and then saves the result. All data pulls are completed in the data pull stage and the entire process is processed in parallel. The data source adapter can perform streaming pulls on a large batch of data, splitting it into small batches. The batch inference nodes do not need to wait for all data pulls to be completed before starting to infer. Through the message queue mode, multiple batch inference nodes can perform distributed concurrent processing on the same large batch of inference tasks to improve the inference efficiency. The messages in the message queue use common compression techniques to compress the data, reducing the network transmission pressure. After the small batch of data is inferred, it can be immediately transmitted to the data source adapter through the message queue and stored immediately, without waiting for all data to be inferred before unified storage. The entire inference process changes the traditional serial process of data pull - batch inference - data storage into a streaming concurrent process, greatly improving the operation efficiency of multiple links.

[0042] A streaming batch inference method and system based on a message queue according to the present invention. Other advantages, objectives, and features of the present invention will be partially reflected by the following description and partially understood by those skilled in the art through the research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0044] Figure 1 It is a diagram of an embodiment of a streaming batch inference method based on a message queue according to the present invention.

[0045] Figure 2 It is a diagram of an embodiment of a streaming batch inference system based on a message queue according to the present invention.

[0046] Figure 3 It is a diagram of another embodiment of a streaming batch inference method and system based on a message queue according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] The following further describes the present invention in detail in conjunction with the drawings and embodiments, so that those skilled in the art can implement it with reference to the specification. As shown in the figure, the present invention provides a streaming batch inference method based on a message queue, including:

[0048] S100. In a complete large batch data source, a data source adapter is used to stream and pull small batch data, and after multiple executions of streaming and pulling, the complete large batch data source is segmented to form multiple small batch data.

[0049] S200. For multiple small batch data, the multiple small batch data are put into a batch inference queue through a message queue middleware, and batch inference in the message queue mode is performed immediately to obtain multiple small batch data inference results.

[0050] S300. Compress the multiple small batch data inference results and send them to an inference result queue through the message queue mode for consumption by a concurrent data source adapter.

[0051] S400. The concurrent data source adapter listens to the inference result queue, decompresses each one one by one, and saves them to a data source database one by one, realizing the streaming pull, concurrent inference, and batch storage of a complete large batch data source.

[0052] The principles and effects of the above technical solution are as follows: The present invention provides a streaming batch inference method based on a message queue, including: in a complete large batch of data sources, a data source adapter is used to stream and pull small batches of data, and after multiple executions of streaming pulls, the complete large batch of data sources is segmented to form multiple small batches of data; the multiple small batches of data are put into a batch inference queue through a message queue middleware for immediate batch inference in the message queue mode to obtain multiple small batch data inference results; the multiple small batch data inference results are compressed and sent to an inference result queue through the message queue mode for consumption by a concurrent data source adapter; the concurrent data source adapter listens to the inference result queue, decompresses each one by one and saves them to the data source database one by one, realizing the streaming pull, concurrent inference, and batch storage of the complete large batch of data sources; the data source adapter pulls specified range of data from the data source, and the way of pulling data from the data source is streaming. After the data source adapter pulls part of the data, a small batch of data can be formed; the data source includes: a database or a file system; such a pull of a complete large batch of data can be segmented into a streaming pull of many small batches of data; among them, after the small batch of data is compressed, an inference message can be formed, and through the message queue mode, each small batch of data is sent to the batch inference queue for consumption by batch inference nodes for small batch batch inference; the batch inference nodes can be one or more, listening to the batch inference queue, receiving the small batch inference messages sent from the data source adapter, decompressing to obtain small batch of data, and using a specified model for batch inference to obtain small batch inference results. Among them, the inference results are compressed to form result messages, and through the message queue mode, a small batch of inference result data is sent to the inference result queue for consumption by the data source adapter to save the small batch inference results; the data source adapter listens to the inference result queue, decompresses the message after receiving the small batch inference results to obtain the small batch inference results, and then saves the results; in the data pulling stage, all data pulling is completed and the entire process is processed in parallel; the data source adapter can perform streaming pull on a large batch of data, split it into small batches, and the batch inference nodes do not need to wait for all data pulling to be completed to start inference; through the message queue mode, multiple batch inference nodes can perform distributed concurrent processing on the same large batch of inference tasks to improve the inference efficiency; the messages in the message queue use common compression technologies to compress the data, reducing the network transmission pressure; after the small batch of data is inferred, it can be immediately transmitted to the data source adapter through the message queue and immediately stored in the database without waiting for all data to be inferred before unified storage; the entire inference process changes the traditional serial process of data pulling - batch inference - data storage into a streaming concurrent process, greatly improving the operation efficiency of multiple links.

[0053] In one embodiment, S100 includes:

[0054] S101. Set the data pulling range in the complete large - batch data source; according to the data pulling range, stream - pull a small batch of data through the data source adapter.

[0055] S102. By repeatedly performing the stream - pulling of the data source adapter, split the complete large - batch data source to form multiple small - batch data.

[0056] The principle and effect of the above - mentioned technical solution are as follows: Set the data pulling range in the complete large - batch data source; according to the data pulling range, stream - pull a small batch of data through the data source adapter; by repeatedly performing the stream - pulling of the data source adapter, split the complete large - batch data source to form multiple small - batch data; the data source includes: a database or a file system; by repeatedly performing the stream - pulling of the data source adapter, splitting the complete large - batch data source to form multiple small - batch data includes: during the process of pulling a small batch of data, determine in real - time whether the data in the complete large - batch data source has been completely pulled, and obtain the data pulling determination information; according to the data pulling determination information, determine whether to end the data pulling or continue to perform the stream - pulling of the data source adapter; according to the data pulling determination information, determine whether to end the data pulling or continue to perform the stream - pulling of the data source adapter includes: according to the data pulling determination information, if it is determined that the data in the complete large - batch data source has been completely pulled, then the data pulling ends; if it is determined that the data in the complete large - batch data source has not been completely pulled, then continue to stream - pull a small batch of data through the data source adapter until the data in the complete large - batch data source has been completely pulled, and split the complete large - batch data source to form multiple small - batch data.

[0057] In one embodiment, S200 includes:

[0058] S201. Compress each of the multiple small - batch data to form multiple messages to be inferred.

[0059] S202. Through the message queue mode, perform batch inference in the batch inference queue of the batch inference node for the multiple messages to be inferred through the batch inference queue model in real - time, and obtain multiple small - batch data inference results.

[0060] The batch inference node includes one or more.

[0061] The principles and effects of the above technical solution are as follows: Multiple small-batch data are compressed one by one to form multiple messages to be inferred; through the message queue mode, multiple messages to be inferred are batch-inferred in the batch inference queue of the batch inference node through the batch inference queue model to obtain multiple small-batch data inference results; there are one or more batch inference nodes; through the message queue mode, batch inference of multiple messages to be inferred in the batch inference queue of the batch inference node includes: sending multiple messages to be inferred to the batch inference queue of the batch inference node through the message queue mode; listening to the batch inference queue, receiving multiple messages to be inferred sent from the data source adapter, decompressing each of the multiple messages to be inferred to obtain multiple decompressed small-batch data for consumption by the batch inference node; the batch inference node does not wait for all the data in the complete large-batch data source to be pulled, but immediately batch-infers multiple decompressed small-batch data through the batch inference queue model of the batch inference node; when there is one batch inference node, multiple messages to be inferred are batch-inferred in the batch inference queue of one batch inference node in a single-node message queue mode; when there are multiple batch inference nodes, multiple messages to be inferred are distributed among multiple batch inference nodes for distributed concurrent processing of large-batch inference tasks; obtaining multiple small-batch data inference results; distributed concurrent processing of large-batch inference tasks by multiple messages to be inferred distributed among multiple batch inference nodes includes: querying the node inference task amounts and node inference processing speeds of multiple batch inference nodes; multiple batch inference nodes include: the first batch inference node, the second batch inference node, and the Nth batch inference node; respectively obtaining the first node inference task amount and the first node inference processing speed of the first batch inference node, the second node inference task amount and the second node inference processing speed of the second batch inference node, and the Nth node inference task amount and the Nth node inference processing speed of the Nth batch inference node; comparing the first node inference task amount, the second node inference task amount, and the Nth node inference task amount; sorting the first batch inference node, the second batch inference node, and the Nth batch inference node according to the size of the node inference task amount, the smaller the node inference task amount, the higher the sorting priority, to obtain the first priority order of the batch inference nodes; sorting the first batch inference node, the second batch inference node, and the Nth batch inference node according to the speed of the node inference processing, the faster the node inference processing speed, the higher the sorting priority, to obtain the second priority order of the batch inference nodes; according to the first priority order and the second priority order, when the first priority order and the second priority order are the same, the one with the highest first priority order and the highest second priority order is the batch inference node with the highest parallel priority, followed by the batch inference node with the second highest parallel priority until the batch inference node with the lowest parallel priority;According to the first priority order and the second priority order, when the first priority order and the second priority order are inconsistent, the one with the highest weighted average of the first priority order and the second priority order is the parallel highest priority batch inference node, followed by the parallel second highest priority batch inference node until the parallel lowest priority batch inference node, and the parallel priority batch inference sorting is obtained; the distribution of multiple messages to be inferred from more to less is sent to the parallel highest priority batch inference node, the parallel second highest priority batch inference node until the parallel lowest priority batch inference node; when the parallel priority batch inference sorting changes, the dynamic parallel priority batch inference sorting is obtained, and the large-scale inference task is adaptively and dynamically processed according to the distribution of multiple messages to be inferred.

[0062] In one embodiment, S300 includes:

[0063] S301, compressing the inference results of multiple small batch data to form multiple small batch data inference result messages;

[0064] S302, through the message queue mode, sending multiple small batch data inference result messages to the inference result queue for consumption by the concurrent data source adapter.

[0065] The principle and effect of the above technical solution are: compressing the inference results of multiple small batch data to form multiple small batch data inference result messages; through the message queue mode, sending multiple small batch data inference result messages to the inference result queue for consumption by the concurrent data source adapter; after the small batch inference results are compressed to form result messages, through the message queue mode, sending a small batch inference result data to the inference result queue for consumption by the data source adapter; completing all data pulling in the data pulling stage and processing the entire process in parallel; using common compression techniques to compress the data of the messages in the message queue to reduce the network transmission pressure.

[0066] In one embodiment, S400 includes:

[0067] S401, the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages and decompresses them one by one to obtain multiple decompressed small batch data inference results;

[0068] S402, saving the multiple decompressed small batch data inference results to the data source database one by one to achieve the streaming pulling and concurrent inference batch storage of the complete large-scale data source.

[0069] The principles and effects of the above technical solution are as follows: The concurrent data source adapter listens to the inference result queue, receives multiple small-batch data inference result messages and decompresses them one by one to obtain multiple decompressed small-batch data inference results; saves the multiple decompressed small-batch data inference results to the data source database one by one, realizing the streaming pull, concurrent inference, and batch storage of the complete large-scale data source; saves the data of the small-batch inference results; after the small-batch data is inferred, it can be immediately transmitted to the data source adapter through the message queue and stored immediately, without waiting to save all the data after inference; the entire inference process changes the traditional serial process of data pull - batch inference - data storage into a streaming concurrent process, greatly improving the operation efficiency of multiple links.

[0070] The present invention provides a streaming batch inference system based on a message queue, including:

[0071] A data source adapter streaming pull module, in the complete large-scale data source, pulls small-batch data through the data source adapter and performs multiple executions of streaming pull to split the complete large-scale data source into multiple small-batch data;

[0072] A queue model instant inference module, puts multiple small-batch data into the batch inference queue through the message queue middleware, and immediately performs batch inference in the message queue mode to obtain multiple small-batch data inference results;

[0073] A concurrent data source adaptation module, compresses multiple small-batch data inference results and sends them to the inference result queue in the message queue mode for consumption by the concurrent data source adapter;

[0074] An inference result listening and storage module, the concurrent data source adapter listens to the inference result queue and decompresses and saves them to the data source database one by one, realizing the streaming pull, concurrent inference, and batch storage of the complete large-scale data source.

[0075] The principles and effects of the above technical solution are as follows: The present invention provides a streaming batch inference system based on a message queue, including: a data source adapter streaming pull module, which, in a complete large batch of data sources, streams and pulls small batches of data through the data source adapter and performs multiple executions of streaming pulls to split the complete large batch of data sources into multiple small batches of data; a queue model instant inference module, which puts multiple small batches of data into a batch inference queue through a message queue middleware and instantaneously performs batch inference in the message queue mode to obtain multiple small batch data inference results; a concurrent data source adaptation module, which compresses the multiple small batch data inference results and sends them to an inference result queue through the message queue mode for consumption by the concurrent data source adapter; an inference result listening and warehousing module, where the concurrent data source adapter listens to the inference result queue and decompresses and saves each one to the data source database one by one, realizing the streaming pull, concurrent inference, and batch warehousing of the complete large batch of data sources; the data is pulled from the data source within a specified range through the data source adapter, and the data is pulled from the data source in a streaming manner. After the data source adapter pulls a part of the data, a small batch of data can be formed; the data source includes: a database or a file system; such a pull of a complete large batch of data can be split into a streaming pull of many small batches of data; after the small batch of data is compressed, an inference message can be formed, and through the message queue mode, each small batch of data is sent to the batch inference queue for consumption by the batch inference node to perform small batch batch inference; the batch inference node can be one or more, listens to the batch inference queue, receives the small batch inference message sent from the data source adapter, decompresses it to obtain the small batch of data, and performs batch inference using a specified model to obtain the small batch inference result. The inference result is compressed to form a result message, and through the message queue mode, a small batch inference result data is sent to the inference result queue for consumption by the data source adapter to save the small batch inference result; the data source adapter listens to the inference result queue, decompresses the message after receiving the small batch inference result to obtain the small batch inference result, and then saves the result; all data pulls are completed during the data pull phase, and the entire process is processed in parallel; the data source adapter can perform a streaming pull on a large batch of data and split it into small batches. The batch inference node does not need to wait for all data pulls to be completed before starting the inference; through the message queue mode, multiple batch inference nodes can perform distributed concurrent processing on the same large batch of inference tasks, improving the inference efficiency; the messages in the message queue use common compression technologies to compress the data, reducing the network transmission pressure; after the small batch of data is inferred, it can be immediately transmitted to the data source adapter through the message queue and immediately warehoused without waiting for all data to be inferred before unified saving; the entire inference process changes the traditional serial process of data pull - batch inference - data saving into a streaming concurrent process, greatly improving the operating efficiency of multiple links.

[0076] In one embodiment, the data source adapter streaming pull module includes:

[0077] A range setting adaptation pull unit that sets a data pull range in a complete large batch data source; and pulls a small batch of data through the data source adapter according to the data pull range;

[0078] A data source splitting unit that executes in a loop, and splits the complete large batch data source into multiple small batches of data by executing the data source adapter streaming pull multiple times.

[0079] The principle and effect of the above technical solution are as follows: The data source adapter streaming pull module includes: a range setting adaptation pull unit that sets a data pull range in a complete large batch data source; and pulls a small batch of data through the data source adapter according to the data pull range; a data source splitting unit that executes in a loop, and splits the complete large batch data source into multiple small batches of data by executing the data source adapter streaming pull multiple times; the data source includes: a database or a file system; splitting the complete large batch data source into multiple small batches of data by executing the data source adapter streaming pull multiple times includes: during the process of pulling a small batch of data, determining in real time whether the data in the complete large batch data source has been pulled, and obtaining data pull determination information; determining whether to end the data pull or continue to execute the data source adapter streaming pull according to the data pull determination information; determining whether to end the data pull or continue to execute the data source adapter streaming pull according to the data pull determination information includes: according to the data pull determination information, if it is determined that the data in the complete large batch data source has been pulled, then the data pull ends, and if it is determined that the data in the complete large batch data source has not been pulled, then continue to pull a small batch of data through the data source adapter until the data in the complete large batch data source has been pulled, and split the complete large batch data source into multiple small batches of data.

[0080] In one embodiment, the queue model instant inference module includes:

[0081] A data compression inference message unit that compresses multiple small batches of data one by one to form multiple messages to be inferred;

[0082] A message queue distribution concurrent inference unit that, through the message queue mode, performs batch inference in the batch inference queue of the batch inference node on multiple messages to be inferred through the batch inference queue model, and obtains multiple small batch data inference results;

[0083] The batch inference node includes one or more.

[0084] The principles and effects of the above technical solution are as follows: The queue model instant inference module includes: a data compression inference message unit that compresses multiple small-batch data one by one to form multiple messages to be inferred; a message queue distributed concurrent inference unit that, through the message queue mode, performs batch inference on multiple messages to be inferred in the batch inference queue of the batch inference node through the batch inference queue model to obtain multiple small-batch data inference results; the batch inference node includes one or more; performing batch inference on multiple messages to be inferred in the batch inference queue of the batch inference node through the message queue mode includes: sending multiple messages to be inferred to the batch inference queue of the batch inference node through the message queue mode; listening to the batch inference queue, receiving multiple messages to be inferred sent from the data source adapter, decompressing each of the multiple messages to be inferred to obtain multiple decompressed small-batch data for consumption by the batch inference node; the batch inference node does not wait for all the data in the complete large-batch data source to be pulled, and immediately performs batch inference on multiple decompressed small-batch data through the batch inference queue model of the batch inference node; when there is one batch inference node, multiple messages to be inferred are in the batch inference queue of one batch inference node, and single-node message queue mode batch inference is performed; when there are multiple batch inference nodes, multiple messages to be inferred are distributed among multiple batch inference nodes for distributed concurrent processing of large-batch inference tasks; obtaining multiple small-batch data inference results; distributing multiple messages to be inferred among multiple batch inference nodes for distributed concurrent processing of large-batch inference tasks includes: querying the node inference task amounts and node inference processing speeds of multiple batch inference nodes; the multiple batch inference nodes include: the first batch inference node, the second batch inference node, and the Nth batch inference node; respectively obtaining the first node inference task amount and the first node inference processing speed of the first batch inference node, the second node inference task amount and the second node inference processing speed of the second batch inference node, and the Nth node inference task amount and the Nth node inference processing speed of the Nth batch inference node; comparing the first node inference task amount, the second node inference task amount, and the Nth node inference task amount; sorting the first batch inference node, the second batch inference node, and the Nth batch inference node according to the size of the node inference task amount, with the smaller the node inference task amount, the higher the sorting priority, to obtain the first priority order of the batch inference nodes; sorting the first batch inference node, the second batch inference node, and the Nth batch inference node according to the speed of the node inference processing, with the faster the node inference processing speed, the higher the sorting priority, to obtain the second priority order of the batch inference nodes; according to the first priority order and the second priority order, when the first priority order and the second priority order are the same, the one with the highest first priority order and the highest second priority order is the batch inference node with the highest parallel priority, followed by the batch inference node with the second highest parallel priority until the batch inference node with the lowest parallel priority;According to the first priority order and the second priority order, when the first priority order and the second priority order are inconsistent, the one with the highest weighted average of the first priority order and the second priority order is the parallel highest priority batch inference node, followed by the parallel second highest priority batch inference node until the parallel lowest priority batch inference node, and the parallel priority batch inference sorting is obtained; distribute multiple messages to be inferred from most to least to the parallel highest priority batch inference node, the parallel second highest priority batch inference node until the parallel lowest priority batch inference node; when the parallel priority batch inference sorting changes, obtain the dynamic parallel priority batch inference sorting, and adaptively dynamically parallel priority batch inference sorting according to the distribution of multiple messages to be inferred, and perform distributed concurrent processing of large-scale inference tasks.

[0085] In one embodiment, the concurrent data source adaptation module includes:

[0086] The data inference result compression unit compresses multiple small batch data inference results to form multiple small batch data inference result messages;

[0087] The inference message queue sending unit sends multiple small batch data inference result messages to the inference result queue through the message queue mode for consumption by the concurrent data source adapter.

[0088] The principle and effect of the above technical solution are as follows: The concurrent data source adaptation module includes: The data inference result compression unit compresses multiple small batch data inference results to form multiple small batch data inference result messages; The inference message queue sending unit sends multiple small batch data inference result messages to the inference result queue through the message queue mode for consumption by the concurrent data source adapter; The small batch inference results are compressed to form result messages, and through the message queue mode, a small batch inference result data is sent to the inference result queue for consumption by the data source adapter; All data pulling is completed in the data pulling stage and the entire process is processed in parallel; The messages in the message queue use common compression technologies to compress the data, reducing the network transmission pressure.

[0089] In one embodiment, the inference result monitoring and warehousing module includes:

[0090] The inference queue monitoring and decompression unit, the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages and decompresses them one by one to obtain multiple decompressed small batch data inference results;

[0091] The inference result batch-by-batch warehousing unit saves multiple decompressed small batch data inference results to the data source database one by one, realizing the streaming pulling and concurrent inference batch-by-batch warehousing of the complete large-scale data source.

[0092] The principle and effect of the above technical solution are as follows: The inference result monitoring and warehousing module includes: an inference queue monitoring and decompression unit, a concurrent data source adapter monitors the inference result queue, receives multiple small-batch data inference result messages and decompresses them one by one to obtain multiple decompressed small-batch data inference results; an inference result batch-by-batch warehousing unit saves the multiple decompressed small-batch data inference results to the data source database one by one, realizing the streaming pull, concurrent inference, and batch-by-batch warehousing of the complete large-scale data source; saves the data of the small-batch inference results; after the small-batch data inference is completed, it can be immediately transmitted to the data source adapter through the message queue and warehoused immediately, without waiting to save them uniformly after all the data inferences are completed; the entire inference process changes the traditional serial process of data pull - batch inference - data saving into a streaming concurrent process, greatly improving the operation efficiency of multiple links.

[0093] Although the embodiments of the present invention have been disclosed as above, it is not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the field, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrated and described examples here.

Claims

1. A streaming batch inference method based on a message queue, characterized in that Including: S100, in a complete large - batch data source, streaming - pull small - batch data through a data source adapter and, after multiple executions of streaming - pull, split the complete large - batch data source to form multiple small - batch data; S200, for multiple small - batch data, put the multiple small - batch data into a batch inference queue through a message queue middleware, and immediately perform batch inference in the message queue mode to obtain multiple small - batch data inference results; S300, compress the multiple small - batch data inference results and send them to an inference result queue through the message queue mode for consumption by a concurrent data source adapter; S400, the concurrent data source adapter listens to the inference result queue, decompresses each one and saves them one by one to the data source database, realizing the streaming - pull concurrent inference and batch - by - batch storage of the complete large - batch data source.

2. The streaming batch inference method based on a message queue according to claim 1, wherein S100 includes: S101, set a data pull range in the complete large - batch data source; according to the data pull range, stream - pull small - batch data through a data source adapter; S102, through multiple executions of streaming - pull by the data source adapter, split the complete large - batch data source to form multiple small - batch data.

3. The streaming batch inference method based on a message queue according to claim 1, wherein S200 includes: S201, for multiple small - batch data, compress each small - batch data to form multiple messages to be inferred; S202, in the message queue mode, perform batch inference on the multiple messages to be inferred in the batch inference queue of the batch inference node through the batch inference queue model immediately to obtain multiple small - batch data inference results; The batch inference node includes one or more.

4. A streaming batch inference method based on a message queue according to claim 1, wherein S300 includes: S301, compress the multiple small - batch data inference results to form multiple small - batch data inference result messages; S302, through the message queue mode, send the multiple small - batch data inference result messages to the inference result queue for consumption by a concurrent data source adapter.

5. A streaming batch inference method based on a message queue according to claim 1, characterized in that, S400 includes: S401, the concurrent data source adapter listens to the inference result queue, receives multiple small - batch data inference result messages, decompresses each one to obtain multiple decompressed small - batch data inference results; S402, save the multiple decompressed small - batch data inference results one by one to the data source database, realizing the streaming - pull concurrent inference and batch - by - batch storage of the complete large - batch data source.

6. A streaming batch inference system based on a message queue, characterized in that, Including: A data source adapter streaming - pull module, in a complete large - batch data source, stream - pull small - batch data through a data source adapter and, after multiple executions of streaming - pull, split the complete large - batch data source to form multiple small - batch data; A queue model immediate inference module, for multiple small - batch data, put the multiple small - batch data into a batch inference queue through a message queue middleware, and immediately perform batch inference in the message queue mode to obtain multiple small - batch data inference results; A concurrent data source adaptation module, compress the multiple small - batch data inference results and send them to an inference result queue through the message queue mode for consumption by a concurrent data source adapter; The Inference Result Monitoring and Storing Module, where the concurrent data source adapter monitors the inference result queue, decompresses each one one by one, and saves them into the data source database one by one, to achieve the streaming pull of a complete large batch of data sources and the concurrent inference and batch storage in batches.

7. The streaming batch inference system based on a message queue according to claim 6, wherein The Data Source Adapter Streaming Pulling Module, including: The Range Setting and Adaptation Pulling Unit, which sets the data pulling range in the complete large batch of data sources; according to the data pulling range, it pulls a small batch of data through the data source adapter in a streaming manner; The Loop Execution Data Source Splitting Unit, which splits the complete large batch of data sources by executing the data source adapter streaming pull multiple times to form multiple small batches of data.

8. A streaming batch inference system based on a message queue according to claim 6, characterized in that, The Queue Model Instant Inference Module, including: The Data Compression Inference Message Unit, which compresses each small batch of data one by one to form multiple messages to be inferred; The Message Queue Distributed Concurrent Inference Unit, which, through the message queue mode, performs batch inference in the batch inference queue of the batch inference node for multiple messages to be inferred, and obtains multiple small batch data inference results; The batch inference node includes one or more.

9. The streaming batch inference system based on a message queue according to claim 6, wherein The Concurrent Data Source Adaptation Module, including: The Data Inference Result Compression Unit, which compresses multiple small batch data inference results to form multiple small batch data inference result messages; The Inference Message Queue Sending Unit, which sends multiple small batch data inference result messages into the inference result queue through the message queue mode for consumption by the concurrent data source adapter.

10. The streaming batch inference system based on a message queue according to claim 6, wherein The Inference Result Monitoring and Storing Module, including: The Inference Queue Monitoring and Decompression Unit, where the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages, decompresses each one one by one, and obtains multiple decompressed small batch data inference results; The Inference Result Batch-by-Batch Storing Unit, which saves multiple decompressed small batch data inference results into the data source database one by one, to achieve the streaming pull of a complete large batch of data sources and the concurrent inference and batch storage in batches.

Images