Streaming batch reasoning method and system based on message queue

Abstract

The invention provides a streaming batch reasoning method and system based on a message queue. Comprising the steps that in a complete large-batch data source, small-batch data are pulled in a streaming mode through a data source adapter, and the complete large-batch data source is segmented through multiple times of execution of streaming pulling, so that multiple pieces of small-batch data are formed; the multiple pieces of small-batch data are put into a batch reasoning queue through message queue middleware, batch reasoning in a message queue mode is conducted in time, and multiple pieces of small-batch data reasoning results are obtained; compressing a plurality of small-batch data reasoning results, and sending the compressed small-batch data reasoning results to a reasoning result queue in a message queue mode for consumption of a concurrent data source adapter; and the concurrent data source adapter monitors the reasoning result queues, decompresses the reasoning result queues one by one and stores the decompressed reasoning result queues into the data source database one by one, so that stream-type pulling and concurrent reasoning batch storage of complete large-batch data sources

Description

Technical Field

[0001] The present invention relates to the technical field of concurrent reasoning data processing of streaming pull message queues, and more specifically, to a streaming batch reasoning method and system based on a message queue. Background Art

[0002] At present, with the explosive growth of data scale and the rapid increase in the complexity of data sources, the speed, efficiency and technical requirements for data processing and information reasoning are getting higher and higher; specific problems include: how to pull small batch data from a large batch data source and split it into multiple small batches of data, how to instantly perform batch reasoning in the message queue mode in the batch reasoning queue of the batch reasoning node, how to adapt to concurrent data sources and compress multiple small batch data reasoning results and send them to the reasoning result queue for adaptive consumption, how to monitor the reasoning results and save them one by one to the data source database, and pull concurrent reasoning from the complete large batch data source into the warehouse in batches. These problems have yet to be solved; therefore, it is necessary to propose a streaming batch reasoning method and system based on message queues to at least partially solve the problems existing in the prior art. Summary of the invention

[0003] A series of simplified concepts are introduced in the content of the invention, which will be further described in detail in the specific implementation method section; the content of the invention of the present invention does not mean to attempt to limit the key features and essential technical features of the technical solution claimed for protection, nor does it mean to attempt to determine the scope of protection of the technical solution claimed for protection.

[0004] To at least partially solve the above problems, the present invention provides a streaming batch reasoning method based on a message queue, comprising:

[0005] S100, in a complete large-batch data source, streamingly pull small batches of data through a data source adapter and perform streaming pulls multiple times to split the complete large-batch data source into multiple small batches of data;

[0006] S200, putting multiple small batches of data into a batch reasoning queue through a message queue middleware, and immediately performing batch reasoning in a message queue mode to obtain multiple small batches of data reasoning results;

[0007] S300, compressing multiple small batch data inference results and sending them to the inference result queue through the message queue mode for consumption by the concurrent data source adapter;

[0008] S400, the concurrent data source adapter listens to the inference result queue and decompresses and saves them one by one to the data source database, so as to realize the streaming pull of complete large-scale data sources and concurrent inference and batch storage.

[0009] Preferably, S100 includes:

[0010] S101, setting a data pull range in a complete large-batch data source; and streamingly pulling small batches of data through a data source adapter according to the data pull range;

[0011] S102, by executing the data source adapter streaming pull multiple times, the complete large batch data source is split into multiple small batches of data.

[0012] Preferably, S200 includes:

[0013] S201, compressing multiple small batches of data one by one to form multiple messages to be inferred;

[0014] S202, using a message queue mode, in a batch inference queue of a batch inference node, batch inference in the message queue mode is performed in real time using a batch inference queue model to obtain multiple small batch data inference results;

[0015] A batch inference node includes one or more.

[0016] Preferably, S300 includes:

[0017] S301, compressing multiple small batch data inference results to form multiple small batch data inference result messages;

[0018] S302, through the message queue mode, sending multiple small batch data inference result messages to the inference result queue for consumption by the concurrent data source adapter.

[0019] Preferably, S400 includes:

[0020] S401, the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages, decompresses them one by one, and obtains multiple decompressed small batch data inference results;

[0021] S402, saving multiple decompressed small batch data inference results one by one to the data source database, so as to realize the streaming pull and concurrent inference of the complete large batch data source and store them in batches.

[0022] The present invention provides a streaming batch reasoning system based on a message queue, comprising:

[0023] The data source adapter streaming pull module, in the complete large-batch data source, streams and pulls small batches of data through the data source adapter and performs streaming pulls multiple times to split the complete large-batch data source into multiple small batches of data;

[0024] The queue model instant reasoning module puts multiple small batches of data into the batch reasoning queue through the message queue middleware, and performs batch reasoning in the message queue mode in real time to obtain multiple small batch data reasoning results;

[0025] The concurrent data source adapter module compresses multiple small batch data inference results and sends them to the inference result queue through the message queue mode for consumption by the concurrent data source adapter;

[0026] The inference result listening and warehousing module, the concurrent data source adapter listens to the inference result queue and decompresses them one by one and saves them to the data source database one by one, realizing the complete streaming pull of large batches of data sources and concurrent inference and warehousing in batches.

[0027] Preferably, the data source adapter streaming pull module includes:

[0028] The scope setting adapter pull unit sets the data pull range in the complete large-batch data source; according to the data pull range, small batches of data are pulled in a streaming manner through the data source adapter;

[0029] The data source splitting unit is executed in a loop. By executing the data source adapter streaming pull multiple times, the complete large batch data source is split into multiple small batches of data.

[0030] Preferably, the queue model instant reasoning module includes:

[0031] The data compression inference message unit compresses multiple small batches of data one by one to form multiple messages to be inferred;

[0032] The message queue distributed concurrent reasoning unit uses the message queue mode to perform batch reasoning in the batch reasoning queue of the batch reasoning node in real time through the batch reasoning queue model, and obtains multiple small batch data reasoning results;

[0033] A batch inference node includes one or more.

[0034] Preferably, the concurrent data source adapter module includes:

[0035] A data inference result compression unit compresses multiple small batch data inference results to form multiple small batch data inference result messages;

[0036] The inference message queue sending unit sends multiple small batch data inference result messages to the inference result queue through the message queue mode for consumption by the concurrent data source adapter.

[0037] Preferably, the inference result monitoring storage module includes:

[0038] The inference queue monitors the decompression unit, and the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages, decompresses them one by one, and obtains multiple decompressed small batch data inference results;

[0039] The inference results are stored in batches, and multiple decompressed small batch data inference results are saved one by one to the data source database, realizing the streaming pull and concurrent inference of complete large batch data sources and storing them in batches.

[0040] Compared with the prior art, the present invention has at least the following beneficial effects:

[0041] The present invention discloses a streaming batch reasoning method and system based on a message queue. In a complete large-batch data source, a data source adapter is used to stream small batch data and the streaming pull is performed multiple times to segment the complete large-batch data source to form multiple small batches of data. The multiple small batches of data are placed into a batch reasoning queue through a message queue middleware, and batch reasoning in a message queue mode is performed immediately to obtain multiple small batch data reasoning results. The multiple small batch data reasoning results are compressed and sent to a reasoning result queue through a message queue mode for consumption by a concurrent data source adapter. The concurrent data source adapter monitors the reasoning result queue and decompresses and saves them one by one to the data source. Database, to achieve complete large-scale data source streaming pull concurrent reasoning batch storage; pull the specified range of data from the data source through the data source adapter, the way to pull data from the data source is streaming, after the data source adapter pulls part of the data, it can form a small batch of data; data source includes: database or file system; such a complete large batch of data pull can be divided into many small batches of data streaming pull; after the small batch data is compressed, it can form an inference message, through the message queue mode, each small batch of data is sent to the batch inference queue for consumption by the batch inference node, and small batch batch inference is performed; batch inference nodes can be one or more , listen to the batch inference queue, receive the small batch inference message sent from the data source adapter, decompress it to get the small batch data, use the specified model to perform batch inference, and get the small batch inference result. The inference result is compressed to form a result message. Through the message queue mode, a small batch of inference result data is sent to the inference result queue for consumption by the data source adapter, and the small batch inference result is saved; the data source adapter listens to the inference result queue, decompresses the message after receiving the small batch inference result, gets the small batch inference result, and then saves the result; completes all data pulling in the data pulling stage and processes the entire process in parallel; the data source adapter can process large batches of data The data is pulled in a streaming manner and split into small batches. The batch inference node can start inference without waiting for all the data to be pulled. Through the message queue mode, multiple batch inference nodes can perform distributed concurrent processing on the same large batch inference task, thereby improving the inference efficiency. The messages in the message queue use common compression technology to compress the data, thereby reducing the pressure on network transmission. After the small batch data is inferred, it can be immediately transmitted to the data source adapter through the message queue and immediately stored in the warehouse, without waiting for all the data to be inferred before being saved uniformly. The entire inference process changes the traditional data pulling-batch inference-data storage serial process into a streaming concurrent process, greatly improving the operating efficiency of multiple links.

[0042] The present invention describes a message queue-based streaming batch reasoning method and system. Other advantages, objectives and features of the present invention will be partially reflected in the following description, and will also be partially understood by technicians in the field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 This is a diagram of an embodiment of a streaming batch reasoning method based on a message queue described in the present invention.

[0045] Figure 2 This is a diagram of an implementation example of a streaming batch reasoning system based on a message queue described in the present invention.

[0046] Figure 3 This is a diagram of another embodiment of a streaming batch reasoning method and system based on a message queue described in the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments so that those skilled in the art can implement it according to the instructions; as shown in the figure, the present invention provides a streaming batch reasoning method based on a message queue, including:

[0048] S100, in a complete large-batch data source, streamingly pull small batches of data through a data source adapter and perform streaming pulls multiple times to split the complete large-batch data source into multiple small batches of data;

[0049] S200, putting multiple small batches of data into a batch reasoning queue through a message queue middleware, and immediately performing batch reasoning in a message queue mode to obtain multiple small batches of data reasoning results;

[0050] S300, compressing multiple small batch data inference results and sending them to the inference result queue through the message queue mode for consumption by the concurrent data source adapter;

[0051] S400, the concurrent data source adapter listens to the inference result queue and decompresses and saves them one by one to the data source database, so as to realize the streaming pull of complete large-scale data sources and concurrent inference and batch storage.

[0052] The principle and effect of the above technical solution are as follows: The present invention provides a streaming batch reasoning method based on a message queue, including: in a complete large-batch data source, streamingly pulling small batches of data through a data source adapter and performing streaming pulling multiple times to divide the complete large-batch data source into multiple small batches of data; putting multiple small batches of data into a batch reasoning queue through a message queue middleware, and immediately performing batch reasoning in a message queue mode to obtain multiple small batch data reasoning results; compressing multiple small batch data reasoning results, and sending them to the reasoning result queue through a message queue mode for consumption by a concurrent data source adapter; the concurrent data source adapter monitors the reasoning result queue and performs Decompress them one by one and save them to the data source database one by one, so as to realize the streaming pull of complete large-batch data sources and concurrent reasoning and batch storage; pull the data in the specified range from the data source through the data source adapter, and the data source adapter can form a small batch of data after pulling part of the data; the data source includes: database or file system; such a complete large-batch data pull can be divided into streaming pull of many small batches of data; after the small batch data is compressed, it can form an inference message, and through the message queue mode, each small batch of data is sent to the batch reasoning queue for consumption by the batch reasoning node to perform small batch batch reasoning; the batch reasoning node can It can be one or more, listening to the batch reasoning queue, receiving the small batch reasoning message sent from the data source adapter, decompressing it to obtain the small batch data, using the specified model for batch reasoning, and obtaining the small batch reasoning results. The reasoning results are compressed to form result messages, and through the message queue mode, a small batch of reasoning result data is sent to the reasoning result queue for consumption by the data source adapter, and the small batch reasoning results are saved; the data source adapter listens to the reasoning result queue, decompresses the message after receiving the small batch reasoning result, obtains the small batch reasoning result, and then saves the result; completes all data pulling in the data pulling stage and processes the entire process in parallel; the data source adapter can Large batches of data are pulled in a streaming manner and split into small batches. Batch inference nodes can start inference without waiting for all data to be pulled. Through the message queue mode, multiple batch inference nodes can perform distributed concurrent processing on the same large batch inference task, improving inference efficiency. Messages in the message queue use common compression technology to compress data and reduce network transmission pressure. After the small batch data is inferred, it can be immediately transmitted to the data source adapter through the message queue and immediately stored in the warehouse, without waiting for all data to be inferred before being saved uniformly. The entire inference process changes the traditional data pulling-batch inference-data storage serial process into a streaming concurrent process, greatly improving the operating efficiency of multiple links.

[0053] In one embodiment, S100 includes:

[0054] S101, setting a data pull range in a complete large-batch data source; and streamingly pulling small batches of data through a data source adapter according to the data pull range;

[0055] S102, by executing the data source adapter streaming pull multiple times, the complete large batch data source is split into multiple small batches of data.

[0056] The principle and effect of the above technical solution are: setting a data pull range in a complete large-batch data source; according to the data pull range, streamingly pulling small batches of data through a data source adapter; by executing the data source adapter streaming pull multiple times, the complete large-batch data source is split to form multiple small batches of data; the data source includes: a database or a file system; by executing the data source adapter streaming pull multiple times, the complete large-batch data source is split to form multiple small batches of data including: in the process of executing the small-batch data pull, it is determined in real time whether the data in the complete large-batch data source has been pulled, and the data pull determination information is obtained. Information; according to the data pull judgment information, determine whether to end data pull or continue to execute the data source adapter streaming pull; according to the data pull judgment information, determine whether to end data pull or continue to execute the data source adapter streaming pull including: according to the data pull judgment information, if the data in the complete large batch data source has been pulled, it is determined to be yes, then the data pull is ended; if the data in the complete large batch data source has not been pulled, it is determined to be otherwise, and small batches of data are continued to be pulled through the data source adapter streaming until the data in the complete large batch data source is pulled. The complete large batch data source is split into multiple small batches of data.

[0057] In one embodiment, S200 includes:

[0058] S201, compressing multiple small batches of data one by one to form multiple messages to be inferred;

[0059] S202, using a message queue mode, in a batch inference queue of a batch inference node, batch inference in the message queue mode is performed in real time using a batch inference queue model to obtain multiple small batch data inference results;

[0060] A batch inference node includes one or more.

[0061] The principle and effect of the above technical solution are: compressing multiple small batches of data one by one to form multiple messages to be inferred; through the message queue mode, batch reasoning of the multiple messages to be inferred in the batch reasoning queue of the batch reasoning node is performed in real time through the batch reasoning queue model to obtain multiple small batch data reasoning results; the batch reasoning node includes one or more; through the message queue mode, batch reasoning of the multiple messages to be inferred in the batch reasoning queue of the batch reasoning node includes: sending multiple messages to be inferred to the batch reasoning queue of the batch reasoning node through the message queue mode; listening to the batch reasoning queue and receiving the messages sent from the data source adapter Multiple messages to be inferred are decompressed one by one to obtain multiple decompressed small batches of data for consumption by batch inference nodes; batch inference nodes do not wait for all data in the complete large batch data source to be pulled, and batch inference is performed on multiple decompressed small batches of data instantly through the batch inference queue model of the batch inference node; when there is one batch inference node, multiple messages to be inferred are in the batch inference queue of a batch inference node, and batch inference is performed in a single-node message queue mode; when there are multiple batch inference nodes, multiple messages to be inferred are distributed in multiple batch inference nodes for distributed concurrent processing of large batch inference tasks; multiple small batch data inference results are obtained; multiple messages to be inferred are distributed in multiple The batch reasoning nodes perform distributed concurrent processing of large batch reasoning tasks, including: querying the node reasoning task volume and node reasoning processing speed of multiple batch reasoning nodes; the multiple batch reasoning nodes include: a first batch reasoning node, a second batch reasoning node, and an Nth batch reasoning node; respectively obtaining the first node reasoning task volume and the first node reasoning processing speed of the first batch reasoning node, the second node reasoning task volume and the second node reasoning processing speed of the second batch reasoning node, and the Nth node reasoning task volume and the Nth node reasoning processing speed of the Nth batch reasoning node; comparing the first node reasoning task volume, the second node reasoning task volume, and the Nth node reasoning task volume; performing the first batch reasoning according to the size of the node reasoning task volume The first batch inference node, the second batch inference node and the Nth batch inference node are sorted. The smaller the node inference task volume, the higher the sorting priority, and the first priority order of the batch inference node is obtained; the first batch inference node, the second batch inference node and the Nth batch inference node are sorted according to the speed of the node inference processing. The faster the node inference processing speed, the higher the sorting priority, and the second priority order of the batch inference node is obtained; according to the first priority order and the second priority order, when the first priority order and the second priority order are the same, the parallel highest priority batch inference node is the one with the highest first priority order and the highest second priority order, followed by the parallel second highest priority batch inference node until the parallel lowest priority batch inference node;According to the first priority order and the second priority order, when the first priority order and the second priority order are inconsistent, the highest weighted average of the first priority order and the second priority order is the parallel highest priority batch inference node, followed by the parallel second highest priority batch inference node until the parallel lowest priority batch inference node, and the parallel priority batch inference order is obtained; the multiple messages to be inferred are distributed from most to least and sent to the parallel highest priority batch inference node, the parallel second highest priority batch inference node until the parallel lowest priority batch inference node; when the parallel priority batch inference order changes, the dynamic parallel priority batch inference order is obtained, and the dynamic parallel priority batch inference order is adaptively distributed according to the distribution of multiple messages to be inferred, and distributed concurrent processing of large batch inference tasks is performed. ;

[0062] In one embodiment, S300 includes:

[0063] S301, compressing multiple small batch data inference results to form multiple small batch data inference result messages;

[0064] S302, through the message queue mode, sending multiple small batch data inference result messages to the inference result queue for consumption by the concurrent data source adapter.

[0065] The principle and effect of the above technical solution are: compress multiple small batch data inference results to form multiple small batch data inference result messages; send multiple small batch data inference result messages to the inference result queue through the message queue mode for consumption by the concurrent data source adapter; the small batch inference results are compressed to form result messages, and a small batch of inference result data is sent to the inference result queue through the message queue mode for consumption by the data source adapter; complete all data pulling in the data pulling stage and perform the entire process processing in parallel; the messages in the message queue use common compression technology to compress the data to reduce network transmission pressure.

[0066] In one embodiment, S400 includes:

[0067] S401, the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages, decompresses them one by one, and obtains multiple decompressed small batch data inference results;

[0068] S402, saving multiple decompressed small batch data inference results one by one to the data source database, so as to realize the streaming pull and concurrent inference of the complete large batch data source and store them in batches.

[0069] The principle and effect of the above technical solution are as follows: the concurrent data source adapter listens to the inference result queue, receives multiple small batch data inference result messages, decompresses them one by one, and obtains multiple decompressed small batch data inference results; saves multiple decompressed small batch data inference results one by one to the data source database, realizes the streaming pull and concurrent inference of the complete large batch data source and puts them into the warehouse in batches; saves the small batch inference results; after the small batch data is inferred, it can be immediately transmitted to the data source adapter through the message queue and put into the warehouse immediately, without waiting for all the data to be inferred before saving them uniformly; the entire inference process changes the traditional data pull-batch inference-data storage serial process into a streaming concurrent process, which greatly improves the operating efficiency of multiple links.

[0070] The present invention provides a streaming batch reasoning system based on a message queue, comprising:

[0071] The data source adapter streaming pull module, in the complete large-batch data source, streams and pulls small batches of data through the data source adapter and performs streaming pulls multiple times to split the complete large-batch data source into multiple small batches of data;

[0072] The queue model instant reasoning module puts multiple small batches of data into the batch reasoning queue through the message queue middleware, and performs batch reasoning in the message queue mode in real time to obtain multiple small batch data reasoning results;

[0073] The concurrent data source adapter module compresses multiple small batch data inference results and sends them to the inference result queue through the message queue mode for consumption by the concurrent data source adapter;

[0074] The inference result listening and warehousing module, the concurrent data source adapter listens to the inference result queue and decompresses them one by one and saves them to the data source database one by one, realizing the complete streaming pull of large batches of data sources and concurrent inference and warehousing in batches.

[0075] The principle and effect of the above technical solution are as follows: the present invention provides a streaming batch reasoning system based on message queue, including: a data source adapter streaming pull module, in a complete large batch data source, streaming pulls small batch data through the data source adapter and performs streaming pull multiple times, the complete large batch data source is segmented to form multiple small batches of data; a queue model instant reasoning module, multiple small batches of data are put into the batch reasoning queue through the message queue middleware, and batch reasoning in the message queue mode is performed instantly to obtain multiple small batch data reasoning results; a concurrent data source adaptation module, compresses multiple small batch data reasoning results, and sends them to the reasoning result queue through the message queue mode for concurrent data source adaptation. The inference result listening module is used for storage. The concurrent data source adapter listens to the inference result queue and decompresses and saves them one by one to the data source database, so as to realize the streaming pull of complete large-scale data sources and concurrent inference and storage in batches. The data source adapter pulls data in a specified range from the data source. The data source adapter pulls data from the data source in a streaming manner. After pulling part of the data, a small batch of data can be formed. The data source includes: a database or a file system. Such a complete large-scale data pull can be divided into streaming pulls of many small batches of data. After the small batch data is compressed, an inference message can be formed. Through the message queue mode, each small batch of data is sent to the batch inference queue for batch inference nodes. Point consumption, small batch batch reasoning; batch reasoning nodes, which can be one or more, listen to the batch reasoning queue, receive the small batch reasoning message sent from the data source adapter, decompress it to get the small batch data, use the specified model to perform batch reasoning, and get the small batch reasoning result. The reasoning result is compressed to form a result message. Through the message queue mode, a small batch reasoning result data is sent to the reasoning result queue for consumption by the data source adapter, and the small batch reasoning result is saved; the data source adapter listens to the reasoning result queue, decompresses the message after receiving the small batch reasoning result, gets the small batch reasoning result, and then saves the result; complete all data pulling in the data pulling stage and carry out the whole process in parallel Processing; the data source adapter can stream and pull large batches of data and split them into small batches. The batch inference node can start inference without waiting for all the data to be pulled. Through the message queue mode, multiple batch inference nodes can perform distributed concurrent processing on the same large batch of inference tasks to improve inference efficiency. The messages in the message queue use common compression technology to compress the data to reduce network transmission pressure. After the small batch data is inferred, it can be immediately transmitted to the data source adapter through the message queue and stored immediately, without waiting for all the data to be inferred before being saved uniformly. The entire inference process changes the traditional data pulling-batch inference-data storage serial process into a streaming concurrent process, greatly improving the operating efficiency of multiple links.

[0076] In one embodiment, a data source adapter streaming pull module includes:

[0077] The scope setting adapter pull unit sets the data pull range in the complete large-batch data source; according to the data pull range, small batches of data are pulled in a streaming manner through the data source adapter;

[0078] The data source splitting unit is executed in a loop. By executing the data source adapter streaming pull multiple times, the complete large batch data source is split into multiple small batches of data.

[0079] The principle and effect of the above technical solution are as follows: the data source adapter streaming pull module includes: a range setting adaptation pull unit, which sets the data pull range in the complete large-batch data source; according to the data pull range, stream-pull small batches of data through the data source adapter; cyclically execute the data source splitting unit, and split the complete large-batch data source into multiple small batches of data by executing the data source adapter streaming pull multiple times; the data source includes: a database or a file system; by executing the data source adapter streaming pull multiple times, the complete large-batch data source is split into multiple small batches of data, including: in the process of executing the small-batch data pull, the complete large-batch data is judged in real time According to whether the data has been pulled from the source, data pulling judgment information is obtained; according to the data pulling judgment information, it is determined whether to end data pulling or continue to execute the data source adapter streaming pulling; according to the data pulling judgment information, it is determined whether to end data pulling or continue to execute the data source adapter streaming pulling. The method includes: according to the data pulling judgment information, if the data in the complete large-batch data source has been pulled, it is determined that yes, then the data pulling is ended; if the data in the complete large-batch data source has not been pulled, it is determined that otherwise, small batches of data are continued to be pulled through the data source adapter streaming until the data in the complete large-batch data source is pulled. The complete large-batch data source is split to form multiple small batches of data.

[0080] In one embodiment, the queue model instant reasoning module includes:

[0081] The data compression inference message unit compresses multiple small batches of data one by one to form multiple messages to be inferred;

[0082] The message queue distributed concurrent reasoning unit uses the message queue mode to perform batch reasoning in the batch reasoning queue of the batch reasoning node in real time through the batch reasoning queue model, and obtains multiple small batch data reasoning results;

[0083] A batch inference node includes one or more.

[0084] The principle and effect of the above technical solution are as follows: the queue model instant reasoning module includes: a data compression reasoning message unit, which compresses multiple small batches of data one by one to form multiple messages to be reasoned; a message queue distributed concurrent reasoning unit, which uses a message queue mode to perform batch reasoning of the message queue mode in real time in the batch reasoning queue of the batch reasoning node through the batch reasoning queue model to obtain multiple small batch data reasoning results; the batch reasoning node includes one or more; through the message queue mode, the batch reasoning of the message queue mode in the batch reasoning queue of the batch reasoning node includes: sending multiple messages to be reasoned to the batch reasoning node through the message queue mode The batch inference queue is used to listen to the batch inference queue, receive multiple messages to be inferred from the data source adapter, and decompress the multiple messages to be inferred one by one to obtain multiple decompressed small batches of data for consumption by the batch inference node. The batch inference node does not wait for all the data in the complete large batch data source to be pulled. Through the batch inference queue model of the batch inference node, batch inference is performed on multiple decompressed small batches of data immediately. When there is one batch inference node, multiple messages to be inferred are in the batch inference queue of a batch inference node, and batch inference in the single-node message queue mode is performed. When there are multiple batch inference nodes, multiple messages to be inferred are distributed in multiple batch inference nodes for distributed concurrent processing of large batch inference tasks. Multiple small batches are obtained. Data reasoning results; multiple messages to be reasoned are distributed in multiple batch reasoning nodes for distributed concurrent processing of large batch reasoning tasks, including: querying the node reasoning task volume and node reasoning processing speed of multiple batch reasoning nodes; multiple batch reasoning nodes include: a first batch reasoning node, a second batch reasoning node and an Nth batch reasoning node; respectively obtaining the first node reasoning task volume and the first node reasoning processing speed of the first batch reasoning node, the second node reasoning task volume and the second node reasoning processing speed of the second batch reasoning node, and the Nth node reasoning task volume and the Nth node reasoning processing speed of the Nth batch reasoning node; comparing the first node reasoning task volume, the second node reasoning task volume and the Nth node reasoning task volume; and sorting the nodes according to the node reasoning task volume. The first batch reasoning node, the second batch reasoning node and the Nth batch reasoning node are sorted according to the speed of node reasoning processing. The smaller the node reasoning task volume, the higher the sorting priority, and the first priority order of the batch reasoning nodes is obtained; the first batch reasoning node, the second batch reasoning node and the Nth batch reasoning node are sorted according to the speed of node reasoning processing. The faster the node reasoning processing speed, the higher the sorting priority is, and the second priority order of the batch reasoning nodes is obtained; according to the first priority order and the second priority order, when the first priority order and the second priority order are the same, the parallel highest priority batch reasoning node is the one with the highest first priority order and the highest second priority order, followed by the parallel second highest priority batch reasoning node until the parallel lowest priority batch reasoning node;According to the first priority order and the second priority order, when the first priority order and the second priority order are inconsistent, the highest weighted average of the first priority order and the second priority order is the parallel highest priority batch inference node, followed by the parallel second highest priority batch inference node until the parallel lowest priority batch inference node, and the parallel priority batch inference order is obtained; the multiple messages to be inferred are distributed from most to least and sent to the parallel highest priority batch inference node, the parallel second highest priority batch inference node until the parallel lowest priority batch inference node; when the parallel priority batch inference order changes, the dynamic parallel priority batch inference order is obtained, and the dynamic parallel priority batch inference order is adaptively distributed according to the distribution of multiple messages to be inferred, and distributed concurrent processing of large batch inference tasks is performed. ;

[0085] In one embodiment, the concurrent data source adapter module includes:

[0086] A data inference result compression unit compresses multiple small batch data inference results to form multiple small batch data inference result messages;

[0087] The inference message queue sending unit sends multiple small batch data inference result messages to the inference result queue through the message queue mode for consumption by the concurrent data source adapter.

[0088] The principle and effect of the above technical solution are: the concurrent data source adapter module includes: a data inference result compression unit, which compresses multiple small batch data inference results to form multiple small batch data inference result messages; an inference message queue sending unit, which sends multiple small batch data inference result messages to the inference result queue through the message queue mode for consumption by the concurrent data source adapter; the small batch inference results are compressed to form a result message, and a small batch of inference result data is sent to the inference result queue through the message queue mode for consumption by the data source adapter; all data pulling is completed in the data pulling stage and the entire process is processed in parallel; the messages in the message queue use common compression technology to compress the data to reduce network transmission pressure.

[0089] In one embodiment, the inference result monitoring storage module includes:

[0090] The inference queue monitors the decompression unit, and the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages, decompresses them one by one, and obtains multiple decompressed small batch data inference results;

[0091] The inference results are stored in batches, and multiple decompressed small batch data inference results are saved one by one to the data source database, realizing the streaming pull and concurrent inference of complete large batch data sources and storing them in batches.

[0092] The principle and effect of the above technical solution are: an inference result monitoring and warehousing module, including: an inference queue monitoring and decompression unit, a concurrent data source adapter monitoring the inference result queue, receiving multiple small batch data inference result messages and decompressing them one by one, and obtaining multiple decompressed small batch data inference results; an inference result batch warehousing unit, saving multiple decompressed small batch data inference results one by one to the data source database, realizing the streaming pull and concurrent inference of a complete large batch data source and warehousing in batches; saving the small batch inference results; after the small batch data is inferred, it can be immediately transmitted to the data source adapter through the message queue and immediately stored, without waiting for all the data to be inferred before saving them uniformly; the entire inference process changes the traditional data pull-batch inference-data storage serial process into a streaming concurrent process, greatly improving the operating efficiency of multiple links.

[0093] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They are fully applicable to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A streaming batch reasoning method based on a message queue, characterized in that: include: S100, in a complete large-batch data source, streamingly pull small batches of data through a data source adapter and perform streaming pulls multiple times to split the complete large-batch data source into multiple small batches of data; S200, putting multiple small batches of data into a batch reasoning queue through a message queue middleware, and immediately performing batch reasoning in a message queue mode to obtain multiple small batches of data reasoning results; S300, compressing multiple small batch data inference results and sending them to the inference result queue through the message queue mode for consumption by the concurrent data source adapter; S400, the concurrent data source adapter listens to the inference result queue and decompresses and saves them one by one to the data source database, so as to realize the streaming pull of complete large-scale data sources and concurrent inference and batch storage.

2. According to the message queue-based streaming batch reasoning method of claim 1, it is characterized in that: S100 includes: S101, setting a data pull range in a complete large-batch data source; and streamingly pulling small batches of data through a data source adapter according to the data pull range; S102, by executing the data source adapter streaming pull multiple times, the complete large batch data source is split into multiple small batches of data.

3. According to the message queue-based streaming batch reasoning method of claim 1, it is characterized in that: S200 includes: S201, compressing multiple small batches of data one by one to form multiple messages to be inferred; S202, using a message queue mode, in a batch inference queue of a batch inference node, batch inference in the message queue mode is performed in real time using a batch inference queue model to obtain multiple small batch data inference results; A batch inference node includes one or more.

4. According to the message queue-based streaming batch reasoning method of claim 1, it is characterized in that: S300 includes: S301, compressing multiple small batch data inference results to form multiple small batch data inference result messages; S302, through the message queue mode, sending multiple small batch data inference result messages to the inference result queue for consumption by the concurrent data source adapter.

5. According to the message queue-based streaming batch reasoning method of claim 1, it is characterized in that: S400 includes: S401, the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages, decompresses them one by one, and obtains multiple decompressed small batch data inference results; S402, saving multiple decompressed small batch data inference results one by one to the data source database, so as to realize the streaming pull and concurrent inference of the complete large batch data source and store them in batches.

6. A streaming batch reasoning system based on a message queue, characterized in that: include: The data source adapter streaming pull module, in the complete large-batch data source, streams and pulls small batches of data through the data source adapter and performs streaming pulls multiple times to split the complete large-batch data source into multiple small batches of data; The queue model instant reasoning module puts multiple small batches of data into the batch reasoning queue through the message queue middleware, and performs batch reasoning in the message queue mode in real time to obtain multiple small batch data reasoning results; The concurrent data source adapter module compresses multiple small batch data inference results and sends them to the inference result queue through the message queue mode for consumption by the concurrent data source adapter; The inference result listening and warehousing module, the concurrent data source adapter listens to the inference result queue and decompresses them one by one and saves them to the data source database one by one, realizing the complete streaming pull of large batches of data sources and concurrent inference and warehousing in batches.

7. A streaming batch reasoning system based on a message queue according to claim 6, characterized in that: Data source adapter streaming pull module, including: The scope setting adapter pull unit sets the data pull range in the complete large-batch data source; according to the data pull range, small batches of data are pulled in a streaming manner through the data source adapter; The data source splitting unit is executed in a loop. By executing the data source adapter streaming pull multiple times, the complete large batch data source is split into multiple small batches of data.

8. The streaming batch reasoning system based on message queue according to claim 6 is characterized in that: Queue model instant reasoning module, including: The data compression inference message unit compresses multiple small batches of data one by one to form multiple messages to be inferred; The message queue distributed concurrent reasoning unit uses the message queue mode to perform batch reasoning in the batch reasoning queue of the batch reasoning node in real time through the batch reasoning queue model, and obtains multiple small batch data reasoning results; A batch inference node includes one or more.

9. A streaming batch reasoning system based on a message queue according to claim 6, characterized in that: Concurrent data source adapter module, including: A data inference result compression unit compresses multiple small batch data inference results to form multiple small batch data inference result messages; The inference message queue sending unit sends multiple small batch data inference result messages to the inference result queue through the message queue mode for consumption by the concurrent data source adapter.

10. A streaming batch reasoning system based on a message queue according to claim 6, characterized in that: The inference result monitoring and storage module includes: The inference queue monitors the decompression unit, and the concurrent data source adapter monitors the inference result queue, receives multiple small batch data inference result messages, decompresses them one by one, and obtains multiple decompressed small batch data inference results; The inference results are stored in batches, and multiple decompressed small batch data inference results are saved one by one to the data source database, realizing the streaming pull and concurrent inference of complete large batch data sources and storing them in batches.

Images