Data processing system, method for managing the order of data processing, and program

Abstract

To maintain time-series order of data input to a sequential microservice in a system in which a plurality of microservices is connected in parallel to the sequential microservice.SOLUTION: A data processing system includes an order manager. The order manager passes, to a sequential microservice, data corresponding to a minimum timestamp message having the smallest timestamp in a message queue containing a predetermined number of messages consumed from a plurality of microservices. The order manager then continues to consume new messages one by one from the plurality of microservices until a message with a timestamp greater than the minimum timestamp message is obtained, and adds the message with the greater timestamp to the message queue when the message with the greater timestamp is obtained.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This disclosure relates to a data processing system, a method for managing the order of data processing, and a program.

Background Art

[0002] Japanese Patent No. 7031424 discloses a prior art related to a system for distributing data processing by microservices. In this prior art, messages are communicated between multiple microservices, and the messages received by each microservice are stored in a queue in the order of arrival, and predetermined message processing is performed in the order of storage. Also, in this prior art, by comparing timestamps between multiple received messages, a message that has arrived late is detected. And if there is a message that has arrived late, the message processing state is rolled back to the state before the message that has arrived late, and message processing is performed again in the correct order of arrival.

[0003] In the above prior art, even if there is a delay in the arrival of messages, each of the multiple microservices can perform message processing in the correct order of arrival. However, in the prior art, a system in which multiple microservices are connected in parallel to a sequential microservice that sequentially processes data arranged in time series has not been sufficiently studied.

[0004] In addition, as a document showing the technical level of the technical field related to this disclosure, Patent No. 7126712 can be exemplified in addition to Japanese Patent No. 7031424.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

[0006] This disclosure has been made in view of the above-mentioned issues. One purpose of this disclosure is to maintain the chronological order of data input to a sequential microservice in a system in which multiple microservices are connected in parallel to a sequential microservice. [Means for solving the problem]

[0007] This disclosure provides a data processing system for achieving the above objectives. The data processing system of this disclosure comprises a sequential microservice that processes data arranged in a time series sequentially, a plurality of microservices connected in parallel to the sequential microservice, and an order manager. The order manager has the function of managing the order in which the sequential microservice processes data processed in parallel by the plurality of microservices. Specifically, the order manager is configured to repeatedly execute the following first, second, and third processes. The first process is to pass to the sequential microservice data corresponding to the minimum timestamp message, which has the smallest timestamp among a message queue containing a predetermined number of messages consumed from the plurality of microservices. The second process is to continue consuming new messages one by one from the plurality of microservices until a message with a timestamp larger than the minimum timestamp message is obtained. The third process is to add the message with the larger timestamp to the message queue when a message with a larger timestamp is obtained.

[0008] The data processing system of this disclosure may further include a message broker and a database. The message broker has the function of mediating message metadata between multiple microservices and sequential microservices, and the database stores the payload corresponding to each message. The order manager may be configured to retrieve the payload corresponding to the metadata consumed from the message broker from the database.

[0009] This disclosure provides a method for achieving the above objective. The method of this disclosure is a method for managing the order in which a sequential microservice processes data processed in parallel by multiple microservices, and comprises the following three repeatedly performed steps: The first step is to pass to the sequential microservice data corresponding to the minimum timestamp message, which has the smallest timestamp among a message queue containing a predetermined number of messages consumed from the multiple microservices. The second step is to continue consuming new messages one by one from the multiple microservices until a message with a timestamp greater than the minimum timestamp message is obtained. The third step is to add the message with the larger timestamp to the message queue when a message with a larger timestamp is obtained.

[0010] This disclosure provides a program for achieving the above objectives. The program of this disclosure is a program that includes a plurality of instructions executed by at least one processor. These plurality of instructions are configured to cause at least one processor to function as a sequential microservice that processes data arranged in a time series sequentially, a plurality of microservices connected in parallel to the sequential microservice, and an order manager that manages the order in which the sequential microservice processes the data processed in parallel by the plurality of microservices. Furthermore, these plurality of instructions are configured to cause the order manager to repeatedly execute the first, second, and third processes described above. The program of this disclosure may be recorded on a computer-readable recording medium.

[0011] Furthermore, this disclosure provides another data processing system with a different configuration from the data processing system described above. This other data processing system comprises at least one message producer, a plurality of message consumers, a message broker, and a database. The message broker has the function of mediating message metadata between at least one message producer and the plurality of message consumers, and the database stores the payload corresponding to each message. Each of the plurality of message consumers is configured to retrieve the payload corresponding to the metadata it consumed from the message broker from the database. [Effects of the Invention]

[0012] According to the technology disclosed herein, if a newly consumed message from multiple microservices in a parallel relationship is older than a message that has already been processed, the older message is discarded without being passed to the sequential microservice. Then, the data corresponding to the message with the smallest timestamp among a predetermined number of messages is passed to the sequential microservice. This makes it possible to maintain the chronological order of the data input to the sequential microservice while processing data in parallel across multiple microservices, thereby improving the data processing speed in the system. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This diagram illustrates the processes performed by the order manager. [Figure 3] This figure shows an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This figure shows an example of the configuration of a data processing system according to the third embodiment. [Figure 5] This figure shows an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This figure shows a comparative example to the third embodiment. [Modes for carrying out the invention]

[0014] 1. First Embodiment The data processing system disclosed herein is a system that processes data in a distributed manner using microservices. Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment. The data processing system 10 comprises a plurality of regular microservices 11 and at least one sequential microservice 12. In the example shown in Figure 1, one sequential microservice 12 is provided for three regular microservices 11.

[0015] Regular microservice 11 is a microservice that is not sequential microservice 12. While sequential microservice 12 is a microservice that sequentially processes data arranged in time series, regular microservice 11 does not require the input data to be in time series order. Each regular microservice 11 has differences in service content.

[0016] Regular microservice 11 is connected in parallel to sequential microservice 12. This means that there is no dependency relationship between regular microservices 11, and each regular microservice 11 can process data in parallel. Each regular microservice 11 independently sends a message to sequential microservice 12. Also, the communication between each regular microservice 11 and sequential microservice 12 is asynchronous.

[0017] Since each regular microservice 11 processes data in parallel and is asynchronous with sequential microservice 12, the order of the data reaching sequential microservice 12 is not necessarily the order in which it should be processed. For this reason, data processing system 10 further includes an order manager 13. Order manager 13 has a function of managing the processing order of data processed in parallel by regular microservice 11 in sequential microservice 12.

[0018] As shown in FIG. 1, order manager 13 can be configured as part of sequential microservice 12, or can be configured as a component separate from sequential microservice 12. Order manager 13 is provided between regular microservice 11 and sequential microservice 12, and receives the message sent from regular microservice 11 to sequential microservice 12.

[0019] Figure 2 shows a specific example of the process performed by the order manager 13. The order manager 13 is a message queue that can store a predetermined number of messages consumed from the regular microservice 11. In the example shown in Figure 2, a total of 10 messages are stored in the order manager 13. Note that the messages are denoted as msg. Also, in Figure 2, TS(x) attached to each message represents a timestamp, and the smaller the value of x, the older the message. Hereinafter, the process performed by the order manager 13 will be described in the order of steps.

[0020] In step A, the order manager 13 sorts the messages stored in the message queue in ascending order of the timestamp. In the order of the messages in step A, the smallest timestamp is TS(4), and the message having the timestamp TS(4) is the minimum timestamp message.

[0021] Next, in step B, the order manager 13 processes the minimum timestamp message in the message queue and passes the data corresponding to the minimum timestamp message to the sequential microservice 12. Thereby, the message having the timestamp TS(4) is processed.

[0022] Next, in step C, the order manager 13 consumes one new message from the regular microservice 11. Which regular microservice 11 to consume the message from depends on the message sending timing of each regular microservice 11. In the example shown in Figure 2, the message having the timestamp TS(3) is newly consumed.

[0023] Next, in step D, the order manager 13 compares the timestamp of the message newly consumed in step C with the timestamp of the minimum timestamp message processed in step B. Depending on the result of this comparison, the processing performed by the order manager 13 in the next step will differ.

[0024] In the case of step D, the timestamp TS(3) of the newly consumed message is smaller than the timestamp TS(4) of the last processed message with the smallest timestamp. In other words, the newly consumed message is older than the message that was processed previously. In this case, processing the data of the newly consumed message would mean that data that should be processed chronologically is being processed in reverse order, which could cause problems for the service.

[0025] Therefore, in step E, the order manager 13 discards the message that was newly consumed in step C, i.e., the message with timestamp TS(3), without adding it to the message queue.

[0026] Next, in step F, the order manager 13 consumes another new message from the regular microservice 11. In the example shown in Figure 2, a new message with timestamp TS(11) is consumed.

[0027] Next, in step G, the order manager 13 compares the timestamp of the message newly consumed in step F with the timestamp of the previously processed minimum timestamp message. The previously processed minimum timestamp message is the message with timestamp TS(4) that was processed in step B.

[0028] The order manager 13 continues to consume new messages one by one from the regular microservice 11 until it obtains a message with a timestamp greater than the minimum timestamp message. In the case of step G, the timestamp TS(11) of the newly consumed message is greater than the timestamp TS(4) of the previously processed minimum timestamp message. In other words, the newly consumed message is newer than the previously processed message. In this case, by processing the data of the newly consumed message, the data that should be processed chronologically can be processed chronologically.

[0029] Therefore, in step H, the order manager 13 adds the message newly consumed in step C, that is, the message with timestamp TS(11), to the message queue. Then, the order manager 13 sorts the messages stored in the message queue in order of timestamp.

[0030] By managing the processing order in the manner described above, the order manager 13 can maintain the chronological order of data input to the sequential microservice 12 while processing data in parallel across multiple regular microservices 11. As a result, the data processing system 10 can achieve high data processing speed.

[0031] 2. Second Embodiment Figure 3 shows an example of the configuration of the data processing system 20 according to the second embodiment. In Figure 3, elements with the same reference numerals as the data processing system 10 according to the first embodiment are elements that have the same function. The data processing system 20 is a system that uses a message broker 14 for the exchange of multiple messages. In addition to multiple regular microservices 11, sequential microservices 12, and an order manager 13, the data processing system 20 includes a message broker 14 and a database 15.

[0032] Messages output from each regular microservice 11 include metadata and a payload. The message broker 14 mediates the message metadata between the regular microservice 11 and the sequential microservice 12. The message broker 14 has multiple partitions. Metadata sent from each regular microservice 11 is stored in the corresponding partition. Meanwhile, the message payload is stored in the database 15. An example of a payload is image data.

[0033] In the data processing system 20, the order manager 13 consumes metadata one by one from the message broker 14. The processing of the consumed metadata is the same as the processing of messages in the first embodiment. That is, the order manager 13 processes the metadata with the smallest timestamp among the predetermined number of metadata consumed from the message broker 14. Then, each time one metadata in the message queue is processed, the order manager 13 consumes one new metadata from the message broker 14 and compares its timestamp with the previously processed metadata. If the timestamp of the newly consumed metadata is less than or equal to that of the previously processed metadata, the order manager 13 discards the newly consumed metadata and consumes another new metadata from the message broker 14. On the other hand, if the timestamp of the newly consumed metadata is greater than that of the previously processed metadata, the order manager 13 processes the newly consumed metadata.

[0034] When processing metadata, the order manager 13 retrieves the payload corresponding to that metadata from the database 15. Then, it passes the retrieved payload along with the metadata to the sequential microservice 12. This allows the sequential microservice 12 to process data (payloads) that should be processed chronologically in chronological order.

[0035] With the data processing system 20 configured as described above, similar to the first embodiment, it is possible to maintain the chronological order of data input to the sequential microservice 12 while processing data in parallel with multiple regular microservices 11. Furthermore, since the data passed via the message broker 14 consists only of metadata, the waiting time required for data transfer can be reduced compared to the case where the entire message is transferred, and consequently, the overall working time of the data processing system 20 can be reduced.

[0036] 3. Third Embodiment Figures 4 and 5 show an example of the configuration of the data processing system 30 according to the third embodiment. The data processing system 30 is similar to the data processing system 20 according to the second embodiment in that metadata is passed via a message broker and the payload is passed via a database. However, unlike the data processing system 20, the data processing system 30 does not require sequential microservices as an essential component.

[0037] The data processing system 30 has a configuration in which multiple microservices are connected in multiple stages via a message broker. The data processing system 30 is a system in which images obtained from a camera are processed by microservice A, and the images obtained from the camera and the data obtained from the image processing by microservice A are processed by microservice B. More specifically, the data processing system 30 is a system in which microservice B is executed by an image processing pipeline that requires both the image and the data output by microservice A.

[0038] Figure 4 shows the configuration of the preceding stage of the data processing system 30. A message producer 31 is located at the very top of the data processing system 30. In this embodiment, the message producer is assumed to be a camera stream reader that distributes images acquired by the camera. The messages generated by the message producer 31 include metadata and an image as a payload.

[0039] Messages generated by message producer 31 are processed by microservice A. Microservice A comprises multiple message consumers 33. These multiple message consumers 33 process messages in a distributed manner to generate the data required by microservice B.

[0040] Metadata and images are exchanged separately between the message producer 31 and each message consumer 33. The data processing system 30 includes a message broker 32 for exchanging metadata between the message producer 31 and each message consumer 33. The message broker 32 has multiple partitions. Metadata sent from the message producer 31 is stored in these multiple partitions according to predetermined rules.

[0041] Furthermore, the data processing system 30 includes a database 36 for transferring images between the message producer 31 and each message consumer 33. The message producer 31 sends metadata to the message broker 32 and simultaneously stores the corresponding image in the database 36.

[0042] The metadata stored in each partition of the message broker 32 is consumed one by one by the corresponding message consumer 33. Each time a message consumer 33 consumes metadata, it searches the database 36 using the consumed metadata as a search key. The message consumer 33 then retrieves the image obtained from the database 36 and processes the retrieved image. Microservice A is executed when this process is performed in each message consumer 33.

[0043] Figure 5 shows the configuration of the downstream data processing system 30. More specifically, Figure 5 shows the configuration of the downstream data processing system 30 when focusing on one of the multiple message consumers 33 provided by microservice A. The data obtained from processing by microservice A's message consumer 33 is processed by microservice B. Microservice B has multiple message consumers 35. Microservice B distributes the processing of the data obtained by microservice A's message consumer 33 using these multiple message consumers 35.

[0044] However, the data obtained through processing by the message consumer 33 is divided into metadata and payload, i.e., net data. The metadata and net data are then passed separately between the message consumer 33 and each message consumer 35. The data processing system 30 includes a message broker 34 for passing metadata between the message consumer 33 and each message consumer 35. The message broker 34 has multiple partitions. The metadata sent from the message consumer 33 is stored in these multiple partitions according to predetermined rules.

[0045] Furthermore, the message consumer 33 sends metadata to the message broker 34 and simultaneously stores the corresponding data (payload) in the database 36. In the database 36, associations are made between each image input from the message producer 31 and each piece of data input from the message consumer 33. The database 36 may be a relational database, a NoSQL database, or especially a graph database.

[0046] The metadata stored in each partition of the message broker 34 is consumed one by one by the corresponding message consumer 35. Each time a message consumer 35 consumes metadata, it searches the database 36 using the consumed metadata as a search key. The message consumer 35 then retrieves the image and data set obtained from the database 36 and processes the retrieved image and data set. Microservice B is executed when this process is performed in each message consumer 35.

[0047] The effects of the data processing system 30 configured as described above can be explained by comparing it with a comparative example. Figure 6 shows the configuration of data processing system 100 as a comparative example to data processing system 30. In Figure 6, elements with the same reference numerals as data processing system 30 are elements that have the same function.

[0048] As a comparative example, data processing system 100 is a system that, like data processing system 30, processes images with microservice A, and further processes the images and the data obtained from the image processing by microservice A with microservice B. However, while data processing system 30 uses a message broker and a database for data transfer, data processing system 100 transfers all data using a message broker.

[0049] In the data processing system 100, the image (including metadata) generated by the message producer 31 is sent to the message broker 32 and stored in multiple partitions provided by the message broker 32 according to predetermined rules. The image stored in each partition of the message broker 32 is consumed one by one by the corresponding message consumer 33 and processed by the message consumer 33. Microservice A is executed when the image processing is performed by each message consumer 33.

[0050] The data (including metadata) obtained from processing by each message consumer 33 of microservice A is sent to the message broker 34 and stored in multiple partitions provided by the message broker 34 according to predetermined rules. The data stored in each partition of the message broker 34 is consumed one by one by the corresponding message consumer 35. Furthermore, the message consumer 35 consumes an image from the message broker 32. The message consumer 35 matches the consumed image with the data and processes matching image and data as a set. Microservice B is executed when this processing is performed in each message consumer 35.

[0051] In the comparative example data processing system 100 configured as described above, the image is transferred multiple times by the message broker 32 in the image processing pipeline. As a result, increased traffic in the message broker 32 causes waiting time, increasing the overall working time of the data processing system 100. In this regard, in the data processing system 30 according to the third embodiment, the image is transferred via the database 36, so the waiting time in the message broker 32 is reduced.

[0052] Furthermore, in the comparative example data processing system 100, microservice B is required to perform the task of matching the data output from microservice A with the image coming from the stream. This task is not always easy and requires a lot of memory. In particular, if a microservice requires the output of multiple other microservices as its input, matching becomes even more difficult and the memory used by that microservice increases dramatically in a short time. In this regard, in the data processing system 30 according to the third embodiment, when microservice B consumes metadata, the set of images and data associated with that metadata is obtained from the database 36, so there is no need for microservice B to perform complex matching.

[0053] 4. Hardware Each of the above-described embodiments of the data processing system can be configured, for example, by a server. The server constituting the data processing system includes at least one processor and at least one memory for storing a data processing program executed by that at least one processor. The data processing program includes a plurality of instructions and can be recorded on a non-temporary computer-readable recording medium. Pipeline processing in the data processing system is performed by executing the plurality of instructions of the data processing program on at least one processor. [Explanation of symbols]

[0054] 10, 20, 30 Data Processing Systems 11 Regular Microservices 12. Sequential Microservices 13 Order Manager 14, 32, 34 Message Broker 15, 36 Databases 31 Message Producer 33, 35 Message Consumer

Claims

1. A data processing system, Sequential microservices process data arranged in a time series sequentially, Multiple microservices connected in parallel to the aforementioned sequential microservice, The system includes an order manager that manages the order in which the sequential microservices process data processed in parallel by the aforementioned multiple microservices, The aforementioned order manager, Pass to the sequential microservice data corresponding to the minimum timestamp message, which has the smallest timestamp among a message queue containing a predetermined number of messages obtained from the aforementioned multiple microservices, The process involves continuously obtaining new messages one by one from the multiple microservices until a message with a timestamp greater than the minimum timestamp message is obtained. The system is configured to repeatedly add a message with a larger timestamp to the message queue when a message with a larger timestamp than the aforementioned is obtained. A data processing system characterized by the following features.

2. In the data processing system according to claim 1, A message broker that mediates message metadata between the aforementioned multiple microservices and the aforementioned sequential microservices, It further includes a database in which the payload corresponding to each message is stored, The order manager is configured to retrieve from the database the payload corresponding to the metadata obtained from the message broker. A data processing system characterized by the following features.

3. A method for managing the order in which sequential microservices process data processed in parallel by multiple microservices, Pass to the sequential microservice data corresponding to the minimum timestamp message, which has the smallest timestamp among a message queue containing a predetermined number of messages obtained from the aforementioned multiple microservices, The process involves continuously obtaining new messages one by one from the multiple microservices until a message with a timestamp greater than the minimum timestamp message is obtained. If a message with a larger timestamp than the aforementioned is obtained, the message with the larger timestamp is added to the message queue, and this process is repeated. A method characterized by the following:

4. A program containing multiple instructions that are executed by at least one processor, The aforementioned multiple instructions are, Sequential microservices process data arranged in a time series sequentially, Multiple microservices connected in parallel to the aforementioned sequential microservice, An order manager is configured to control the order in which the sequential microservices process data processed in parallel by the multiple microservices, and to enable at least one processor to function as such. The aforementioned order manager, Pass to the sequential microservice data corresponding to the minimum timestamp message, which has the smallest timestamp among a message queue containing a predetermined number of messages obtained from the aforementioned multiple microservices, The process involves continuously obtaining new messages one by one from the multiple microservices until a message with a timestamp greater than the minimum timestamp message is obtained. The system is configured to repeatedly add a message with a larger timestamp to the message queue when a message with a larger timestamp than the aforementioned is obtained. A program characterized by the following features.

Images