Data processing system, method for managing processing order of data, and program
The system maintains chronological order in data processing by using an order manager to prioritize messages based on timestamps, ensuring efficient parallel processing of data across multiple microservices.
Patent Information
- Application Number
- JP2023075615
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-01
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2043-05-01
AI Technical Summary
Existing systems fail to maintain the chronological order of data input to a sequential microservice when multiple microservices are connected in parallel, leading to potential processing errors and inefficiencies.
A data processing system that includes a sequential microservice, multiple parallel microservices, and an order manager to manage the processing order, ensuring that only messages with the smallest timestamp are processed first, discarding older messages and adding newer ones to maintain chronological order.
This approach ensures that data is processed in chronological order while allowing parallel processing, enhancing data processing speed and efficiency by avoiding reverse ordering of data inputs.
Smart Images

Figure 2024160521000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a data processing system, a method for managing a data processing order, and a program. [Background technology]
[0002] Japanese Patent No. 7031424 discloses a conventional technology related to a system for distributed data processing using microservices. In this conventional technology, messages are communicated between multiple microservices, and messages received by each microservice are stored in a queue in the order of arrival, and predetermined message processing is performed in the order of storage. In addition, in this conventional technology, messages that arrive late are detected by comparing timestamps between multiple received messages. Then, if a message arrives late, the system rolls back to the message processing state before the message that arrived late, and the message processing is performed again in the correct order of arrival.
[0003] In the above conventional technology, even if there is a delay in the arrival of messages, each of the multiple microservices can process messages in the correct order of arrival. However, the conventional technology does not fully consider a system in which multiple microservices are connected in parallel to a sequential microservice that processes data arranged in chronological order in sequence.
[0004] In addition to Japanese Patent No. 7,031,424, Japanese Patent No. 7,126,712 can be mentioned as an example of a document showing the state of the art in the technical field related to the present disclosure. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Patent No. 7031424 [Patent Document 2] Patent No. 7126712 Summary of the Invention [Problem to be solved by the invention]
[0006] The present disclosure has been made in view of the above problems. One objective of the present disclosure is to maintain the chronological order of data input to a sequential microservice in a system in which a plurality of microservices are connected in parallel to the sequential microservice. [Means for solving the problem]
[0007] The present disclosure provides a data processing system for achieving the above object. The data processing system of the present disclosure includes a sequential microservice that sequentially processes data arranged in a time series, a plurality of microservices that are connected in parallel to the sequential microservice, and an order manager. The order manager has a function of managing the order in which the sequential microservice processes data that has been processed in parallel by the plurality of microservices. In detail, the order manager is configured to repeatedly execute the following first process, second process, and third process. The first process is to pass data corresponding to a minimum timestamp message that has the smallest timestamp among a message queue that includes a predetermined number of messages consumed from the plurality of microservices to the sequential microservice. The second process is to continue consuming new messages one by one from the plurality of microservices until a message having a larger timestamp than the minimum timestamp message is obtained. And the third process is to add the message having the larger timestamp to the message queue when the message having the larger timestamp is obtained.
[0008] The data processing system of the present disclosure may further include a message broker and a database. The message broker has a function of mediating metadata of messages between the multiple microservices and the sequential microservices, and the database stores payloads corresponding to each message. The order manager may be configured to retrieve the payloads corresponding to the metadata consumed from the message broker from the database.
[0009] The present disclosure provides a method for achieving the above object. The method disclosed herein is a method for managing the order in which a sequential microservice processes data that has been processed in parallel by multiple microservices, and includes the following three steps that are repeatedly performed. In the first step, data corresponding to a minimum timestamp message that has the smallest timestamp in a message queue that contains a predetermined number of messages consumed from multiple microservices is passed to the sequential microservice. In the second step, new messages are continued to be consumed one by one from the multiple microservices until a message having a larger timestamp than the minimum timestamp message is obtained. Then, in the third step, when a message having a larger timestamp is obtained, the message having the larger timestamp is added to the message queue.
[0010] The present disclosure provides a program for achieving the above object. The program of the present disclosure is a program including a plurality of instructions executed by at least one processor. The plurality of instructions are configured to cause the at least one processor to function as a sequential microservice that sequentially processes data arranged in time series, a plurality of microservices connected in parallel to the sequential microservice, and an order manager that manages the order in which the sequential microservice processes data processed in parallel by the plurality of microservices. The plurality of instructions are also configured to cause the order manager to repeatedly execute the first process, the second process, and the third process described above. The program of the present disclosure may be recorded on a computer-readable recording medium.
[0011] The present disclosure also provides another data processing system having a different configuration from the above data processing system. The other data processing system includes at least one message producer, a plurality of message consumers, a message broker, and a database. The message broker has a function of mediating message metadata between the at least one message producer and the plurality of message consumers, and the database stores payloads corresponding to each message. Each of the plurality of message consumers is configured to retrieve from the database a payload corresponding to the metadata consumed from the message broker. Effect of the Invention
[0012] According to the technology disclosed herein, if a newly consumed message from multiple microservices in parallel is older than a processed message, the older message is discarded without being passed to the sequential microservice. Then, data corresponding to the minimum timestamp message, which has the smallest timestamp among a predetermined number of messages, is passed to the sequential microservice. This makes it possible to maintain the chronological order of data input to the sequential microservice while processing data in parallel in multiple microservices, thereby improving the data processing speed in the system. [Brief description of the drawings]
[0013] [Figure 1] 1 is a diagram illustrating an example of a configuration of a data processing system according to a first embodiment. [Diagram 2] FIG. 13 is a diagram illustrating a process performed by an order manager. [Diagram 3] FIG. 11 is a diagram illustrating an example of a configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 13 is a diagram illustrating an example of a configuration of a data processing system according to a third embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of a configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 13 is a diagram showing a comparative example to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] 1. First embodiment The data processing system disclosed in the present disclosure is a system that performs distributed processing of data using microservices. Figure 1 shows an example of the configuration of a data processing system 10 according to a first embodiment. The data processing system 10 includes multiple regular microservices 11 and at least one sequential microservice 12. In the example shown in Figure 1, one sequential microservice 12 is provided for three regular microservices 11.
[0015] The regular microservice 11 is a microservice that is not a sequential microservice 12. While the sequential microservice 12 is a microservice that sequentially processes data arranged in chronological order, the regular microservice 11 does not require the chronological order of input data. Note that the contents of the services of each regular microservice 11 are different.
[0016] The regular microservices 11 are connected in parallel to the sequential microservices 12. This means that there is no dependency between the regular microservices 11, and each regular microservice 11 can process data in parallel. Each regular microservice 11 sends a message to the sequential microservice 12 independently of each other. In addition, communication between each regular microservice 11 and the sequential microservice 12 is asynchronous.
[0017] Since each regular microservice 11 processes data in parallel and is asynchronous with the sequential microservice 12, the order in which data arrives at the sequential microservice 12 is not necessarily the order in which the data should be processed. For this reason, the data processing system 10 further includes an order manager 13. The order manager 13 has a function of managing the processing order in the sequential microservice 12 of the data processed in parallel by the regular microservice 11.
[0018] The order manager 13 can be configured as a part of the sequential microservice 12 as shown in FIG. 1, or can be configured as a component separate from the sequential microservice 12. The order manager 13 is provided between the regular microservice 11 and the sequential microservice 12, and receives messages sent from the regular microservice 11 to the sequential microservice 12.
[0019] FIG. 2 shows a specific example of processing performed by the order manager 13. The order manager 13 is a message queue that can store a predetermined number of messages consumed from the regular microservice 11. In the example shown in FIG. 2, a total of 10 messages are stored in the order manager 13. Note that the messages are represented as msg. Also, TS(x) attached to each message in FIG. 2 represents a timestamp, and the smaller the value of x, the older the message. Below, the processing performed by the order manager 13 will be explained in the order of steps.
[0020] In step A, the order manager 13 sorts the messages stored in the message queue in order of time stamps. In the message arrangement in step A, the smallest time stamp is TS(4), and the message having the time stamp TS(4) is the smallest time stamp message.
[0021] Next, in step B, the order manager 13 processes the minimum timestamp message in the message queue and passes the data corresponding to the minimum timestamp message to the sequential microservice 12, which then processes the message with timestamp TS(4).
[0022] Next, in step C, the order manager 13 consumes one new message from the regular microservice 11. Which regular microservice 11 the message is consumed from depends on the message sending timing of each regular microservice 11. In the example shown in Figure 2, a message having a timestamp TS(3) is newly consumed.
[0023] Next, in step D, the order manager 13 compares the timestamp of the newly consumed message in step C with the timestamp of the minimum timestamp message processed in step B. Depending on the result of this comparison, the order manager 13 will perform different processing in the next step.
[0024] In the case of step D, the timestamp TS(3) of the newly consumed message is smaller than the timestamp TS(4) of the previously processed message with the smallest timestamp. In other words, the newly consumed message is older than the previously processed message. In this case, processing the data of the newly consumed message would result in data being processed in reverse order, which could cause inconvenience to the service.
[0025] Therefore, in step E, the order manager 13 discards the newly consumed message in step C, ie, the message having timestamp TS(3), without adding it to the message queue.
[0026] Next, in step F, the order manager 13 again consumes one new message from the regular microservice 11. In the example shown in Figure 2, a message with timestamp TS(11) is newly consumed.
[0027] Next, in step G, the order manager 13 compares the timestamp of the newly consumed message in step F with the timestamp of the lowest timestamp message previously processed. The lowest timestamp message previously processed is the message having timestamp TS(4) processed in step B.
[0028] The order manager 13 continues to consume new messages one by one from the regular microservice 11 until it obtains a message having a timestamp greater than the minimum timestamp message. In the case of step G, the timestamp TS(11) of the newly consumed message is greater than the timestamp TS(4) of the previously processed minimum timestamp message. In other words, the newly consumed message is newer than the previously processed message. In this case, by processing the data of the newly consumed message, it is possible to process data that should be processed in chronological order in chronological order.
[0029] Therefore, in step H, the order manager 13 adds to the message queue the message newly consumed in step C, i.e., the message having timestamp TS(11). Then, the order manager 13 sorts the messages stored in the message queue in the order of timestamp.
[0030] By managing the processing order by the above-mentioned method, the order manager 13 can maintain the chronological order of data input to the sequential microservice 12 while processing data in parallel in multiple regular microservices 11. As a result, the data processing system 10 can achieve a high data processing speed.
[0031] 2. Second embodiment Figure 3 shows an example of the configuration of a data processing system 20 according to the second embodiment. In Figure 3, elements that are given the same reference numerals as those in the data processing system 10 according to the first embodiment have common functions. The data processing system 20 is a system that uses a message broker 14 for passing multiple messages. In addition to multiple regular microservices 11, sequential microservices 12, and an order manager 13, the data processing system 20 also includes the message broker 14 and a database 15.
[0032] A message output from each regular microservice 11 includes metadata and a payload. The message broker 14 mediates the metadata of the message between the regular microservice 11 and the sequential microservice 12. The message broker 14 has multiple partitions. The metadata sent from each regular microservice 11 is stored in the associated partition. Meanwhile, the payload of the message is stored in a database 15. An example of the payload is image data.
[0033] In the data processing system 20, the order manager 13 consumes metadata one by one from the message broker 14. The processing of the consumed metadata is the same as the processing of the message performed in the first embodiment. That is, the order manager 13 processes the metadata with the smallest timestamp among the predetermined number of metadata consumed from the message broker 14. Then, every time it processes one piece of metadata in the message queue, it consumes one new piece of metadata from the message broker 14 and compares the timestamp with the previously processed metadata. If the timestamp of the newly consumed metadata is equal to or smaller than that of the previously processed metadata, the order manager 13 discards the newly consumed metadata and consumes one new piece of metadata from the message broker 14 again. On the other hand, if the timestamp of the newly consumed metadata is greater than that of the previously processed metadata, the order manager 13 processes the newly consumed metadata.
[0034] When processing metadata, the order manager 13 retrieves the payload corresponding to the metadata from the database 15. Then, the order manager 13 passes the retrieved payload together with the metadata to the sequential microservice 12. This allows data (payload) that should be processed in chronological order to be processed in chronological order in the sequential microservice 12.
[0035] According to the data processing system 20 configured as above, similarly to the first embodiment, it is possible to maintain the chronological order of data input to the sequential microservice 12 while processing data in parallel in a plurality of regular microservices 11. Furthermore, since the data transferred via the message broker 14 is only metadata, it is possible to reduce the waiting time required for data transfer compared to the case where the entire message is transferred, and therefore it is possible to reduce the overall operation time of the data processing system 20.
[0036] 3. Third embodiment 4 and 5 show an example of the configuration of a data processing system 30 according to the third embodiment. The data processing system 30 is common to the data processing system 20 according to the second embodiment in that metadata is transferred via a message broker and payload is transferred via a database. However, unlike the data processing system 20, the data processing system 30 does not require sequential microservices as a required component.
[0037] The data processing system 30 has a configuration in which multiple microservices are connected in multiple stages via a message broker. The data processing system 30 is a system in which an image obtained from a camera is processed by microservice A, and the image obtained from the camera and data obtained by the image processing by microservice A are processed by microservice B. More specifically, the data processing system 30 is a system in which microservice B is executed by an image processing pipeline that requires both the image and the data output by microservice A.
[0038] 4 shows the configuration of the front stage of the data processing system 30. A message producer 31 is located at the most upstream of the data processing system 30. In this embodiment, the message producer is a camera stream reader that distributes images acquired by a camera. A message generated by the message producer 31 includes metadata and an image as a payload.
[0039] Messages generated by message producer 31 are processed by microservice A. Microservice A has multiple message consumers 33. These multiple message consumers 33 process messages in a distributed manner and generate data required by microservice B.
[0040] Metadata and images are passed separately between the message producer 31 and each message consumer 33. The data processing system 30 includes a message broker 32 for passing metadata between the message producer 31 and each message consumer 33. The message broker 32 includes a plurality of partitions. The metadata sent from the message producer 31 is stored in the plurality of partitions according to predetermined rules.
[0041] The data processing system 30 also includes a database 36 for transferring images between the message producer 31 and each message consumer 33. The message producer 31 transmits metadata to the message broker 32 and simultaneously stores the corresponding image in the database 36.
[0042] The metadata stored in each partition of the message broker 32 is consumed one by one by the corresponding message consumer 33. Every time the message consumer 33 consumes metadata, it searches the database 36 using the consumed metadata as a search key. Then, the message consumer 33 retrieves the image obtained by the search from the database 36 and processes the retrieved image. Microservice A is executed by performing such processing in each message consumer 33.
[0043] Fig. 5 shows the configuration of the rear stage of the data processing system 30. More specifically, Fig. 5 shows the configuration of the rear stage of the data processing system 30 when focusing on one of the multiple message consumers 33 provided in microservice A. Data obtained by processing by the message consumer 33 of microservice A is processed by microservice B. Microservice B has multiple message consumers 35. Microservice B performs distributed processing of the data obtained by the message consumer 33 of microservice A using the multiple message consumers 35.
[0044] However, data obtained by processing by the message consumer 33 is divided into metadata and payload, i.e., net data. Then, the metadata and net data are transferred separately between the message consumer 33 and each message consumer 35. The data processing system 30 includes a message broker 34 for transferring metadata between the message consumer 33 and each message consumer 35. The message broker 34 includes multiple partitions. The metadata sent from the message consumer 33 is stored in the multiple partitions according to predetermined rules.
[0045] Furthermore, the message consumer 33 transmits the metadata to the message broker 34 and at the same time stores the corresponding data (payload) in the database 36. In the database 36, associations are made between each image input from the message producer 31 and each data input from the message consumer 33. The database 36 may be a relational database or a NoSQL database, particularly a graph database.
[0046] The metadata stored in each partition of the message broker 34 is consumed one by one by the corresponding message consumer 35. Every time the message consumer 35 consumes metadata, it searches the database 36 using the consumed metadata as a search key. The message consumer 35 then retrieves the set of images and data obtained by the search from the database 36, and processes the retrieved set of images and data. Microservice B is executed by performing such processing in each message consumer 35.
[0047] The effects of data processing system 30 configured as above can be explained by comparing it with a comparative example. Figure 6 is a diagram showing the configuration of data processing system 100 as a comparative example to data processing system 30. In Figure 6, elements that are given the same reference numerals as those in data processing system 30 have the same functions.
[0048] The data processing system 100 as a comparative example is a system that processes an image by microservice A, and further processes the image and data obtained by the image processing by microservice A by microservice B, just like the data processing system 30. However, while the data processing system 30 uses a message broker and a database to transfer data, the data processing system 100 transfers all data using a message broker.
[0049] In the data processing system 100, an image (including metadata) generated by a message producer 31 is sent to a message broker 32 and stored in multiple partitions of the message broker 32 according to predetermined rules. The images stored in each partition of the message broker 32 are consumed one by one by the corresponding message consumer 33 and processed by the message consumer 33. Microservice A is executed by processing the image in each message consumer 33.
[0050] The data (including metadata) obtained through processing by each message consumer 33 of microservice A is sent to the message broker 34 and stored in multiple partitions of the message broker 34 according to predetermined rules. The data stored in each partition of the message broker 34 is consumed one by one by the corresponding message consumer 35. Furthermore, the message consumer 35 consumes images from the message broker 32. The message consumer 35 matches the consumed image with the data, and processes the matching image and data as a set. Microservice B is executed by each message consumer 35 performing such processing.
[0051] In the data processing system 100 of the comparative example configured as described above, images are handed over by the message broker 32 multiple times in the image processing pipeline. This causes waiting time due to increased traffic in the message broker 32, and increases the overall operation time of the data processing system 100. In this regard, in the data processing system 30 according to the third embodiment, images are handed over via the database 36, so the waiting time in the message broker 32 is reduced.
[0052] In addition, in the data processing system 100 of the comparative example, the microservice B needs to perform a matching operation between the data output from the microservice A and the image coming from the stream. This operation is not necessarily easy and requires a lot of memory. In particular, when a microservice requires the output of multiple other microservices as its input, the matching becomes even more difficult and the memory used by the microservice increases dramatically in a short time. In this regard, in the data processing system 30 according to the third embodiment, when the microservice B consumes the metadata, a set of images and data related to the metadata is obtained from the database 36, so that there is no need to perform complex matching in the microservice B.
[0053] 4. Hardware The data processing system according to each of the above-described embodiments can be configured, for example, by a server. The server configuring the data processing system includes at least one processor and at least one memory that stores a data processing program executed by the at least one processor. The data processing program includes a plurality of instructions, and can be recorded on a non-transitory computer-readable recording medium. The instructions of the data processing program are executed by the at least one processor, thereby performing pipeline processing in the data processing system. [Explanation of symbols]
[0054] 10, 20, 30 Data Processing System 11 Regular Microservices 12 Sequential Microservices 13 Order Manager 14, 32, 34 Message Broker 15, 36 Database 31 Message Producers 33, 35 Message Consumer
Claims
1. 1. A data processing system comprising: Sequential microservices that process data in chronological order, A plurality of microservices connected in parallel to the sequential microservice; An order manager that manages an order in which the sequential microservice processes data that has been processed in parallel by the multiple microservices; The order manager: Passing data corresponding to a minimum timestamp message having a minimum timestamp in a message queue including a predetermined number of messages consumed from the plurality of microservices to the sequential microservice; continue consuming new messages one by one from the plurality of microservices until a message having a timestamp greater than the minimum timestamp message is obtained; if a message having a larger timestamp is obtained, adding the message having the larger timestamp to the message queue. A data processing system comprising:
2. 2. The data processing system of claim 1, A message broker that mediates message metadata between the plurality of microservices and the sequential microservice; a database in which a payload corresponding to each message is stored; The order manager is configured to retrieve from the database a payload corresponding to the metadata consumed from the message broker.
23. A data processing system comprising:
3. 1. A data processing system comprising: At least one message producer; Multiple message consumers, a message broker that mediates message metadata between the at least one message producer and the plurality of message consumers; a database in which a payload corresponding to each message is stored; Each of the plurality of message consumers is configured to retrieve from the database a payload corresponding to metadata consumed from the message broker.
23. A data processing system comprising:
4. A method for managing an order in which a sequential microservice processes data that has been processed in parallel by a plurality of microservices, comprising: Passing data corresponding to a minimum timestamp message having a minimum timestamp in a message queue including a predetermined number of messages consumed from the plurality of microservices to the sequential microservice; continue consuming new messages one by one from the plurality of microservices until a message having a timestamp greater than the minimum timestamp message is obtained; if a message having a larger timestamp is obtained, adding the message having the larger timestamp to the message queue. A method comprising:
5. A program including a plurality of instructions for execution by at least one processor, The instructions include: Sequential microservices that process data in chronological order, A plurality of microservices connected in parallel to the sequential microservice; The at least one processor is configured to function as an order manager that manages an order in which the sequential microservice processes data that has been processed in parallel by the multiple microservices; The order manager: Passing data corresponding to a minimum timestamp message having a minimum timestamp in a message queue including a predetermined number of messages consumed from the plurality of microservices to the sequential microservice; continue consuming new messages one by one from the plurality of microservices until a message having a timestamp greater than the minimum timestamp message is obtained; if a message having a larger timestamp is obtained, adding the message having the larger timestamp to the message queue. A program characterized by:
Citation Information
Patent Citations
Payload description for computer messaging
US20180102996A1
Parallel data processing in embedded systems
US20220342722A1
Information processing device, information processing system, and information processing program
JP7031424B2
Data processing device, method, computer program, and recording medium
JP7126712B2
