Data processing management system, data processing management method

Abstract

To prevent a throughput reduction of processing desired to give preference in the case there is a branch instruction in stream processing.SOLUTION: A system for managing data processing comprises a queue laying-out system that lays out a queue with a processing description of stream processing having a branch instruction as an input. The queue laying-out system designates processing to be preferentially executed for the processing description, measures a throughput in each processing after the branch instruction, and acquires a branching percentage that is a percentage to execute each processing after the branch instruction. The queue laying-out system calculates a layout cost for laying out a queue between the branch instruction and the processing after the branch instruction, for each combination of those of processing after the branch instruction, using the throughput and the branching percentage and outputs a layout pattern of queues to prevent a throughput reduction of the preferential processing out of those of processing after the branch instruction.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a data processing management system and a data processing management method.

Background Art

[0002] In the industry, business improvement and business visualization using IoT (Internet of Things) are being promoted. Especially in the manufacturing industry, data analysis utilization that collects, organizes, processes, and analyzes sensor and log data in the factory in real time to create new value is accelerating. In order to achieve real-time data collection, stream processing that sequentially formats the data received from the sensor, converts it into a format for data analysis, and then outputs it to a database is effective.

[0003] Here, Patent Document 1 discloses a technique related to data processing by a plurality of processing elements. That is, Patent Document 1 discloses, "A processor comprising: a plurality of processing elements each having a temporary storage unit for temporarily storing an assigned unit instruction sequence, and capable of executing an instruction included in the unit instruction sequence stored in each of the temporary storage units; and an execution management unit that divides a program, which is an instruction sequence at a level lower than assembly language, into the unit instruction sequences that do not include branch instructions in the middle, and are instruction sequences starting from the leading instruction of the branch destination and ending with the branch instruction, sequentially assigns the divided unit instruction sequences to each of the plurality of processing elements, and causes the plurality of processing elements to execute the assigned unit instruction sequences in parallel."

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the method described in Patent Document 1, the instruction sequence is divided into instruction units, and a queue that enables temporary storage is placed before each instruction sequence. However, in stream processing, the use of a temporary storage queue itself involves data transfer to and reading from the queue system, which can be a source of delay. Therefore, in order to perform more economically efficient processing, it is considered necessary to treat critical branching processes, such as branching processes that output to real-time visualization functions, where throughput delays are critical, as a single stream process without placing a queue, in order to ensure the required throughput speed. Furthermore, in order to prevent throughput degradation for critical branching processes, it is considered necessary to consider the number of workers and the throughput performance of the branch destination process, and to prioritize inserting queues into high-load branching processes that become bottlenecks.

[0006] From this perspective, the present invention aims to provide a data processing management system and a data processing management method that can prevent a decrease in the throughput of a preferred process when there is a branch instruction within the stream processing. [Means for solving the problem]

[0007] According to a first aspect of the present invention, the following data processing management system is provided. The data processing management system includes a queue deployment system that deploys queues as input to a processing description of a stream process having branch instructions. The queue deployment system specifies the processing to be executed preferentially for the processing description, measures the throughput of each processing after the branch instruction, and obtains a branch ratio, which is the proportion of each processing after the branch instruction that is executed. Using the throughput and the branch ratio, the queue deployment system calculates the deployment cost when deploying queues between the branch instruction and the processing after the branch instruction for each combination of processing after the branch instruction, and outputs a queue deployment pattern that prevents a decrease in the throughput of the preferential processing among the processing after the branch instruction.

[0008] According to a second aspect of the present invention, the following data processing management method is provided. The data processing management method is a method performed using a processor. The data processing management method obtains a processing description of a stream process having a branch instruction, specifies a process to be executed preferentially for the processing description, measures the throughput of each process after the branch instruction, obtains a branch ratio which is the proportion of each process after the branch instruction that is executed, calculates the deployment cost when a queue is deployed between the branch instruction and the process after the branch instruction for each combination of processes after the branch instruction using the throughput and the branch ratio, outputs a queue deployment pattern that prevents a decrease in throughput of the preferential process among the processes after the branch instruction, and deploys a queue by referring to the queue deployment pattern. [Effects of the Invention]

[0009] According to the present invention, a data processing management system and a data processing management method are provided that can prevent a decrease in the throughput of a processing task to be prioritized when there is a branch instruction within the stream processing. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of a typical stream processing configuration to which the present invention can be applied. [Figure 2] This figure shows an example of the processing procedure for a worker on a stream processing system. [Figure 3] This code shows an example of data to be input to a stream processing system. [Figure 4] This is code that shows an example of a processing description. [Figure 5] This code shows an example of data output by a stream processing system. [Figure 6] This figure shows an example of the worker management process flow in a stream processing system. [Figure 7] This figure shows the relationship between the system according to the present invention and a related stream processing system. [Figure 8]It is a diagram showing an example of the hardware constituting the system related to the present invention. [Figure 9] It is a diagram showing an input screen for specifying processing description and priority processing in the user input section for the system related to the present invention. [Figure 10] It is a diagram showing an example of the flow until the system related to the present invention reads the processing description and deploys the queue. [Figure 11] It is a diagram showing an example of the processing description input to the system related to the present invention. [Figure 12] It is a diagram showing an example of the processing throughput table that the system related to the present invention has. [Figure 13] It is a diagram showing an example of the queue throughput table that the system related to the present invention has. [Figure 14] It is a diagram showing an example of the branch ratio table that the system related to the present invention has. [Figure 15] It is a diagram showing an example of the screen input in the user input section for the management table that the system related to the present invention has. [Figure 16] It is a diagram showing an example of the calculation information table that the system related to the present invention has. [Figure 17] It is a diagram showing an example of the queue deployment pattern table that the system related to the present invention has. [Figure 18] It is a diagram showing an example of displaying the processing description and queue deployment output by the system related to the present invention. [Figure 19] It is a diagram showing an example of the table display section for displaying the table output by the system related to the present invention. [Figure 20] It is a diagram showing an example of the flow in which the system related to the present invention measures the running stream processing. [Figure 21] It is a flowchart showing an example of the calculation in step 1005. [Figure 22] It is a diagram showing one of the patterns when the system related to the present invention divides the input processing description by a queue. [Figure 23]A diagram showing one pattern when the system related to the present invention divides the input processing description by a queue. [Figure 24] A diagram showing one pattern when the system related to the present invention divides the input processing description by a queue. [Figure 25] A diagram showing one configuration example of the system related to the present invention. [Figure 26] A flowchart showing an example of the processing of the queue deployment system in the configuration example shown in FIG. 25.

Embodiments of the Invention

[0011] First, general stream processing will be described while referring to FIG. 1. FIG. 1 shows a configuration example of general stream processing.

[0012] The stream processing system 108 includes a plurality of workers 110 in the arithmetic unit 109. The number of workers is determined by the number of processors installed on the hardware on which the system operates, and executes the processing described in the processing description 111. The arithmetic unit 109 can execute the processing for the number of workers 110 in parallel. When a processing is assigned to a worker, no other processing is assigned until the processing is completed, and the worker becomes free after completing the processing. When data is input to a free worker, a new processing is assigned.

[0013] Figure 2 shows a typical flowchart of how a worker operates. A free worker first reads input data 102 (S201). Here, an example of input data is explained with reference to Figure 3. Figure 3 is an example of input data 301, which is data entered in JSON (JavaScript Object Notation) format. In JSON format, values ​​are stored in key-value format, and in Figure 3, this means that 000011 is stored in id, M01010111 in name, 100 in sensor-1, and 400 in sensor-2. Input data can be in JSON format, CSV format, numbers, or strings only. In stream processing, a worker will operate and process each input data shown in Figure 3.

[0014] The worker executes the processing described in the assigned processing description on the input data (S202). The processing description will be explained with reference to Figure 4. Figure 4 is an example of a processing description, in which the stream execution processing system 108 executes the processing sequentially by calling functions in Javascript format. For example, line 401 reads the input data, and line 402 inserts the category key into the input data. Line 403 indicates a conditional branch if the key name contains the string Machine-, and line 404 indicates output to the cloud database. The processing description is written in the order of input processing, processing processing, and output processing, making it possible to process and output the input data.

[0015] The worker outputs the processed input data to the output destination specified in the assigned processing description (S203). The output data will be explained with reference to Figure 5. Figure 5 shows an example of JSON formatted output data after processing the input data in Figure 3. Compared to Figure 3, in output data 501, the value of the name key 502 has been changed to Machine-01, and the category key 503, tag key 504, and date key 505 have been added. Thus, in stream processing, it is common to convert values ​​within the data and insert metadata.

[0016] The worker has completed its assigned processing and outputs a processing completion signal to the arithmetic management unit 112 (S204).

[0017] Next, the processing of the arithmetic management unit 112 will be explained with reference to Figure 6. Figure 6 shows a general operation flowchart of the arithmetic management unit 112 that manages workers and processing descriptions in the stream processing system 108.

[0018] The arithmetic management unit 112 reads the processing description (S601). The arithmetic management unit 112 checks whether it is possible to receive input data and output data for workers to execute the processing description, and confirms communication with each device and the database (S602). The arithmetic management unit 112 allocates memory for workers to execute the processing description (S603). When each worker executes the processing description, it allocates the amount of memory necessary for processing from the memory allocated in this step and executes the processing. At this time, the memory allocated by each worker is not shared, and workers release the allocated memory area when they complete processing. The arithmetic management unit 112 puts all workers into a free state (S604) and waits (S605).

[0019] Then, when input data arrives (S606), the process moves to step 607. The arithmetic management unit 112 processes the input data, and in order to do so, it sets an arbitrary free worker to a processing state and executes the processing described in the processing description (S607).

[0020] The worker executes the process according to the flow shown in Figure 2. After assigning the process, the processing of the calculation management unit 112 proceeds back to step 605.

[0021] When a worker outputs a processing completion signal to the arithmetic management unit 112, processing proceeds to step 609 (S608). The arithmetic management unit 112 returns the worker to a free state because it has completed processing (S609). Processing by the arithmetic management unit 112 returns to step 605. In this way, based on the flowchart above, the stream processing system 108 processes the input data 102 in stream format based on the processing description 111 and outputs it as output data 105.

[0022] In stream processing with the configuration described above, achieving high speed requires running workers in parallel, defining a single input and output, and maintaining a constant throughput for each worker to improve performance. On the other hand, data utilization methods have diversified in recent years, and data is used in a wide range of applications, not only for real-time visualization but also for storage in cloud databases, data processing and output for AI, and more.

[0023] In response to the diversification of data analysis and utilization described above, conventional data collection stream processing techniques used branching instructions to differentiate processing methods and store data in multiple databases. However, branching instructions within stream processing can lead to variations in the throughput of each worker, and the overall throughput of the stream processing can deteriorate if the processing at the branch destination, which has a high processing load, monopolizes the workers. Generally, if a high-load process is the bottleneck in stream processing, it is thought that throughput can be improved by inserting a queue, such as a buffer, immediately before it, thereby splitting the stream processing into two.

[0024] Next, each embodiment will be described with reference to Figures 7-26. Note that explanations similar to those described above may be omitted.

[0025] <First Embodiment> <Structure> An example of a system according to the first embodiment is shown in Figure 7. This system can manage stream processing and, as an example, can manage cache in database analysis. This system (data processing management system) is configured by associating three systems, a queue deployment system 113, a user output unit 126, and a user input unit 129, with the stream processing system 108, and comprises a queue deployment system 113, a user output unit 126, and a user input unit 129.

[0026] In this embodiment, the stream processing system 108 receives input data 102 from the data generation location 101. The stream processing system 108 also reads the processing description 111 written by the user in the user input unit 129 and performs the processing in the management calculation unit 112. That is, the stream processing system 108 assigns a worker 110 of the calculation unit 109 to each input data 102 based on the processing description. The stream processing system 108 then outputs output data 105 to the database 104 of the data lake 103 and the visualization service 106.

[0027] Examples of data generation locations 101 include sensors and robot main units deployed in the factory, as well as PLCs (Programmable Logic Controllers) that operate the main units. Examples of data lakes 103 include RDBs (Relational Databases), file systems, and NoSQL databases.

[0028] The queue deployment system 113 is connected to the stream processing system 108. If a branch instruction exists in the processing description 111 entered in the user input unit 129 and a process to be prioritized is specified, the queue deployment calculation unit 114 generates the tables shown in the management table 119. The queue deployment execution unit 125 then divides the processing description 111, deploys the queue 133, and implements it to the stream processing system 108.

[0029] The user input unit 129 receives the processing description 111 to be run on the stream processing system 108 via the processing description input unit 130. If the processing description 111 contains a branch instruction, the user input unit 129 receives the designation of priority processing for the processing destination of the branch instruction via the priority processing input unit 131. The user input unit 129 also receives input for the management table 119 of the queue deployment system 113 via the table input unit 132.

[0030] The user output unit 126 has the function of displaying the management table 119 in the table display unit 127 and displaying the processing description 111 running on the stream processing system 108 and the deployed queue 133 in the result display unit 128.

[0031] Figure 8 shows an example of a hardware configuration for implementing the system according to the first embodiment. The stream processing system 108 and the queue deployment system 113 are deployed on a computer equipped with a processor 203, a storage device 207, and memory 204, and predetermined processing is executed within the memory 204. The data generation device 212, the computer, and the cloud environment 210 are connected and communicate with each other via network 201 through network IF 202.

[0032] The input device 208 consists of a keyboard, mouse, touch panel, or a combination of a mouse and a display. The user input unit 129 can be configured using the input device 208. The output device 209 consists of a display. The user output unit 126 can be configured using the output device 209. The storage device 207 consists of a non-volatile storage medium and can store data output by the stream processing system 108 (pipeline processing system), processing descriptions 111, etc. Queues deployed by the queue deployment system 113 are deployed in the memory 204, and in the case of large queues, they can also be managed internally within the storage device 207.

[0033] The data generation device 212 receives data from the PLC 213 and sensor 214 and outputs it through the network IF 202. This device can, for example, consist of a computer including a processor and memory. On the other hand, if, for example, the PLC 213 has a data output function via the network IF 202, the PLC itself (the configuration having the data output function) falls under the category of this device.

[0034] The cloud environment 210 represents an external computer connected via the internet. In this example, the cloud environment 210 includes a suitable storage device 211 and a processor 215, and the stream processing system 108 can also output to the storage device 211 and applications running on the cloud environment 210.

[0035] <Processing> When a user input unit 129 inputs a processing description 111 to the stream processing system 108 described above, and the processing description 111 includes a branch instruction, the system according to the first embodiment rewrites the data to appropriately allocate queue 133 and transfers the processing description, which improves throughput degradation, to the stream processing system 108. Here, a branch instruction refers to a conditional branch in JavaScript, such as the positive or negative value of an if statement shown in line 403 of Figure 4, or a conditional statement that processes items that meet multiple conditions, such as a switch statement.

[0036] Next, an example of the procedure for the operation of the queue deployment system 113 will be shown. The queue deployment system 113 operates when the processing description 111 entered by the user in the user input unit 129 is transferred to the stream processing system 108. First, the input of the processing description 111 by the user will be explained with reference to Figure 9. Figure 9 shows an example of the input screen in the user input unit. In the screen shown in Figure 9, the design of the functions to be written inside the processing description and their order is made possible by nodes and edges.

[0037] Node 901 is intended to hold an input function, and by connecting it to node 903 (process A) via edge 902, the code that executes function 903 of process A after input function 901 is written as a process description. Each function can be selected from the function palette 904. For example, if the user selects node 905 (if), which has the meaning of a branch instruction, from the palette, the if node can be connected as a process description.

[0038] Here, if a branch instruction exists in the processing description, the user can specify priority processing for the functions connected after the branch instruction for input to the queue deployment system 113. Priority processing is applied to processes that prioritize real-time performance and where throughput delays should be prevented. For example, in Figure 9, node 907 (priority processing) is the process that follows node 906 (if), and is a function that can be specified as priority processing.

[0039] When editing a node to be prioritized for processing using the screen shown in Figure 9, an editing screen 908 is displayed depending on the node selected by the user. The editing screen 908 displays the code related to the node, along with a priority processing check 909. If the user enables the priority processing check 909, the priority processing function is designated as the priority processing. If the processing description designed by the user in the user input unit 129 includes a branch instruction, the processing description and priority processing specification are transferred to the queue deployment system 113 before being transferred to the stream processing system 108.

[0040] Note that the display configuration in Figure 9 is just one example and may be modified as appropriate. For example, in addition to the input screen shown in Figure 9, all processing descriptions may be written as code. Furthermore, the configuration of the node is not limited to the configuration shown in Figure 9.

[0041] For convenience, in this example, the function to be prioritized (node ​​907) is illustrated and explained as "priority processing," but a node that could be a candidate for priority processing may be shown in a simple manner without specifying the priority processing, such as "process C." The user may then select such a node and output the editing screen 908 to specify the priority processing for that node.

[0042] Furthermore, for convenience, in this example, the function for high-load processing is referred to as "high-load processing," but the node related to such processing may also be shown in a simpler form that does not specify that it is high-load processing, such as "process D."

[0043] Selecting any node may output an editing screen 908 for that node. Here, the user may select a node shown in the function palette 904, or a node shown outside of the function palette 904.

[0044] Next, we will describe an example of the operation of the queue deployment system 113. Figure 10 is a flowchart showing an example of the operation of the queue deployment system (stream processing deployment). The steps (processes) in the flowchart will be explained as an example of operation when the processing description shown in Figure 9 is entered.

[0045] First, we will explain the data input / output relationship while referring to Figure 11. The processing flow shown in Figure 11 is a diagram illustrating an example of a processing description 1101 written by the user and its input / output relationship.

[0046] The input data 1102 is first processed by process A1103, and then conditional branching is executed by branch instruction 1104. Depending on the conditions, the input data may proceed to priority process 1105 and then be stored in database A1106, or it may go to database B1108 after process B1107, or it may go to database C1110 from high-load process 1109.

[0047] The queue deployment system 113 receives the processing description 111 and priority processing specification from the user input unit 129 (S1001). The user may also input data related to the management table 119, in which case the queue deployment system 113 receives the data entered into the management table 119. If the processing description 111 contains a branch instruction (if in this example), the process proceeds to step 1003 (S1002). On the other hand, if it does not contain a branch instruction, the process ends.

[0048] The queue deployment system 113 reads the priority processing specifications using the priority processing specification unit 118 and sets a priority processing flag for the conditions of the function and branch instruction specified as priority processing (S1003). Then, the queue deployment system 113 generates a processing throughput table 120, a queue throughput table 121, a branch ratio table 122, and an operation information table 123 based on the processing description 111 (S1004).

[0049] Here, we will explain each table. First, we will explain an example of the processing throughput table 120, referring to Figure 12. Figure 12 shows an example of a processing throughput table. The processing throughput table is data that summarizes the throughput of each process, and in this example, it is a table that holds the processing throughput information 1201 for each function in the processing description 111.

[0050] Next, an example of the queue throughput table 121 will be described with reference to Figure 13. Figure 13 shows an example of the queue throughput table 121. In this example, the queue throughput table 121 is a table that contains the performance of the queues deployed by the queue deployment system 113 and throughput information 1301 when data is transferred to the queues. For example, the queue name topic-A is shown to have a port number of 9080, a capacity of 10MB, and is a system using the open-source software Kafka. The throughput information column is mandatory in the queue throughput table 121, but other queue performance information is optional.

[0051] Next, the branch ratio table 122 will be explained with reference to Figure 14. Figure 14 shows an example of a branch allocation table. The branch ratio table shows the proportion 1401 of input data that is processed in each process branched by a branch instruction in the processing description during a certain period. In the example in Figure 14, 30% of the input data during a certain period is determined to be an if-1 branch, and the processing of the branch if-1 destination in the processing description is performed. Here, the period can be arbitrarily set by the user using the user input unit 129. In this example, the branch ratio is obtained by the branch allocation acquisition unit 117. The user may also input a similar ratio 1501, in which case a screen for inputting into the table may be displayed on the user input unit 129 (more specifically, the table input unit 132), as shown in Figure 15.

[0052] Next, the calculation information table 123 will be explained with reference to Figure 16. Figure 16 shows an example of a calculation information table. The calculation information table 123 holds the number of workers 1601, the inflow rate of input data per unit time 1602, and the measurement period 1603. If there are other variable values ​​required in the calculation, this table can also hold them.

[0053] In this embodiment, throughput information for the processing throughput table 120 and queue throughput table 121, the amount of input data flowing into the calculation information table 123, and the ratio information for the branching ratio table 122 are stored when the processing description is read by the queue deployment calculation unit 114, which measures the data using the flowchart shown in Figure 20, which will be described later. In addition to the method shown in Figure 20, a method may be adopted in which throughput information from a previous run on the stream processing system is read. Alternatively, each piece of information may be generated by the user directly inputting it from the user input unit 129.

[0054] The queue deployment system 113 calculates the throughput improvement value for each queue deployment pattern from the processing throughput table 120, the queue throughput table 121, the branch ratio table 122, and the calculation information table 123 using the flowchart shown in Figure 21, which will be described later, and stores it in the queue deployment pattern table (S1005).

[0055] The queue deployment pattern table 124 is described below. Figure 17 shows an example of the queue deployment pattern table. Each column indicates the branching and improved throughput of the priority processing within the processing description, and each row indicates the pattern for deploying the queue. For example, row 1701 shows a pattern in which the queue is not deployed to priority processing and high-load processing, but is deployed to processing B, and in this case, no improvement in throughput can be expected. Row 1702 shows a pattern in which the queue is not deployed to priority processing, but is deployed to processing B and high-load processing, and the improved throughput is 11 (ms). The deployment cost for each deployment combination when a queue is deployed is calculated by the deployment calculation unit 116.

[0056] The queue deployment execution unit 125 determines where to deploy the queue from the queue deployment pattern table. In the example queue deployment pattern table in Figure 17, the deployment cost at row 1702 is 11 ms, which is considered to offer the greatest throughput improvement, so it is decided to deploy the queue to the branch destination of process B and the high-load process (S1006).

[0057] The queue deployment system 113 divides the processing description in the manner determined in step 1006, transfers the processing description to the stream processing 108, and deploys the queue (S1007). In the example queue deployment pattern table in Figure 17, the queue deployment execution unit 125 deploys the queue to the branch instruction destinations that branch to processing B and high-load processing. At this time, the processing description is separated into processing descriptions from processing B onward, processing descriptions from high-load processing onward, and processing descriptions that include other priority processing. Therefore, for example, if the input data moves to a branch destination processing that includes high-load processing, the data at the time of instruction branching is first transferred to the queue and the worker is released. After that, a worker is assigned to the processing description that includes high-load processing, and processing is performed to read the data that is in the process of being processed and stored in the queue.

[0058] The queue deployment execution unit 125 transfers the processing description of the queue deployment result and the queue deployment status to the result display unit 128 of the user output unit 126, and displays information regarding the queue deployment status (S1008). Figure 18 is an example of a screen displaying the processing description and queue deployment status. Here, it is shown that the if-1 queue node 1801 deployed a queue before processing B, and the if-2 queue node 1802 deployed a queue before high-load processing. In addition to the queue deployment status shown in Figure 18, the management table 119 may also be displayed on the table display unit 127 in the user output unit 126, as shown in Figure 19.

[0059] Next, the specific processing of the queue deployment system 113 will be described. Figure 20 is a flowchart showing an example of the processing when the queue deployment calculation unit 114 measures the processing throughput table, queue throughput table, branch ratio table, and calculation information table. This flowchart is executed in step 1004.

[0060] The load measurement unit 115 transfers the processing description to the stream processing system 108 (S2001). Note that this processing description is the same as the one received from the user input unit 129, without any queue deployment.

[0061] The load measurement unit 115 separates each function in the processing description and measures its throughput (S2002). An example of the unit of a function at this time is the function node level designed on the input screen shown in Figure 9. The throughput value of each function is taken as the average value over a certain period. In addition to this, the load measurement unit 115 also measures the amount of data flowing into the stream processing system 108 (S2003). The load measurement unit 115 may also measure the amount of data flowing per unit time during the measurement period.

[0062] The branch allocation acquisition unit 117 measures the proportion of data that will be transferred to each branch in the branch instructions within the processing description during a certain period (S2004). In other words, the branch allocation acquisition unit 117 measures how long each branch destination process will be running per unit of time.

[0063] The load measurement unit 115 transfers data from the stream processing to the queue and measures the throughput until the stream processing receives the data transferred to the queue (S2005).

[0064] The queue allocation calculation unit 114 stores the results measured in steps 2002, 2003, 2004, and 2005 into the processing throughput table 120, the queue throughput table 121, the branch ratio table 122, and the calculation information table 123, respectively (S2006).

[0065] Next, the process related to step 1005 described above will be explained with reference to Figure 21. Figure 21 shows an example of a flowchart of the queueing calculation in step 1005.

[0066] The queue deployment system 113 generates a queue deployment pattern that indicates combinations for deploying queues to branch instructions (S2101). At this time, a queue deployment pattern is generated based on the number of processors performing stream operations, and no queue deployment pattern is generated for branch instructions that have functions designated as priority processing by the user. The number of workers 1601 in the arithmetic information table 123 may also be referenced.

[0067] Here, examples of queue deployment pattern generation when the processing flow shown in Figure 11 is used as input are shown in Figures 22, 23, and 24.

[0068] Figure 22 shows a pattern in which queue 2201 is placed before the high-load process 2202. In this case, the high-load process is separated from the processing description at the time of user input and operates as a separate processing description. Figure 23 shows a pattern in which process B2303 and the high-load process 2304 are separated, and queues 2301 and 2302 are placed before each process. Similar to Figure 22, process B2303 and the high-load process 2304 are treated as separate processing descriptions. Figure 24 shows a pattern in which queue 2401 is placed only before process B2402. Starting with the pattern in Figure 11 where no queues are placed, a list is generated of the three patterns with queue placements described above.

[0069] The queue deployment system 113 calculates the patterns generated in step 2101 in order from the first one (S2102). Here, for each branch instruction, processes without buffers are treated as the throughput value of the process, and processes with buffers are treated as the queue throughput value as the average service rate in the queue (S2103). For example, with the pattern configuration shown in Figure 11 and the throughput shown in Figure 12, the average service rate of priority processes is 1 / 10.

[0070] The queue deployment system 113 treats (input data inflow) × (branching ratio to each branch) as the average arrival rate (S2104). In the pattern shown in Figure 11, with the input data inflow 1602 and measurement time 1603 shown in Figure 16, and the branch assignment table shown in Figure 14, the input data inflow per unit time (ms) is 0.1, and the average arrival rate for branch if-1 is 0.1 × 0.3 = 0.03.

[0071] The queue deployment system 113 calculates the average utilization rate of the entire stream processing from the average of the average service rate and average arrival rate of each processing (S2105). In the example of step 2105 shown in Figure 11, the if-1 branch is 0.03 × 10 = 0.3, and the average of all instruction branches is 1.1.

[0072] The queue deployment system 113 calculates the waiting time at the branch designated for priority processing from the average utilization rate, adds the throughput value of the priority processing, and determines the processing time when priority processing is determined (S2106). In the example following step 2605 as shown in Figure 11, the waiting time increases indefinitely because the average utilization rate exceeds 1. In the example of stream processing configuration shown in Figure 23, based on the processing throughput table shown in Figure 12, the queue throughput table shown in Figure 13, the partition ratio table shown in Figure 14, and the calculation information table shown in Figure 16, the waiting time at the branch designated for priority processing is 11 (ms).

[0073] The queue deployment system 113 records the results in the queue deployment pattern table (S2107). In the example following step 2105, as shown in the queue deployment pattern table in Figure 17, in terms of throughput at the branch designated as priority processing, no improvement in throughput can be expected when a queue is deployed to process B, an improvement to 13 (ms) can be expected when a queue is deployed to the high-load process, and an improvement to 11 (ms) can be expected when a queue is deployed to both process B and the high-load process.

[0074] If the xth value is not greater than or equal to the maximum, the process proceeds to step 2109, and if it is greater than or equal to the maximum, the process ends (S2108). In step 2109, the process of adding 1 to x is performed. Through the above, it is possible to calculate how much the throughput of priority processing is improved compared to Figure 11 in each queue deployment pattern. In this flow, the waiting time (improved throughput) is calculated by the execution of the deployment calculation unit 116.

[0075] By appropriately allocating queues to the processing descriptions entered by the user using the system described above, it is possible to improve the throughput of priority processing in stream processing.

[0076] Next, a second embodiment will be described. Explanations that are similar to those already described may be omitted. <Second Embodiment> Figure 25 shows the system of the second embodiment. Unlike the first embodiment, the configuration of the user output section is omitted, and the configuration of the user input section is changed. In the system shown in Figure 25, the queue deployment system performs queue deployment calculations on the processing descriptions that are running during stream processing, and rearranges the queues and processing descriptions.

[0077] An example of operation in this embodiment will be described with reference to the figures. Figure 26 is a flowchart showing an example of the operation of the queue deployment system for a stream processing in progress.

[0078] The queue deployment system 113a obtains information on the priority output specification 2501 from the user input unit 129a (S2601). Here, the priority output specification is the specification of an output destination to which the stream processing output should be given priority. An example of such an output destination is one where real-time performance is important.

[0079] The queue deployment system 113a reads the processing description currently running in stream processing (S2602). If the processing description contains a branch instruction and the output destination is designated as a priority output, the process moves to step 2604 (S2603). Otherwise, the process ends. The queue deployment system 113a designates the function of the branch instruction destination whose output destination is designated as a priority output as the priority processing (S2604). The queue deployment system 113a moves to step 1004 in Figure 10 (S2605). In the subsequent steps 1004 and beyond in Figure 10, the queue is deployed in the same manner as in the first embodiment.

[0080] According to the above embodiment, for processes with branch instructions within stream processing, the system specifies important branch destination processes where throughput reduction should not be prevented, calculates the throughput of each branch destination process when a queue is placed for each branch destination process excluding the important branch destination process, based on the data inflow per unit time, the branch allocation ratio, the throughput in each branch process, the throughput when a queue is placed, and the number of workers, creates a queue deployment pattern table summarizing the calculation results, and provides a system that places a queue immediately after the branch process using the lowest cost pattern in the queue deployment pattern table. This system enables queue deployment that improves the throughput of important processes requiring real-time processing. Therefore, by appropriately improving stream processing, an increase in the number of data processes can be expected, and it is believed that economically sound data processing can be achieved.

[0081] The embodiments have been described above, but these embodiments are illustrative examples for illustrating the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The positions of the various components shown in the drawings may not represent their actual locations in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions disclosed in the drawings. Examples of various types of information may be described using terms such as "table" and "list," but these types of information may also be represented by other data structures. For example, various types of information such as "XX table" and "XX list" may also be referred to as "XX information." When describing identification information, terms such as "identification information," "identifier," "name," "ID," and "number" are used, and these terms are interchangeable. The system of the present invention may consist of, for example, one or more computers, as long as it can perform appropriate processing.

[0082] The queue deployment system performs predetermined processing by having its processor 203 execute various programs. Here, the queue deployment calculation unit 114, load measurement unit 115, deployment calculation unit 116, branch allocation acquisition unit 117, priority processing designation unit 118, and queue deployment execution unit 125 are programs and may be stored in an appropriate storage device of the queue deployment system. In addition, other programs that perform predetermined processing (such as processing related to the processing flow described above) may be stored in an appropriate storage device of the queue deployment system. Furthermore, the queue deployment system may be equipped with an interface for inputting and outputting data to an external storage device. The queue deployment system may also execute processing using the external storage device that stores the programs. [Explanation of Symbols]

[0083] 113 Queue Deployment System 126 User Output Section 127 Table display section 128 Result display section 129 User Input Section 130 Processing Description Input Section 131 Priority Processing Input Section 132 Table Input Section

Claims

1. A data processing management system, The aforementioned data processing management system is The system includes a queue deployment system that deploys queues using a processing description of a stream process having branch instructions as input. The aforementioned queue deployment system, Specify the process to be executed preferentially for the aforementioned processing description, The throughput in each process after the aforementioned branch instruction is measured, The branch ratio, which is the proportion of each process after the aforementioned branch instruction that is executed, is obtained. Using the throughput and the branch ratio, the deployment cost when a queue is deployed between the branch instruction and the processing after the branch instruction is calculated for each combination of processing after the branch instruction. Among the processes after the aforementioned branch instruction, the system outputs a queue deployment pattern that prevents a decrease in throughput for priority processing. The aforementioned queue deployment system, Based on the aforementioned deployment cost, the location where the queue will be deployed is determined from the aforementioned queue deployment pattern. Divide the processing description and deploy the queue. A data processing management system characterized by the following features.

2. A data processing management system according to claim 1, The aforementioned queue deployment system, It is connected to a stream processing system that has a finite number of processors, in which the processors process data input in stream format based on a processing description. The aforementioned queue deployment system, It is equipped with a user input section where the user enters information, The aforementioned queue deployment system, A processing description is obtained from the user input unit. A data processing management system characterized by the following features.

3. A data processing management system according to claim 1, The aforementioned queue deployment system, The deployment cost is calculated using the data inflow per unit time and the number of processors operating in stream mode. A data processing management system characterized by the following features.

4. A data processing management system according to claim 3, The aforementioned queue deployment system, Based on the number of processors, multiple patterns are generated for assigning queues to branch instructions. Select one pattern from the multiple patterns that were generated. (1) For each process after a branch instruction, for processes without a buffer, the average service rate is calculated based on the throughput value of the process, and for processes with a buffer, the average service rate is calculated based on the throughput value of the queue. (2) The average arrival rate based on the data inflow amount × the branching rate is calculated for each process after the branching instruction. (3) The average utilization rate of the entire stream processing is calculated from the average of the average service rate × the average arrival rate for each process after the branch instruction. (4) The deployment cost is calculated by adding the throughput of priority processing to the average utilization rate. By selecting each generated pattern and performing the processes described in (1) to (4) above on all patterns, a queue deployment pattern table is generated that summarizes the deployment costs for each queue deployment pattern. A data processing management system characterized by the following features.

5. A data processing management system according to claim 1, The aforementioned queue deployment system, In addition to the throughput of each process after a branch instruction, the throughput when transferring data to the queue is also measured. A data processing management system characterized by the following features.

6. A data processing management system according to claim 1, The aforementioned queue deployment system, In obtaining the aforementioned branch ratio, the amount of time each branched process takes to run per unit of time is measured. A data processing management system characterized by the following features.

7. A data processing management system according to claim 1, It is equipped with a user output unit that presents information to the user, The user output unit is, Displays information about the status of the queue deployment. A data processing management system characterized by the following features.

8. A data processing management system according to claim 2, The user input unit is, By connecting nodes that represent processing at the edge, a screen for inputting the processing description is output. A data processing management system characterized by the following features.

9. A data processing management system according to claim 8, The user input unit is, If a node indicating a process is selected, a screen is displayed specifying that the process indicated by the node should be given priority. A data processing management system characterized by the following features.

10. A data processing and management method using a processor, Obtain the processing description of a stream process that has branch instructions, Specify the process to be executed preferentially for the aforementioned processing description, The throughput in each process after the aforementioned branch instruction is measured, The branch ratio, which is the proportion of each process after the aforementioned branch instruction that is executed, is obtained. Using the throughput and the branch ratio, the deployment cost when a queue is deployed between the branch instruction and the processing after the branch instruction is calculated for each combination of processing after the branch instruction. Among the processes after the aforementioned branch instruction, the system outputs a queue deployment pattern that prevents a decrease in throughput for priority processing. Based on the aforementioned deployment cost, the location where the queue will be deployed is determined from the aforementioned queue deployment pattern. Divide the processing description and deploy the queue. A data processing and management method characterized by the following.

Images