Computer system, and method for controlling data pipeline processing

Abstract

To control a data pipeline so as to complete a process within restriction time, without imparting a change or an influence to an existing data pipeline, when adding a new data pipeline.SOLUTION: A computer system comprises: a data lake server; a database server; and a plurality of data pipeline servers for executing data pipeline processing. The plurality of data pipeline servers includes a control data pipeline server and a non-control data pipeline server. The computer system includes a control information generation section. The control information generation section calculates a control parameter with which a load of the database server is smaller than a prescribed threshold and by which the data pipeline processing can be terminated within restriction time, on the basis of load information acquired from the database server and progress information of the data pipeline processing of the control data pipeline server, and then, transmits the control parameter to the control data pipeline servers.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technology for controlling a data pipeline in an IoT system. [Background technology]

[0002] IoT systems that handle IoT data have data pipelines that execute a data curation process to extract and process data from a data lake that stores data acquired from IoT devices and store it in a database. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-139522 Summary of the Invention [Problem to be solved by the invention]

[0004] When adding a new data pipeline to an IoT system, it is required that the existing data pipelines are not changed or affected, while the data pipelines must complete their processes within a time constraint.

[0005] If a data pipeline writes a large amount of data to a database in order to complete a process within a time constraint, the load on the database increases, causing retries and timeouts. This affects other data pipelines. On the other hand, if the amount of data written to the database by a data pipeline is restricted too much to prevent the load on the database from increasing, the process cannot be completed within the time constraint. Therefore, it is necessary to adjust the processing volume of the data pipeline and the load on the database.

[0006] Patent Document 1 describes a known technology for adjusting processing executed in a system. Patent Document 1 describes the following: "An information processing device has an observation unit, a prediction unit, an adjustment unit, and an instruction unit. The observation unit observes actual load measurements of components to be changed that have inter-component dependencies among multiple components arranged on a distributed processing system. The prediction unit calculates load prediction values ​​until change processing of all components to be changed is completed based on the actual load measurements of the components to be changed. When the predicted load value exceeds a threshold, the adjustment unit adjusts the transmission interval of change instructions that instruct the pre-change target components to start change processing. The instruction unit issues the change instructions to the pre-change target components to be changed based on the transmission interval adjusted by the adjustment unit."

[0007] The technique described in Patent Document 1 does not take into consideration the constraints on the components to be changed, so it is difficult to solve the above-mentioned problems even if the technique described in Patent Document 1 is used. [Means for solving the problem]

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a computer system that handles IoT data includes: a data lake server that manages a data lake that stores IoT data; a database server that manages a database that stores the IoT data that has undergone data processing; and a plurality of data pipeline servers that execute data pipeline processing that acquires the IoT data from the data lake, executes the data processing on the IoT data, and writes the processed IoT data to the database, wherein the plurality of data pipeline servers include a control data pipeline server that can control the data pipeline processing and a non-control data pipeline server that cannot control the data pipeline processing, and the computer system has a control information generation unit that generates control information for adjusting control parameters for controlling the data pipeline processing of the control data pipeline server, and the control information generation unit acquires load information from the database server and acquires progress information regarding a time constraint and progress status of the data pipeline processing from the control data pipeline server, calculates the control parameter or a range of the control parameter such that the load of the database server is smaller than a predetermined threshold and the data pipeline processing can be completed within the time constraint based on the load information and the progress status, generates the control information including the control parameter or the range of the control parameter, and transmits the control information to the control data pipeline server. [Effects of the Invention]

[0009] According to the present invention, the data pipeline processing of the control data pipeline server can be adjusted so that the processing is completed within the time constraint without affecting the non-control data pipeline server. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of the configuration of an IoT system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration and a processing flow of an IoT system according to a first embodiment. [Figure 3] FIG. 2 illustrates an example of a hardware configuration of a data pipeline server according to the first embodiment. [Figure 4] FIG. 4 is a sequence diagram illustrating a processing flow in a control data pipeline server according to the first embodiment. [Figure 5] 10 is a flowchart illustrating an example of a process executed by a control information generating unit of the adjustment unit according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of load information according to the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of progress information according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example of control parameter information according to the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a screen presented by an input / output unit according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a functional configuration and a processing flow of an IoT system according to a second embodiment. [Figure 11] 10 is a flowchart illustrating an example of a process executed by a control information generating unit of the adjustment unit according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.

[0012] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant explanations will be omitted.

[0013] In this specification, the terms "first," "second," "third," etc. are used to identify components and do not necessarily limit the number or order. [Example]

[0014] FIG. 1 is a diagram illustrating a configuration example of an IoT system according to a first embodiment.

[0015] The IoT system includes a data lake server 110, a data pipeline server 120, a database server 130, and a data analysis server 140. The data lake server 110 and the data pipeline server 120 are connected via a network 150-1. The data pipeline server 120 and the database server 130 are connected via a network 150-2. The database server 130 and the data analysis server 140 are connected via a network 150-3. The networks 150-1, 150-2, and 150-3 are LANs (Local Area Networks) or the like.

[0016] The data lake server 110 stores IoT data acquired from IoT devices. The data pipeline server 120 executes data pipeline processing. The data pipeline processing includes acquisition of IoT data from the data lake server 110, data processing such as sorting, extraction, conversion, and integration of the IoT data, and writing of data generated by the data processing to the database server 130. The data analysis server 140 executes various data analyses using the data stored in the database server 130.

[0017] FIG. 2 is a diagram illustrating the functional configuration and processing flow of the IoT system according to the first embodiment.

[0018] The data lake server 110 has a data lake management unit 111 and a data lake 112 that stores IoT data acquired from IoT devices. The data lake management unit 111 stores the IoT data acquired from the IoT devices in the data lake 112, and also reads the IoT data from the data lake 112 in response to a request from the data pipeline server 120 and transmits it to the data pipeline server 120.

[0019] In the IoT system, there are a control data pipeline server 120 that can control data pipeline processing, and a non-control data pipeline server 120 that cannot control data pipeline processing. When there is no need to distinguish between the control data pipeline server 120 and the non-control data pipeline server 120, they are referred to as data pipeline servers 120.

[0020] The control data pipeline server 120 includes a data pipeline processing unit 121 and an adjustment unit 122 .

[0021] The data pipeline processing unit 121 executes data pipeline processing. The data pipeline processing unit 121 includes an acquisition unit 123, an allocation unit 124, a write / data processing unit 125, a control unit 126, and an evaluation unit 127. There may be a plurality of acquisition units 123, allocation units 124, and write / data processing units 125.

[0022] The acquisition unit 123 is a module that executes IoT data acquisition processing and acquires IoT data from the data lake server 110. The distribution unit 124 is a module that executes IoT data distribution processing and distributes the IoT data to the writing / data processing unit 125. For example, the distribution unit 124 distributes the IoT data based on the type of IoT data or the load of the writing / data processing unit 125. The writing / data processing unit 125 is a module that executes data processing and writing processing, executes data processing, and writes data generated by the data processing to the database server 130.

[0023] The control unit 126 controls each of the acquisition unit 123, the allocation unit 124, and the writing / data processing unit 125 based on the control information generated by the adjustment unit 122. The evaluation unit 127 evaluates the progress of the processing by the data pipeline processing unit 121. The evaluation unit 127 outputs the result of the evaluation to the adjustment unit 122 as progress data.

[0024] The adjusting unit 122 generates control information for adjusting control parameters for controlling the data pipeline processing. The adjusting unit 122 includes a load monitoring unit 128 and a control information generating unit 129.

[0025] The load monitoring unit 128 monitors the load on the database server 130. The load monitoring unit 128 acquires information related to the load on the database server 130, and also acquires information related to the processing load on the database management unit 131 via the agent 133. The load monitoring unit 128 outputs the results of the monitoring as load data.

[0026] The control information generator 129 calculates a control parameter or a range of the control parameter based on the progress data and the load data, and generates control information including the control parameter or the range of the control parameter. The control information generator 129 also generates a history that records the calculation results of the control parameter. Note that the control information generator 129 may calculate the control parameter or the range of the control parameter taking the history into consideration.

[0027] The non-control data pipeline server 120 includes a data pipeline processing unit 121 .

[0028] The data pipeline processing unit 121 includes an acquisition unit 123, a distribution unit 124, and a writing / data processing unit 125. There may be a plurality of acquisition units 123, distribution units 124, and writing / data processing units 125.

[0029] Since the non-control data pipeline server 120 does not control data pipeline processing, the data pipeline processing unit 121 does not include a control unit 126 or an evaluation unit 127. In addition, the non-control data pipeline server 120 does not include an adjustment unit 122.

[0030] The database server 130 includes a database management unit 131 and an agent 133 , and also holds a database 132 .

[0031] The database management unit 131 stores the data written by the data pipeline server 120 in the database 132. Furthermore, when the database management unit 131 receives a request from the data analysis server 140, it reads data from the database 132 and transmits the data to the data analysis server 140. The agent 133 monitors the processing load of the database management unit 131.

[0032] FIG. 3 is a diagram illustrating an example of a hardware configuration of the data pipeline server 120 according to the first embodiment.

[0033] The data pipeline server 120 includes a processor 301 , a memory 302 , a non-volatile storage device 303 , a network interface 304 , an input device 305 , and an output device 306 .

[0034] The processor 301 reads a program from the nonvolatile storage device 303, loads it into the memory 302, and executes it. The processor 301 executes processing in accordance with the program, thereby operating as a functional unit that realizes a specific function.

[0035] The memory 302 stores the programs executed by the processor 301 and data used by the programs.

[0036] The nonvolatile storage device 303 is an HDD (Hard Disk Drive) or the like, and stores a data pipeline processing program 311, an adjustment program 312, and an input / output program 313. The data pipeline processing program 311 is a program that realizes the data pipeline processing unit 121, and the adjustment program 312 is a program that realizes the adjustment unit 122. The input / output program 313 is a program that realizes an input / output unit (not shown) that provides a GUI or API for operating the data pipeline server 120.

[0037] The network interface 304 connects to a network. The input device 305 is a keyboard, a mouse, a touch panel, etc. The output device 306 is a display, etc.

[0038] Although the data pipeline server 120 is implemented using one computer, the present invention is not limited to this. For example, the data pipeline server 120 may be implemented using multiple computers, or may be implemented using a virtual computer or a container.

[0039] FIG. 4 is a sequence diagram illustrating the flow of processing in the control data pipeline server 120 according to the first embodiment.

[0040] The control unit 126 transmits a control instruction including a control parameter to the acquisition unit 123, the distribution unit 124, and the writing / data processing unit 125 (S101, S102, S103).

[0041] The acquisition unit 123 executes a data acquisition process (S104). In the data acquisition process, the acquisition unit 123 acquires a predetermined number of IoT data at a predetermined acquisition interval from the data lake server 110. The acquisition interval and the number of IoT data to be acquired can be considered as control parameters for the acquisition unit 123.

[0042] The acquisition unit 123 divides the acquired IoT data group (S105), and outputs a distribution request including the divided IoT data group to the distribution unit 124 (S106).

[0043] The allocating unit 124 executes the allocating process (S107) and outputs a write request including the allocated IoT data group to the writing / data processing unit 125 (S108). The control parameter of the allocating unit 124 may be an allocating rule.

[0044] The write / data processing unit 125 executes data processing (S109). Thereafter, the write / data processing unit 125 executes writing processing (S110). If the data writing fails, the write / data processing unit 125 executes the writing processing again. If the data writing is not completed within a certain period of time, the write / data processing unit 125 outputs a re-sorting request including the IoT data group for which writing failed to the sorting unit 124 (S111). A possible control parameter for the write / data processing unit 125 is the retry interval for the write processing. In addition, the number of write / data processing units 125 can also be set as a control parameter.

[0045] The control unit 126 monitors the processing status of the acquisition unit 123, the distribution unit 124, and the writing / data processing unit 125, and outputs processing information including the current processing status to the evaluation unit 127 (S112). The evaluation unit 127 generates progress information based on the processing information (S113), and outputs it to the adjustment unit 122. Possible information indicating the processing status of the acquisition unit 123, the distribution unit 124, and the writing / data processing unit 125 includes the number of IoT data acquired by the acquisition unit 123, the number of data written by the writing / data processing unit 125, and the number of IoT data that the distribution unit 124 has not distributed.

[0046] The series of processes executed by the acquisition unit 123, the distribution unit 124, and the control unit 126 corresponds to data pipeline processing. The data pipeline processing is executed periodically. The control unit 126 outputs a control instruction when the data pipeline server 120 is started or when control information is received.

[0047] Next, the processing of the adjustment unit 122 will be described.

[0048] The load monitor 128 of the adjuster 122 periodically acquires information about the load of the database server 130 from the database server 130 and generates load information.

[0049] Fig. 5 is a flowchart illustrating an example of processing executed by the control information generating unit 129 of the adjusting unit 122 according to the first embodiment. Fig. 6 is a diagram illustrating an example of load information according to the first embodiment. Fig. 7 is a diagram illustrating an example of progress information according to the first embodiment. Fig. 8 is a diagram illustrating an example of control parameter information according to the first embodiment.

[0050] The control information generator 129 periodically executes the process described below.

[0051] The control information generator 129 reads out the load information and the progress information (S201). Note that the control information generator 129 may also read out the history as necessary.

[0052] 6, the load information includes the processes being executed by the database server 130, the available resource capacity, the resource usage rate, etc. Also, as shown in Fig. 7, the progress information includes the time constraint of the data pipeline processing, the number of times the data pipeline processing is executed, the amount of resources used, etc.

[0053] The control information generation unit 129 predicts the load of the database server 130 based on the load information (S202). The load of the database server 130 is predicted using a prediction model generated by machine learning. The load information is input to the prediction model. The prediction model predicts the load from the current processing time t to t+Δt, with the start time of the data pipeline processing as the reference.

[0054] The control information generator 129 calculates a processing progress index based on the progress information (S203). For example, the control information generator 129 calculates the progress rate of the processing as the processing progress index.

[0055] The control information generator 129 calculates a control parameter or a range of the control parameter based on the load information, the load prediction, and the processing progress index (S204).

[0056] Here, an example of a method for calculating the range of the control parameter will be described.

[0057] (Step 1-1) Based on the load information, the control information generator 129 calculates the maximum throughput H_max(t) that the control data pipeline server 120 can apply to the database server 130. Note that t is a variable that represents the elapsed time from the start of data pipeline processing. For example, when the CPU utilization rate of the database server 130 is used as an index, the control information generator 129 calculates the amount of free resources by subtracting the current CPU utilization rate from the maximum value of the CPU utilization rate, and multiplies the amount of free resources by a constant to calculate the maximum throughput.

[0058] Furthermore, the control information generator 129 performs a similar process based on the load prediction to calculate the maximum throughput H_max(t+Δt).

[0059] (Step 1-2) The control information generator 129 calculates a baseline throughput H_baseline(t) based on the processing progress indicator and the remaining processing time. The baseline throughput H_baseline(t) is the throughput required to complete the data pipeline processing within the remaining processing time. For example, the baseline throughput H_baseline(t) is calculated using equation (1).

[0060]

number

[0061] Here, D(t) represents the amount of unprocessed data, and T represents the time constraint for data pipeline processing. The baseline throughput H_baseline(t) may be calculated using equation (2) which introduces a margin factor r. The margin factor r can be set arbitrarily.

[0062]

number

[0063] Furthermore, the control information generator 129 performs the same process to calculate the baseline throughput H_baseline(t+Δt). Note that D(t+Δt) can be calculated based on the current throughput.

[0064] (Step 1-3) The control information generator 129 calculates the target throughput H_target(t) based on the maximum throughput H_max(t) and the baseline throughput H_baseline(t). For example, a formula such as Equation (3) can be used.

[0065]

number

[0066] Here, the function f is a function that takes into account load fluctuations and fluctuations in the amount of work to be executed within the remaining processing time (remaining workload), such as the amount of fluctuation in the integral (area) of the difference between the maximum throughput H_max(t) and the baseline throughput H_baseline(t) over any time interval including time t. The integral value represents the degree of pressure between the most recent load on the database server 130 and the remaining workload. The function f is defined so that when the load on the database server 130 is heavy, the value of the function f becomes small, and when the remaining workload is large, the value of the function f becomes large.

[0067] Moreover, the control information generator 129 executes the same process to calculate the target throughput H_target(t+Δt).

[0068] (Step 1-4) The control information generator 129 calculates the number of data acquisitions N(t), which is a control parameter, based on the target throughput H_target(t). For example, this can be calculated using equation (4). Here, T_data represents the data acquisition interval.

[0069]

number

[0070] Furthermore, the control information generator 129 calculates the number of data acquisitions N(t+Δt) based on the target throughput H_target(t+Δt).

[0071] (Step 1-5) The control information generator 129 calculates a range determined by the number of acquired data N(t) and N(t+Δt). The data pipeline processor 121 can adjust the control parameters within this range.

[0072] This is an example of a method for calculating the range of the control parameter. If there is no load prediction, only the number of acquired data N(t), which is a control parameter, is calculated.

[0073] As described above, the control information generator 129 calculates an upper limit value of an index representing the processing performance of data pipeline processing based on load information. The control information generator 129 also calculates a lower limit value of the index based on progress information and the time constraint of the data pipeline processing. Furthermore, the control information generator 129 calculates a target value of the index based on the upper and lower limits of the index, and calculates a value or range of a control parameter based on the target value of the index.

[0074] The control information generator 129 generates control information including the control parameters or the ranges of the control parameters, and outputs the control information to the data pipeline processor 121 (S205).

[0075] As shown in FIG. 8, the control information includes information on control parameters such as the acquisition interval, the number of pieces of data to be acquired, and the retry interval for the write process.

[0076] The control information generator 129 generates a history (S206) and ends the process.

[0077] FIG. 9 is a diagram illustrating an example of a screen presented by the input / output unit according to the first embodiment.

[0078] The screen 900 displays information about the prediction model, the progress of the data pipeline processing, the load on the database server 130, and the like.

[0079] The control data pipeline server 120 can control the data pipeline processing in real time, taking into consideration the load on the database server 130 and the time constraints of the data pipeline processing. This reduces the impact on the non-control data pipeline servers 120, and also enables the data pipeline processing of the control data pipeline server 120 to be executed appropriately.

[0080] In this control, there is no need to know the progress of the processing of the non-control data pipeline server 120, and therefore there is no need to connect to the non-control data pipeline server 120 or change the configuration of the non-control data pipeline server 120. [Example]

[0081] In the second embodiment, a control server controls a plurality of control data pipeline servers 120 in an IoT system. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0082] FIG. 10 is a diagram illustrating the functional configuration and processing flow of the IoT system according to the second embodiment.

[0083] The IoT system of the second embodiment includes a control server 160. The control server 160 is connected to a plurality of control data pipeline servers 120.

[0084] The control data pipeline server 120 in the second embodiment does not include the adjustment unit 122. In the second embodiment, the control server 160 includes the adjustment unit 122.

[0085] The processing executed by the data pipeline processing unit 121 of the control data pipeline server 120 in the second embodiment is the same as that in the first embodiment.

[0086] FIG. 11 is a flowchart illustrating an example of processing executed by the control information generator 129 of the adjuster 122 according to the second embodiment.

[0087] The control information generation unit 129 reads out the load information and the progress information acquired from each of the control data pipeline servers 120 (S301). Note that the control information generation unit 129 may also read out the history as necessary.

[0088] The control information generator 129 predicts the load of the database server 130 based on the load information (S302).

[0089] The control information generation unit 129 calculates a processing progress index for each control data pipeline server 120 based on the progress information of each of the control data pipeline servers 120 (S303). For example, the control information generation unit 129 calculates the progress rate of the processing as the processing progress index.

[0090] The control information generator 129 calculates a control parameter or a range of the control parameter based on the load information, the load prediction, and the processing progress index (S304).

[0091] Here, an example of a method for calculating the number of pieces of data to be acquired, which is one of the control parameters, will be described, taking two data pipeline servers 120 as an example.

[0092] (Step 2-1) Based on the load information, the control information generator 129 calculates the maximum throughput H_max(t) that each control data pipeline server 120 can apply to the database server 130. Step 2-1 is the same as step 1-1.

[0093] (Step 2-2) The control information generator 129 executes the process of step 1-2 for each data pipeline processor 121, and calculates the baseline throughputs H_baseline1(t) and H_baseline2(t).

[0094] (Step 2-3) The control information generator 129 calculates the target throughputs H_target1(t) and H_target2(t) based on the maximum throughput H_max(t) and the baseline throughputs H_baseline1(t) and H_baseline2(t). For example, the calculations can be performed using equations (5) and (6).

[0095]

number

[0096]

number

[0097] The function f is a function that takes into account load fluctuations and fluctuations in the workload over the remaining processing time, such as the fluctuation V_1 in the integral (area) of the difference between the maximum throughput H_max(t) and the baseline throughput H_baseline1(t) and the fluctuation V_2 in the integral (area) of the difference between the maximum throughput H_max(t) and the baseline throughput H_baseline2(t). The fluctuations V_1 and V_2 represent the current load on the database server 130 and the degree of pressure on the remaining workload.

[0098] More specifically, the control information generator 129 calculates the target throughput using equation (8) under the constraint of equation (7).

[0099]

number

[0100]

number

[0101] The above is an explanation of an example of a method for calculating the control parameters.

[0102] The control information generator 129 generates control information including the control parameters or the ranges of the control parameters, and outputs the control information to the data pipeline processor 121 (S305).

[0103] The control information generator 129 generates a history (S306) and ends the process.

[0104] The control server 160 can control the data pipeline processing of each control data pipeline server 120 according to the load of the database server 130 and the time constraints and progress of the data pipeline processing of the multiple control data pipeline servers 120 .

[0105] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0106] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0107] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java (registered trademark).

[0108] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

[0109] In the above-described embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected. [Explanation of symbols]

[0110] 110 Data Lake Server 111 Data Lake Management Department 112 Data Lake 120 Data Pipeline Server 121 Data Pipeline Processing Unit 122 Adjustment section 123 Acquisition Department 124 Sorting section 125 Write / Data Processing Section 126 Control Unit 127 Evaluation Department 128 Load monitoring section 129 Control Information Generation Unit 130 Database Server 131 Database Management Department 132 databases 133 Agent 140 Data Analysis Server 150 Network 160 Control Server 301 processor 302 memory 303 Non-volatile storage devices 304 Network Interface 305 Input Device 306 Output Device 311 Data Pipeline Processing Program 312 Adjustment Program 313 Input / Output Program 900 screens

Claims

1. A computer system that handles IoT data, a data lake server that manages a data lake that stores IoT data; a database server that manages a database that stores the IoT data that has undergone data processing; a plurality of data pipeline servers that execute data pipeline processing to acquire the IoT data from the data lake, execute the data processing on the IoT data, and write the processed IoT data to the database; Equipped with the plurality of data pipeline servers include a control data pipeline server capable of controlling the data pipeline processing and a non-control data pipeline server not capable of controlling the data pipeline processing; the computer system includes a control information generation unit that generates control information for adjusting control parameters for controlling the data pipeline processing of the control data pipeline server, The control information generation unit Obtaining load information from the database server; obtaining progress information relating to a time constraint and a progress status of the data pipeline processing from the control data pipeline server; calculating the control parameter or a range of the control parameter that allows the load of the database server to be smaller than a predetermined threshold and the data pipeline processing to be completed within a time constraint based on the load information and the progress status, and generating the control information including the control parameter or the range of the control parameter; a computer system that transmits the control information to the control data pipeline server;

2. 2. The computer system of claim 1, The control information generation unit calculating a first index representing the processing performance of the data pipeline processing when the load on the database server becomes smaller than a predetermined threshold based on the load information; calculating a second index representing a processing performance of the data pipeline processing required to complete the data pipeline processing within the time constraint based on the progress status; A computer system that calculates the control parameter or a range of the control parameter based on the first index and the second index.

3. 3. The computer system according to claim 2, a control server connected to a plurality of the control data pipeline servers and having the control information generation unit; The control information generation unit calculating the second indicator for each of the plurality of control data pipeline servers based on the load information and the progress information for each of the plurality of control data pipeline servers; A computer system comprising: a computer that calculates, for each of the plurality of control data pipeline servers, the control parameter or a range of the control parameter based on the first index and a plurality of the second indexes.

4. 3. The computer system according to claim 2, A computer system characterized in that the control information generation unit is included in the control data pipeline server.

5. 3. The computer system according to claim 2, The data pipeline processing includes: a first task of acquiring the IoT data from the data lake; a second task of performing the data processing on the IoT data and writing the processed IoT data to the database; a third task that distributes the IoT data acquired by the first task to the second task; Including, The computer system is characterized in that the control information generation unit calculates the control parameter or a range of the control parameter for controlling at least one of the first task and the third task.

6. 6. The computer system according to claim 5, The control information generation unit calculates at least one of the parallel number or range of parallel numbers of the second task, and the waiting time or range of waiting time for re-executing the second task if writing of the IoT data for which the data processing has been performed fails.

7. A method for controlling data pipeline processing in a computer system that handles IoT data, comprising: The computer system a data lake server that manages a data lake that stores IoT data; a database server that manages a database that stores the IoT data that has undergone data processing; a plurality of data pipeline servers that execute data pipeline processing to acquire the IoT data from the data lake, execute the data processing on the IoT data, and write the processed IoT data to the database; Including, the plurality of data pipeline servers include a control data pipeline server capable of controlling the data pipeline processing and a non-control data pipeline server not capable of controlling the data pipeline processing; the computer system includes a control information generation unit that generates control information for adjusting control parameters for controlling the data pipeline processing of the control data pipeline server, The method for controlling data pipeline processing includes: a first step in which the control information generation unit acquires load information from the database server; a second step in which the control information generation unit acquires progress information relating to a time constraint and a progress status of the data pipeline processing from the control data pipeline server; a third step in which the control information generation unit calculates, based on the load information and the progress status, the control parameters or a range of the control parameters that will cause the load of the database server to be smaller than a predetermined threshold and allow the data pipeline processing to be completed within a time constraint, and generates the control information including the control parameters or the range of the control parameters; a fourth step in which the control information generation unit transmits the control information to the control data pipeline server; 1. A method for controlling data pipeline processing, comprising:

8. 8. A method for controlling data pipeline processing according to claim 7, comprising: The third step includes: a fifth step in which the control information generation unit calculates, based on the load information, a first index representing processing performance of the data pipeline processing in which the load applied to the database server becomes smaller than a predetermined threshold; a sixth step in which the control information generation unit calculates, based on the progress status, a second index representing processing performance of the data pipeline processing required to complete the data pipeline processing within the time constraint; and a seventh step in which the control information generation unit calculates the control parameter or a range of the control parameter based on the first index and the second index.

9. 9. A method for controlling data pipeline processing according to claim 8, comprising: the computer system includes a control server connected to a plurality of the control data pipeline servers and having the control information generation unit; the sixth step includes a step in which the control information generating unit calculates the second indicator for each of the plurality of control data pipeline servers based on the load information and the progress information for each of the plurality of control data pipeline servers; The seventh step of the control method for data pipeline processing is characterized in that it includes a step in which the control information generation unit calculates the control parameter or the range of the control parameter for each of the plurality of control data pipeline servers based on the first index and a plurality of the second indexes.

10. 9. A method for controlling data pipeline processing according to claim 8, comprising: The control method for data pipeline processing, wherein the control information generation unit is included in the control data pipeline server.

11. 9. A method for controlling data pipeline processing according to claim 8, comprising: The data pipeline processing includes: a first task of acquiring the IoT data from the data lake; a second task of performing the data processing on the IoT data and writing the processed IoT data to the database; a third task that distributes the IoT data acquired by the first task to the second task; Including, The seventh step of the control method for data pipeline processing includes a step in which the control information generation unit calculates the control parameter or a range of the control parameter for controlling at least one of the first task and the third task.

12. 12. A method for controlling data pipeline processing according to claim 11, comprising: The seventh step is a control method for data pipeline processing, characterized in that it includes a step in which the control information generation unit calculates at least one of the parallel number or range of parallel numbers of the second task, and the waiting time or range of waiting time for re-executing the second task if writing of the IoT data for which the data processing has been performed fails.

Images