Data processing system and data processing method

Abstract

To provide a data processing system capable of achieving data processing setting changes without interruption in multi-stage data processing, and a data processing method.SOLUTION: In the data processing system, a data control management unit changes some data processing settings or creates new data processing settings without stopping multi-stage data processing by starting the data processing unit necessary for data processing and stopping the data processing unit unnecessary for data processing. When performing multi-stage data processing, the subsequent data processing unit determines the data processing unit that has performed the data processing by reading the tag assigned in the previous data processing unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a data processing system and a data processing method. [Background technology]

[0002] One way to utilize IoT (Internet of Things) is to improve business operations by collecting data from various locations, from devices located at the edge of factory sites to machines on cloud services, and then processing and analyzing the data.To utilize IoT efficiently, a data collection platform is required to collect, process, and store data that can accommodate differences in data formats at each location and fluctuations in data volume.

[0003] In a data collection platform, data processing is carried out in multiple stages. For example, it is possible to separate data processing units such as data collection, data processing, and data storage into data processing units, and then connect multiple data processing units to carry out data processing. Data processing units can be implemented in various forms, such as VMs (Virtual Machines), containers, and processes.

[0004] Dividing a data processing flow into multiple data processing units makes it easy to scale out only processes with high data processing loads or to branch off from a single data processing result and perform data processing. In this specification, a data processing flow refers to a series of operations that acquire data from a data source, process the data, and store it in a data sink. Each data processing unit is connected via a message queue. Using a message queue makes it possible to ensure data availability and control the order of data. When the scale of data collection increases and a large number of data processing flows need to be handled, operating the data processing units for all flows independently consumes more computer resources than necessary. Therefore, a data collection platform is required to execute multiple data processing operations for different data processing flows in parallel within a single data processing unit.

[0005] In a situation where data processing units that simultaneously execute multiple data processing flows are connected in multiple stages to process data, it is considered that some of the data processing settings may need to be changed due to changes in the business operations for which data is collected or changes in the information to be acquired.In one data processing unit, data processing settings related to data processing flows that need to be changed and data processing settings related to data processing flows that do not need to be changed are packaged.

[0006] When changing data processing settings, it is generally necessary to temporarily suspend data processing, which poses a problem in data processing units that include multiple data processing settings, as this also stops data processing for data processing flows that do not need to be changed.When performing data processing for purposes that require real-time performance, a system is required that can continue data processing without being affected by changes to the data processing settings of other data processing flows.

[0007] The technology described in Patent Document 1 is known as a technology for non-disruptively updating data processing settings in data processing. Patent Document 1 describes that "a management system manages to which virtual machine a request is to be transferred from a load balancer, and when the setting for a device that accepts requests from a client is changed from a load balancer of a first processing environment to a device of a second processing environment, the management system manages to transfer requests from the load balancer of the first processing environment to a virtual machine of the second processing environment, and manages to prevent requests from being transferred from the load balancer of the first processing environment to a virtual machine of the first processing environment." [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 2018-88114 Summary of the Invention [Problem to be solved by the invention]

[0009] The technology described in Patent Document 1 shows a system upgrade method using Blue-Green deployment, which starts two systems, an old system and a new system, and switches between the data processing units by placing a switch in front of the data processing unit, and is an invention that makes it possible to execute a request from a client on a device after the update even if the request was sent to a device before the update.By using the technology described in Patent Document 1, it is possible to update data processing settings.

[0010] However, when the data processing portion is multi-stage, unless it is possible to know whether the data processing has been executed in the data processing unit of the preceding stage using the old data processing setting or the new data processing setting, it is impossible to determine whether the data processing should be executed in the data processing unit of the subsequent stage using the old data processing setting or the new data processing setting. In applications where multi-stage data processing is performed, the technology of Patent Document 1 is not necessarily effective.

[0011] An object of the present invention is to realize non-disruptive data processing setting changes in multi-stage data processing in a data processing system. [Means for solving the problem]

[0012] A data processing system according to one aspect of the present invention is a data processing system that executes multi-stage data processing by changing a part of a data processing setting or adding a new data processing setting in a state where a plurality of data processing settings are mixed in a data processing unit, and that has a data control design unit and a data control management unit, and the data control design unit assigns tags as identifiers to the data control settings, and the data control management unit: By starting the data processing unit necessary for the data processing and stopping the data processing unit not necessary for the data processing, part of the data processing settings can be changed or new data processing settings can be added without stopping the multi-stage data processing, and when executing the multi-stage data processing, the data processing unit in the subsequent stage can determine the data processing unit that executed the data processing by reading the tag assigned by the data processing unit in the previous stage. [Effects of the Invention]

[0013] According to one aspect of the present invention, in a data processing system, it is possible to change data processing settings without stopping multi-stage data processing. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 illustrates an example of the configuration of a data processing system according to a first and second embodiments. [Figure 2] FIG. 1 illustrates an example of a computer according to a first embodiment and a second embodiment. [Figure 3] 3A and 3B are diagrams illustrating an example of a user input unit and a user output unit according to the first and second embodiments. [Figure 4] FIG. 10 is a diagram illustrating an example of a user output unit according to the first and second embodiments. [Figure 5] FIG. 10 is a diagram illustrating an example of a user output unit according to the first and second embodiments. [Figure 6] FIG. 2 is a diagram illustrating an operation procedure of the first embodiment. [Figure 7] FIG. 10 is a diagram showing a data processing flow before the start of an operation procedure in the first and second embodiments. [Figure 8] FIG. 10 is a diagram illustrating an example of data control settings according to the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating a data processing flow after a partial data control setting change in the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a data processing flow when a data control setting change is completed in the first embodiment. [Figure 11]FIG. 10 is a diagram illustrating an operation procedure of the second embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of an input screen for data control settings to be added according to the second embodiment. [Figure 13] FIG. 10 is a diagram showing a data processing flow after adding a data control setting in the second embodiment. [Figure 14] FIG. 10 is a diagram illustrating a data processing flow when the addition of a data control setting according to the second embodiment is completed. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.

[0016] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant explanations will be omitted.

[0017] To facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings etc. [Example]

[0018] In Example 1, in a situation where data processing units that simultaneously execute multiple data processing flows are connected in multiple stages to perform data processing, when there is a change in the business for which data is collected or a change in the information to be acquired, a procedure for changing the data control settings without interruption is shown.

[0019] FIG. 1 is a diagram illustrating an example of the configuration of a data processing system according to a first embodiment. The data processing system includes a data control design unit 1, a data control management unit 2, a data source 3, a message queue 4, a data processing unit 5, and a data sink 6.

[0020] The data control design unit 1 includes a user input unit 11, a user output unit 12, and a data control setting management unit 13 as functional components.

[0021] The user input unit 11 includes a data control setting designation unit 111, a data control setting application instruction unit 112, and a data control change determination unit 113. The data control setting designation unit 111 accepts input related to data control settings and transmits the input contents to the data control setting management unit 13. The data control settings refer to the settings of the queue 41 that transfers data in the message queue 4 and the data processing settings 51 that are executed by the data processing unit 5.

[0022] The data control setting application instruction unit 112 receives input regarding the activation or deactivation of the data control settings managed by the data control setting management unit 13, and transmits the input content to the data control management unit 2. The data control change determination unit 113 receives input regarding whether to reflect the data control settings input by the data control setting designation unit 111 and activated by input from the data control setting application instruction unit 112, or to perform a switchback without reflecting the settings, and transmits the input content to the data control management unit 2.

[0023] The user output unit 12 includes a data control content display unit 121, a data processing flow operation status display unit 122, and a data processing unit operation status display unit 123. The data control content display unit 121 displays the setting contents of the data control settings managed by the data control setting management unit 13. The data processing flow operation status display unit 122 displays the operation status of the data control settings managed by the data control setting management unit 13. The data processing unit operation status display unit 123 displays the data processing unit 5 and the data processing settings 51 executed within the data processing unit 5, and indicates the operation status of the data processing unit 5.

[0024] The data control setting management unit 13 includes a data control setting storage unit 131 and a data processing setting tag generation unit 132. The data control setting storage unit 131 stores the contents input by the data control setting designation unit 111. The data processing setting tag generation unit 132 generates a unique tag for the data processing setting 51 of the data control setting stored in the data control setting storage unit 131.

[0025] The data control management unit 2 includes, as functional components, a queue setting generation unit 21, a data processing setting generation unit 22, and a data processing unit start / stop execution unit 23.

[0026] The queue setting generation unit 21 acquires the data control settings stored in the data control setting storage unit 131 based on input from the data control setting application instruction unit 112 and the data control change determination unit 113, and if the required queue 41 does not exist, creates a queue 41 in the message queue 4.

[0027] The data processing setting generation unit 22 acquires the data control settings stored in the data control setting memory unit 131 based on input from the data control setting application instruction unit 112 and the data control change determination unit 113, assigns a tag generated by the data processing setting tag generation unit 132, and creates a data processing setting 51 that runs in the data processing unit 5.

[0028] The data processing unit start / stop execution unit 23 executes start and stop of the data processing unit 5 including the data processing settings 51 in response to inputs from the data control setting application instruction unit 112 and the data control change determination unit 113 .

[0029] The data source 3 sends data to be processed by the system to the message queue 4. The data source 3 can be in any form, such as a sensor, IoT device, gateway, or database, as long as it can generate data.

[0030] The message queue 4 uses a queue 41 created for each data processing flow to transfer data between the data source 3 and the data processing unit 5, between the data processing units 5 and 5, and between the data processing unit and the data sink. When a failure occurs in the data processing unit 5 and data is lost, the message queue 4 performs operations such as restoring the lost data from the data stored in the queue 41 and rearranging the data processed by the data processing unit 5 in the order in which it was sent by the data source 3.

[0031] The data processing unit 5 executes data processing in accordance with the data processing settings 51 created for each data processing flow. The data processing settings 51 describe the queue 41 from which data is acquired, the data processing content, and the queue 41 or data sink 6 to which the data is sent. When adding or changing the data processing settings 51 to the data processing unit 5, it is necessary to start the data processing unit 5 anew.

[0032] The data sink 6 stores the data processed by the data processing unit 5. The data sink 6 may be in any form, such as an IoT device, an IoT application, or a database, as long as it can receive data.

[0033] FIG. 2 is a diagram showing an example of a computer. The data control design unit 1, data control management unit 2, message queue 4, and data processing unit 5 are configured as a computer 7 as shown in FIG. 2. The computer 7 has a processor 71, a main memory device 72, a secondary memory device 73, a network interface 74, and an input / output device 75. The processor 71 executes a program stored in the main memory device 72. The processor 71 executes processing in accordance with the program, thereby operating as a functional unit (module) that realizes a specific function.

[0034] In the following description, when a process is described using a functional unit as the subject, it means that the processor 71 is executing a program that realizes the functional unit. The main storage device 72 is a memory or the like, and stores the program executed by the processor 71 and the data used by the program. The secondary storage device 73 is a hard disk drive (HDD) or solid state drive (SSD), and stores data persistently. The program and data stored in the main storage device 72 may be stored in the secondary storage device 73. In this case, the processor 71 reads the program and data from the secondary storage device 73 and loads them into the main storage device 72. The network interface 74 communicates with other devices via a network. The input / output device 75 accepts user input and displays calculation results to the user. The input / output device 75 can be omitted. The computer 7 may be virtualized and operate in a form such as a VM or container.

[0035] FIG. 3 is a diagram showing an example of a user input unit and a user output unit. 3 shows an example of the user input unit and user output unit when flow1, flow2, and flow3 exist and flow3 is changed to flow3'. The data processing flow operation status display unit 122 displays the start, stop, etc. status of the relevant data processing flow. The data control setting application instruction unit 112 accepts input regarding the calling, start, stop, etc. of the editing screen for the relevant data processing flow. When editing input is made in the data control setting application instruction unit 112, the editing screen for the relevant data processing flow is displayed in the data control setting specification unit 111.

[0036] The data control setting specification unit 111 specifies the input source of the data handled by the data processing flow, the data control settings, and the output destination. In this embodiment, an example of executing two stages of data processing has been described, but three or more stages may be used. Also, in this embodiment, the number of specified output destinations is two, but the specified number can be any natural number. After creating the data control settings in the data control setting specification unit 111, inputting "Save" saves the data control settings in the data control setting storage unit 131. Inputting "Close" closes the data control setting specification unit 111.

[0037] The data control setting specification unit 111 may be displayed on the same screen as the data control setting application instruction unit 112, as shown in Figure 3, or on a separate screen. The data control change determination unit 113 is displayed for the data processing flow for which the data control settings are to be changed. In the example of Figure 3, this applies to flow3 and flow3'. The data control change determination unit 113 accepts an input of "revert" for the data processing flow before the change, and an input of "confirm" for the data processing flow after the change. The data control content display unit 121 displays the data processing settings for the data processing flow. This may be the same screen as the data control setting specification unit 111, or a separate screen. When a separate screen is used, the data control setting application instruction unit 112 accepts an input to display the data processing flow.

[0038] FIG. 4 is a diagram illustrating an example of the user output unit. The data processing unit operating status display unit 123 displays the relationship between the data processing unit 5 and the data processing settings 51 operating in the data processing unit 5, and the tags of each data processing setting 51, based on the information stored in the data control setting storage unit 131. It may also display the relationship between the data source 3, the message queue 4, and the data sink 6. It may also display the operating status of the relevant data processing unit 5. In the example of FIG. 4, data processing units 1 and 2 are operating, and data processing units 1' and 2' are running, which are displayed by separating them with solid and dotted lines. Other methods of expression may also be used to distinguish between the operating statuses.

[0039] FIG. 5 is a diagram illustrating an example of the user output unit. The data processing unit operating status display unit 123 can display in the table format shown in Fig. 5 instead of Fig. 4. Based on the information stored in the data control setting storage unit 131, the data processing unit operating status display unit 123 displays the relationship between the data processing unit 5 and the data processing settings 51 operating in the data processing unit 5, and the tags of each data processing setting 51. It may also display the relationship between the data source 3, message queue 4, and data sink 6. It may also display the operating status of the relevant data processing unit 5.

[0040] FIG. 6 is a diagram showing an operation procedure of the first embodiment. In step S101, the data processing control settings are changed, and in step S102, the changed data processing unit is started.

[0041] In step S103, it is confirmed whether the data control settings changed in step S101 were as intended. The optimal data control settings differ depending on the site where the data collection platform is used. When changing the data control settings due to changes in IoT utilization operations, the user needs a period of trial and error before creating the optimal data control settings.

[0042] If the data stored in the data sink 6 in step S102 is not as intended by the user, the user inputs "switch back" from the data control change determining unit 113 and the process proceeds to step S104.

[0043] If the data stored in the data sink 6 in step S102 is as intended by the user, the user inputs "confirm" from the data control change determination unit 113 and the process proceeds to step S105.

[0044] In step S104, the state is returned to the state before the data control setting change. Based on the information on the data processing unit 5 and the data processing setting 51 stored in the data control setting storage unit 131, the data processing unit start / stop execution unit 23 stops the data processing unit 5 started in step S102, and sets the state so that only the data processing unit 5 before the data control setting change is operating.

[0045] FIG. 7 is a diagram showing a data processing flow before the start of the operation procedure. Three data processing flows are shown in Figure 7. Data sent by data source 3-1 passes through queue 41-1, data processing setting 51-1, queue 41-4, and data processing setting 51-4 in that order, via message queue 4 and data processing unit 5, and is then stored in data sink 6-1.

[0046] The data processing settings 51-1 describe data processing such as adding metadata such as key names, decoding encoded data, and numeric-character conversion for data received from the data source 3-1, and the data processing unit 5-1 processes the data based on the described contents. The data processing contents are not limited to these. The data processing settings 51-4 describe data processing such as inserting data into a relational database to store the data in the data sink 6-1, executing an application programming interface, and writing to a file, and the data processing unit 5-2 processes the data based on the described contents. The data processing contents are not limited to these.

[0047] If an unexpected failure occurs during data processing in the data processing unit 5-1, there is a possibility that data sent from the data source 3-1 may be lost. Therefore, by inserting a message queue 41-1 between the data source 3-1 and the data processing unit 5-1 and temporarily storing data in the message queue 41-1, even if data is lost in the data processing unit 5-1, it is possible to recover data from the message queue 41-1.

[0048] Furthermore, if an unintended failure occurs during data processing in the data processing unit 5-2, there is a possibility that the data sent from the data processing unit 5-1 may be lost. Therefore, by inserting a message queue 41-4 between the data processing units 5-1 and 5-2 and temporarily storing data in the message queue 41-4, even if data is lost in the data processing unit 5-1, it is possible to recover the data from the message queue 41-4.

[0049] Because the message queue 4 is an important function that ensures data availability in the data processing flow, when changing the data control settings by input to the data control setting specification unit 111, the data control management unit 2 makes no changes to the queue 41 and only makes changes to the data processing setting 51. In FIG. 7, the tag "1101" is set for the data processing setting 51-1, and the tag "1102" is set for the data processing setting 51-4. Since the tag "1101" is assigned to data processed using the data processing setting 51-1, the data processing unit 5 and data sink 6 at the subsequent stage can confirm that the data is "data processed on the data processing unit 5-1 using the data processing setting 51-1" by checking the tag value "1101." The tag value can have any format or string length as long as it is unique within the system.

[0050] The data source 3-2 and the data source 3-3 are different data sources 3 from the data source 3-1. The queues 41 through which the message queue 4 passes, the data processing settings 51 executed by the data processing unit 5, and the data sinks 6 are also different. The data processing unit 5-1 processes the data received from the data source 3-1, the data source 3-2, and the data source 3-3 using different data processing settings 51-1, 51-2, and 51-3, but the data processing settings 51-1, 51-2, and 51-3 are executed by the same data processing unit 5-1. It is possible to execute the data processing settings 51-1, 51-2, and 51-3 in separate data processing units 5, but this would consume more computer resources than executing them in a single data processing unit 5.

[0051] When the number of data processing flows increases, in order to prevent an increase in the amount of computer resources consumed, the data collection infrastructure is required to execute multiple data processing operations of different data processing flows in parallel within one data processing unit 5. In this embodiment, multiple data processing settings 51 are operated in one data processing unit 5.

[0052] When it becomes necessary to change the data input from the data source 3 or the data processing performed by the data processing unit 5 due to a change in the IoT-utilizing business, the process proceeds to step S101.

[0053] In step S101, the user inputs new data control settings from the data control setting designation unit 111.

[0054] Fig. 8 is a diagram showing an example of a data control setting input screen. Fig. 8 shows an example of changing the data control flow related to data source 3-3 in Fig. 7. The data processing settings 51-3 and 51-6 operating in Fig. 7 are changed, and two data sinks 6 other than data sink 6-3 are specified as output destinations. The newly created data control settings are stored in the data control setting storage unit 131. In this embodiment 1, it is assumed that the data processing flows related to data source 3-1 and data source 3-2 do not require any changes to the data control settings.

[0055] In step S102, the user performs an input to activate the data control setting created in step S101 from the data control setting application instruction unit 112. The data processing setting generation unit 22 creates new data processing settings 51 based on the data control setting created in step S101. The data processing setting generation unit 22 includes the new tag created by the data processing setting tag generation unit 132 in the new data processing setting 51 for the data source 3-3.

[0056] Furthermore, the data processing setting generation unit 22 also includes a tag newly created by the data processing setting tag generation unit 132 in the data processing settings 51 for the data source 3-1 and the data source 3-2, even though the data processing settings remain unchanged. This is because the data processing content does not change, but the data processing unit 5 that executes the data processing is different, and it is possible to determine at a later stage which data processing unit 5 executed the data processing. The data processing unit start / stop execution unit 23 starts a new data processing unit 5 based on the relationship between the data processing settings 51 and the data processing unit 5 stored in the data control setting storage unit 131.

[0057] In the first embodiment, the data processing unit start / stop execution unit 23 starts a data processing unit 5-1' containing data processing settings 51-1' and 51-2' for the data source 3-1 and data source 3-2 containing new tags, and a new data processing setting 51-3' for the data source 3-3 created by the data processing setting generation unit 22, and starts a data processing unit 5-2' containing data processing settings 51-4' and 51-5' for the data source 3-1 and data source 3-2 containing new tags, and a new data processing setting 51-6' for the data source 3-3 created by the data processing setting generation unit 22.

[0058] Figure 9 shows the data processing flow after some of the data control settings have been changed. In addition to Figure 7, data processing units 5-1' and 5-2' are now operational. Data sent by data source 3-1 passes through queue 41-1, and data processing is performed by both data processing settings 51-1 and 51-1'. The results of data processing by data processing settings 51-1 and 51-1' are identical except for the tag values.

[0059] The results of data processing by data processing setting 51-1 and data processing setting 51-1' are passed through queue 41-4, data processing is executed by both data processing setting 51-4 and data processing setting 51-4', and stored in data sink 6-1. The results of data processing by data processing setting 51-4 and data processing setting 51-4' are identical except for the tag value. The data sent by data source 3-1 is stored in duplicate in data sink 6-1.

[0060] Since the data collection platform needs to process data without missing any data, data loss is prevented by processing data in duplicate when the data control settings are changed. The data sink 6-1 can determine which data processing unit 5 processed the data by reading the tag value. The data sink 6-1 may save all of the duplicated data stored, or may discard one of the data based on the tag information. Data sent by the data source 3-2 is processed in the same way as data sent by the data source 3-1.

[0061] Data sent from data source 3-3 passes through queue 41-3, and data processing is performed by both data processing setting 51-3 and data processing setting 51-3'. The results of data processing by data processing setting 51-3 are different from the results of data processing by data processing setting 51-3'. The results of data processing by data processing setting 51-3 are placed in queue 41-6. The results of data processing by data processing setting 51-3' are also placed in queue 41-6. Data processing setting 51-6 acquires only the data in queue 41-6 that has been processed by data processing setting 51-3 and has been tagged "3101", processes the data, and stores it in data sink 6-3. Therefore, data sink 6-3 stores only data that was processed before the data control setting was changed.

[0062] The data processing setting 51-6' acquires only the data in the queue 41-6 that has been processed by the data processing setting 51-3' and has been tagged with "3201", processes the data, and stores it in the data sinks 6-4 and 6-5. Therefore, the data sinks 6-4 and 6-5 store only the data that has been processed after the data control setting change.

[0063] In the first embodiment, when switching back to the state before the data control setting change is performed in step S104, the state in Fig. 9 returns to the state in Fig. 7. All data processing flows related to data source 3-1, data source 3-2, and data source 3-3 can continue data processing with the settings before the data control setting change without stopping data processing in the operation of step S104. The user creates new data control settings again in step S101.

[0064] In step S105, the data processing unit 5 that was operating before the data control setting is stopped. Based on the information on the data processing unit 5 and the data processing setting 51 stored in the data control setting storage unit 131, the data processing unit start / stop execution unit 23 stops the data processing unit 5 that has been operating since before step S101, and sets up a state in which only the data processing unit 5 that was started in step S102 is operating.

[0065] FIG. 10 is a diagram showing the data processing flow when the data control setting change is completed. Data processing unit 5-1' and data processing unit 5-2' operate as the data processing unit 5. Data processing unit 5-1' executes data processing of data processing settings 51-1' and data processing settings 51-2' for data source 3-1 and data source 3-2, for which no change in data control settings is required, and changed data processing settings 51-3 for data source 3-3, for which a change in data control settings is required. Data processing unit 5-2' executes data processing of data processing settings 51-4' and data processing settings 51-5' for data source 3-1 and data source 3-2, for which no change in data control settings is required, and changed data processing settings 51-6' for data source 3-3, for which a change in data control settings is required. As described above, in a situation where data processing units that simultaneously execute multiple data processing flows are connected in multiple stages to perform data processing, it is possible to change the data control settings without interruption when changes occur in the business for which data is collected or the information to be obtained. [Example]

[0066] In the second embodiment, a procedure is shown in which a data processing unit that simultaneously executes multiple data processing flows is connected in multiple stages to perform data processing, and a data control setting is added without interruption when a business operation for which data is to be collected is added.

[0067] An example of the system configuration of the data processing system is the same as that shown in FIG. 1 in the first embodiment. An example of the computer is the same as that shown in FIG. 2 in the first embodiment. An example of the user input unit and the user output unit is the same as that shown in FIG. 3 in the first embodiment. An example of the user output unit is the same as that shown in FIG. 4 and FIG. 5 in the first embodiment. FIG. 11 is a diagram showing the operation procedure of the second embodiment. Steps S102 to S105 are the same as those in the first embodiment.

[0068] The data processing flow before the start of the operation procedure is the same as that shown in Fig. 7 in the first embodiment. When additional data processing is required due to changes in the IoT-utilizing business, the process proceeds to step 106.

[0069] In step S 106 , the user inputs the data control setting to be added from data control setting designation unit 111 .

[0070] FIG. 12 is a diagram showing an example of an input screen for adding data control settings. This setting acquires data from a new data source and stores the data in an existing data sink and a new data sink. The newly created data control setting is stored in the data control setting storage unit 131. In this second embodiment, it is assumed that the data processing flows related to data source 3-1, data source 3-2, and data source 3-3 do not require any change in the data control setting.

[0071] In step S102, the user performs an input to activate the data control setting created in step S106 from the data control setting application instruction unit 112. The data processing setting generation unit 22 creates new data processing settings 51 based on the data control setting created in step S106. The data processing setting generation unit 22 includes the tag created by the data processing setting tag generation unit 132 in the data processing setting 51 for data source 3-4. Furthermore, the data processing setting generation unit 22 also includes the tag newly created by the data processing setting tag generation unit 132 in the data processing settings 51 for data source 3-1, data source 3-2, and data source 3-3, although there is no change in the data processing settings.

[0072] This is because the data processing content does not change, but the data processing unit 5 that executes the data processing is different, and which data processing unit 5 executed the data processing is determined at a later stage. The data processing unit start / stop execution unit 23 starts up a new data processing unit 5 based on the relationship between the data processing setting 51 stored in the data control setting storage unit 131 and the data processing unit 5.

[0073] In the second embodiment, the data processing unit start / stop execution unit 23 starts the data processing unit 5-1" which includes the data processing settings 51-1", 51-2", 51-3" for the data source 3-1, 3-2 and 3-3 containing the new tag, and the new data processing setting 51-a for the data source 3-4 created by the data processing setting generation unit 22, and starts the data processing unit 5-2" which includes the data processing settings 51-4", 51-5", 51-6" for the data source 3-1, 3-2 and 3-3 containing the new tag, and the new data processing setting 51-b for the data source 3-4 created by the data processing setting generation unit 22.

[0074] Figure 13 is a diagram showing the data processing flow after the addition of the data control setting. In addition to Figure 7, data processing unit 5-1" and data processing unit 5-2" are in operation. Data sent by data source 3-1 passes through queue 41-1, and data processing is executed by both data processing setting 51-1 and data processing setting 51-1". The data processing result by data processing setting 51-1 and the data processing result by data processing setting 51-1" are identical except for the tag value. The data processing result by data processing setting 51-1 and data processing setting 51-1" passes through queue 41-4, and data processing is executed by both data processing setting 51-4 and data processing setting 51-4" and stored in data sink 6-1. The data processing result by data processing setting 51-4 and the data processing result by data processing setting 51-4" are identical except for the tag value.

[0075] The data sent by the data source 3-1 is stored in duplicate in the data sink 6-1. Because the data collection platform requires data processing without data loss, data loss is prevented by processing the data duplicately when adding a data control setting. The data sink 6-1 can determine which data processing unit 5 processed the data by reading the tag value. The data sink 6-1 may save all of the duplicated data stored, or it may discard one of the data based on the tag information. Data sent by the data source 3-2 and data sent by the data source 3-3 is processed in the same way as data sent by the data source 3-1. Data sent from the data source 3-4 passes through the queue 41-a and is processed by the data processing setting 51-a. The data processing results by the data processing setting 51-a are stored in the queue 41-b. The data processing setting 51-b acquires the data from the queue 41-b, processes it, and stores it in the data sink 6-4 and the data sink 6-6.

[0076] In step S103, it is confirmed whether the data control settings added in step S106 are as intended. If the data stored in data sink 6 in step S102 is not as intended by the user, the user inputs "switch back" from data control change determination unit 113 and proceeds to step S104. If the data stored in data sink 6 in step S102 is as intended by the user, the user inputs "confirm" from data control change determination unit 113 and proceeds to step S105.

[0077] In step S104, the state is returned to the state before the data control setting was added. Based on the information on the data processing unit 5 and the data processing setting 51 stored in the data control setting storage unit 131, the data processing unit start / stop execution unit 23 stops the data processing unit 5 started in step S102, and a state is created in which only the data processing unit 5 before the data control setting was changed is in operation. In this embodiment 2, the state returns from that of FIG. 13 to that of FIG. 7. All data processing flows related to data source 3-1, data source 3-2, and data source 3-3 can continue data processing without stopping data processing with the operation of step S104. The user again creates a data control setting to be added in step S106.

[0078] In step S105, the data processing unit 5 that was operating before the data control setting is stopped. Based on the information on the data processing unit 5 and the data processing setting 51 stored in the data control setting storage unit 131, the data processing unit start / stop execution unit 23 stops the data processing unit 5 that has been operating since before step S106, and sets up a state in which only the data processing unit 5 that was started in step S102 is operating.

[0079] FIG. 14 is a diagram showing the data processing flow when the addition of data control settings is completed. Data processing unit 5-1" and data processing unit 5-2" operate as the data processing unit 5. Data processing unit 5-1" executes data processing of data processing settings 51-1", data processing settings 51-2", and data processing settings 51-3" for data source 3-1, data source 3-2, and data source 3-3, for which no change in data control settings is required, as well as data processing setting 51-a for data source 3-4, which is an added data control setting. Data processing unit 5-2" executes data processing of data processing settings 51-4", data processing setting 51-5", and data processing setting 51-6" for data source 3-1, data source 3-2, and data source 3-3, for which no change in data control settings is required, as well as data processing setting 51-b for data source 3-4, which is an added data control setting.

[0080] According to the second embodiment, in a situation where data processing units that simultaneously execute multiple data processing flows are connected in multiple stages to perform data processing, when a new business operation for which data is to be collected is added, data control settings can be added without interruption.

[0081] The computer system of the above embodiment has a data control design unit that generates identifiers to be assigned to the data processing settings of each system before and after the data processing setting change when changing part of the data processing settings in multi-stage data processing in an environment where multiple data processing settings are mixed in a data processing unit, and a data control management unit that creates a data processing unit in units containing data processing settings that are subject to the data processing setting change and data processing settings that are not subject to the data processing setting change, and starts up all stages of each system before and after the data processing setting change to execute data processing, and the system that executed the data processing is identified using the identifier in the data processing unit of the subsequent stage.

[0082] According to the above embodiment, in a use case in which multiple multi-stage data processing flows are executed in parallel, when changing a certain data processing setting, it is possible to change the data processing flow to be changed without stopping the data processing flow that is not to be changed. [Explanation of symbols]

[0083] 1 Data Control Design Department 2 Data Control Management Department 3 Data Sources 4. Message Queues 5 Data processing section 6 Data Sync 7 Calculator 11 User input section 12 User output section 13 Data Control Settings Management Unit 21 Queue setting generation unit 22 Data processing setting generation unit 23 Data processing unit start / stop execution unit 41 Queue 51 Data Processing Settings 71 processors 72 Main storage 73 Secondary main storage 74 Network Interface 75 Input / Output Devices 111 Data control setting specification section 112 Data control setting application instruction section 113 Data Control Change Decision Unit 121 Data control content display section 122 Data processing flow operation status display section 123 Data processing unit operating status display unit 131 Data control setting memory unit 132 Data processing setting tag generation unit

Claims

1. A data processing system in which data processing units that simultaneously execute multiple data processing flows are connected in multiple stages to perform data processing, and in which multiple data processing settings are mixed in the data processing units, some of the data processing settings are changed or new data processing settings are added to perform multi-stage data processing, It has a data control design department and a data control management department. The data control design unit assigning a tag as an identifier to each of the data processing settings; The data control management unit by starting up the data processing unit necessary for the data processing and stopping the data processing unit unnecessary for the data processing, changing a part of the data processing setting or adding a new data processing setting without stopping the multi-stage data processing, When performing the multi-stage data processing, The data processing unit in the latter stage A data processing system characterized in that the data processing unit that executed the data processing is identified by reading the tag attached in the data processing unit at the previous stage.

2. The data processing unit The data processing settings include those that are subject to data processing setting change and those that are not subject to data processing setting change, The data processing unit in the latter stage 2. The data processing system according to claim 1, wherein whether the data processing setting is a change target or a non-change target is determined by reading the tag.

3. The data control design unit a data control setting storage unit that stores the data processing settings for a plurality of the data processing flows and the data processing units in which the data processing settings operate; The data control management unit a data processing unit start / stop execution unit that instructs all of the data processing units that execute the data processing settings of the data processing flow to start and stop the data processing units when the data processing settings are changed or added, The data processing unit start / stop execution unit 2. The data processing system according to claim 1, wherein the system starts up a data processing unit required for the data processing and stops a data processing unit not required for the data processing.

4. The data processing unit start / stop execution unit Determine whether the data control content after changing the data processing settings is as intended, If it is determined that the data control content after the data processing setting change is as intended, 4. The data processing system according to claim 3, wherein the data processing unit is stopped before the data processing setting is changed, and the data processing unit is started after the data processing setting is changed.

5. The data processing unit start / stop execution unit If it is determined that the data control content after the data processing setting change is not as intended, The data processing system according to claim 4, characterized in that the data processing unit before the data processing setting change is started and the data processing unit after the data processing setting change is stopped, thereby switching back to the data processing setting before the change without interruption.

6. The data processing unit start / stop execution unit Determine whether the data control content after adding data processing settings is as intended, If it is determined that the data control content after the addition of the data processing setting is as intended, 4. The data processing system according to claim 3, wherein the data processing unit is stopped before the data processing setting is added, and the data processing unit is started after the data processing setting is added.

7. The data processing unit start / stop execution unit If it is determined that the data control content after the data processing setting is added is not as intended, The data processing system according to claim 6, characterized in that the data processing unit before the data processing setting is added is started and the data processing unit after the data processing setting is added is stopped, thereby switching back to the data processing setting before the addition without interruption.

8. The data control design unit 2. The data processing system according to claim 1, further comprising a display unit that displays the operating status of said data processing unit, said data processing settings, and said tags.

9. The data control design unit The data processing unit is configured as a multi-stage data processing unit including at least a front stage and a rear stage, the first stage includes a first data processing unit before the data processing setting is changed and a second data processing unit after the data processing setting is changed, the latter stage includes a third data processing unit before the data processing setting is changed and a fourth data processing unit after the data processing setting is changed, the first data processing unit has a first data processing setting; the second data processing unit has a second data processing setting that is a change of the first data processing setting, the third data processing unit has a third data processing setting; the fourth data processing unit has a fourth data processing setting obtained by changing the third data processing setting, The data control design unit generating a first tag and attaching the first tag to the first data processing configuration; generating a second tag and attaching the second tag to the second data processing configuration; generating a third tag and attaching the third tag to the third data processing configuration; generating a fourth tag and attaching the fourth tag to the fourth data processing configuration; The data control management unit activating all of the first data processing unit, the second data processing unit, the third data processing unit, and the fourth data processing unit to execute the data processing; The data processing system according to claim 1, characterized in that the data sent from the first data processing unit and the second data processing unit in the preceding stage to the third data processing unit and the fourth data processing unit in the following stage, respectively, is identified using the first tag, the second tag, the third tag, and the fourth tag.

10. The data control design unit The data processing unit is configured as a multi-stage data processing unit including at least a front stage and a rear stage, the front stage includes a first data processing unit before the data processing setting is added and a second data processing unit after the data processing setting is added, the latter stage includes a third data processing unit before the data processing setting is added and a fourth data processing unit after the data processing setting is added, the first data processing unit has a first data processing setting; the second data processing unit has a second data processing setting added to the first data processing setting, the third data processing unit has a third data processing setting; the fourth data processing unit has a fourth data processing setting added to the third data processing setting, The data control design unit generating a first tag and attaching the first tag to the first data processing configuration; generating a second tag and attaching the second tag to the second data processing configuration; generating a third tag and attaching the third tag to the third data processing configuration; generating a fourth tag and attaching the fourth tag to the fourth data processing configuration; The data control management unit activating all of the first data processing unit, the second data processing unit, the third data processing unit, and the fourth data processing unit to execute the data processing; The data processing system according to claim 1, characterized in that the data sent from the first data processing unit and the second data processing unit in the preceding stage to the third data processing unit and the fourth data processing unit in the following stage, respectively, is identified using the first tag, the second tag, the third tag, and the fourth tag.

11. A data processing method in which data processing is performed by a computer having a processor, with data processing units that simultaneously execute multiple data processing flows being connected in multiple stages, and with multiple data processing settings coexisting in the data processing units, a part of the data processing settings is changed or a new data processing setting is added to perform multi-stage data processing, assigning, by the processor, a tag that is an identifier to each of the data processing settings; a step of changing a part of the data processing setting or adding a new data processing setting by starting up the data processing unit required for the data processing and stopping the data processing unit not required for the data processing, without stopping the multi-stage data processing; a step of determining, by the processor, the data processing unit that executed the multi-stage data processing by reading the tag added in the data processing unit of the preceding stage; A data processing method comprising:

12. A step of determining by the processor whether the data control content after the data processing setting change is as intended; If the processor determines that the data control content after the data processing setting change is as intended, stopping the data processing unit before the data processing setting change and starting the data processing unit after the data processing setting change; If the processor determines that the data control content after the change of the data processing setting is not as intended, a step of non-disruptively switching back to the data processing setting before the change by starting up the data processing unit before the change of the data processing setting and stopping the data processing unit after the change of the data processing setting; 12. The data processing method of claim 11, further comprising:

13. A step of determining by the processor whether the data control content after the data processing setting is added is as intended; If the processor determines that the data control content after the addition of the data processing setting is as intended, stopping the data processing unit before adding the data processing setting and starting the data processing unit after adding the data processing setting; If the processor determines that the data control content after the data processing setting is added is not as intended, a step of starting up the data processing unit before the addition of the data processing setting and stopping the data processing unit after the addition of the data processing setting, thereby switching back to the data processing setting before the addition without interruption; 12. The data processing method of claim 11, further comprising:

Images