Information Processing System

Abstract

To realize parallelization of processing of data from a plurality of start points to one or more end points and merging of the data after the parallel processing, in a distributed environment.SOLUTION: An information processing system includes one or more information processing devices for processing information. The information processing device includes: a division function that divides processing information indicating a processing procedure of data from a plurality of start points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition; a determination function that uniquely determines one of the information processing devices to be responsible for each of the pieces of processing information divided by the division function; and an execution function that executes the processing in the information processing device determined by the determination function.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a distributed environment. Information Processing System Regarding. [Background technology]

[0002] Data-driven decision-making is emphasized, and analysis combining multiple data is required. In addition, the spread of IoT (Internet of Things) has led to an increase in the volume of data. A method called MapReduce is generally known as a method for processing large amounts of data at high speed. In MapReduce, as shown in Non-Patent Document 1, data is divided and distributed to multiple information processing devices called workers, and each worker executes processing on the assigned data and aggregates the execution results. In MapReduce, the same processing is executed in parallel and distributed manner to speed up processing. In addition, Patent Document 1 discloses a method in which an information processing device that accepts a request divides a series of processing in a flow into units that can be processed in parallel based on dependencies between the processing, specifies an information processing device that executes each processing, and copies the processing to each information processing device and executes it in parallel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-16494 A [Non-patent literature]

[0004] [Non-Patent Document 1] J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Cluster", OSDI, 2004. Summary of the Invention [Problem to be solved by the invention]

[0005] Among processing programs (hereinafter referred to as data flows) that execute a series of processes from a starting point (i.e., input) that is an input processing part to an end point (i.e., output) that is an output processing part, for example, in the case of a data flow with multiple starting points, if the multiple starting points are on one information processing device, that point can become a bottleneck. In Non-Patent Document 1 and Patent Document 1, the processing of input data is divided and executed by multiple information processing devices, but parallelization of different processes for different data and merging of each process are not considered. In order to merge each process, information such as the information processing device to be the merging destination, the merging point, and the data format in the merging process is required, and this information must be shared between the information processing devices that perform the merging before the execution of the data flow. However, in the conventional technology, the means required for sharing this information are not presented, and the merging of data after parallel processing cannot be realized.

[0006] The present invention has been made in consideration of the above background, and its purpose is to realize parallel processing of data from multiple starting points to one or more end points in a distributed environment, and to realize the merging of data after parallel processing. [Means for solving the problem]

[0007] In order to solve such problems, the present invention includes one or more information processing devices for processing information, the information processing devices having a division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing devices as a division condition, a determination function for uniquely determining which of the information processing devices is to be in charge of each of the processing information divided by the division function, and an execution function for executing the processing in the information processing device determined by the determination function. the processing information includes a plurality of processing units that are arranged between the start points and the end points and sequentially process data from at least one of the start points, and one or more transfer units that are arranged between any of the plurality of processing units and transfer data between the processing units, and when dividing the processing information, the division function divides a range in which the processing information is recorded into a plurality of areas based on the transfer units, divides the transfer units into a plurality of divided transfer units, divides the divided divided transfer units into the respective areas, and allocates the divided divided transfer units to one of the plurality of divided transfer units. a division and transfer unit that is a transfer source belonging to the same area as the one of the division and transfer units, and connects the other of the plurality of division and transfer units to the processing unit that is a transfer destination belonging to the same area as the other division and transfer unit, and one of the decision functions belonging to the information processing device sequentially selects one information processing device from among the information processing devices, sequentially assigns the selected one information processing device to be in charge of each of the areas, records each of the areas and the one information processing device in correspondence with each other in area assignment information, and distributes the recorded area assignment information to each of the information processing devices. It is characterized by: The information processing device further includes one or more information processing devices for processing information, the information processing device having a division function for dividing processing information indicating a processing procedure for data from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition, a determination function for uniquely determining the responsibility of each piece of the processing information divided by the division function to one of the information processing devices, and an execution function for executing processing in the information processing device determined by the determination function, the processing information including a plurality of processing units arranged between each of the starting points and the end point and sequentially processing data from at least one of the respective starting points, and one or more transfer units arranged between any of the processing units among the plurality of processing units and transferring data between the processing units, the division function, when dividing the processing information, dividing a range in which the processing information is recorded into a plurality of areas based on the transfer units, and dividing the transfer units into a plurality of divided transfer units, dividing and allocating each of the divided divided transfer units to each of the areas, connecting one of the plurality of divided transfer units to the processing unit that is a transfer source belonging to the same area as the one of the divided transfer units, the division and transfer unit connected to the information processing device of the other division and transfer unit is connected to the processing unit of the transfer destination which belongs to the same area as the other division and transfer unit, and transmits and receives information to and from the information processing device, the management device has area assignment information for management device which records a division identifier which uniquely identifies each of the areas and a device identifier which uniquely identifies each of the information processing devices which are in charge of each of the areas in association with each other, the determination function belonging to each of the information processing devices transmits to the management device a registration request for assigning a designated area among the areas to the information processing device to which the determination function belongs, when the management device receives the registration request, refers to the area assignment information for management device based on the received registration request, and, on condition that the assignment of the designated area is not registered from the result of the reference, registers the division identifier of the designated area in association with the device identifier of the request source in the area assignment information for management device, and transmits a response to the determination function indicating that the division identifier has been registered as the assignment of the designated area, and if the assignment of the designated area has already been registered, transmits a response to the determination function indicating that the division identifier cannot be registered as the assignment of the designated area, and the determination function which has received the responseThe designated area is registered as the area in charge of the information processing device to which the determination function belongs, on condition that the response is a response indicating that the designated area has been registered as the area in charge, and the area in charge of the designated area is uniquely determined. The information processing device further includes one or more information processing devices for processing information, the information processing device having a division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition, a determination function for uniquely determining the responsibility of each piece of the processing information divided by the division function to one of the information processing devices, and an execution function for executing the processing in the information processing device determined by the determination function, the processing information including a plurality of processing units arranged between each of the starting points and the end point and sequentially processing data from at least one of the respective starting points, and one or more transfer units arranged between any of the processing units among the plurality of processing units and transferring data between the processing units, the division function dividing a range in which the processing information is recorded into a plurality of areas based on the transfer units when dividing the processing information, The transfer unit is divided into a number of divided transfer units, and each of the divided divided transfer units is divided and allocated to each of the areas, and one of the multiple divided transfer units is connected to the processing unit at the source which belongs to the same area as the one divided transfer unit, and the other of the multiple divided transfer units is connected to the processing unit at the destination which belongs to the same area as the other divided transfer unit, the information processing devices are respectively located at a number of different bases, and when the transfer unit in the processing information is assigned an execution base identifier that uniquely identifies the execution base of the information processing device, the division function converts the execution base identifier into a division identifier that uniquely identifies each of the areas and assigns the converted division identifier to each of the areas, and the determination function uniquely determines the information processing device that will be responsible for each of the areas based on the division identifier assigned to each of the areas. The information processing device further includes one or more information processing devices for processing information, the information processing device having a division function for dividing processing information indicating a processing procedure for data from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition, a determination function for uniquely determining the handling of each piece of the processing information divided by the division function to one of the information processing devices, and an execution function for executing processing in the information processing device determined by the determination function, the processing information including a plurality of processing units arranged between each of the starting points and the end point and sequentially processing data from at least one of the respective starting points, and one or more transfer units arranged between any of the processing units among the plurality of processing units and transferring data between the processing units, the division function, when dividing the processing information, dividing a range in which the processing information is recorded into a plurality of areas based on the transfer units, and dividing the transfer units into a plurality of divided transfer units, dividing and allocating each of the divided divided transfer units to each of the areas, connecting one of the plurality of divided transfer units to the processing unit that is a transfer source belonging to the same area as the one of the divided transfer units, a division transfer unit that transfers the data to the processing unit of the information processing device that is a transfer destination and that belongs to the same area as the other division transfer unit; and the information processing device further has a communication function that controls communication between the information processing devices determined by the determination function, the division function assigns a division identifier that uniquely identifies each of the areas to each of the areas, the communication function stores a first division identifier that uniquely identifies the area to which the one division transfer unit belongs in association with the division transfer unit as first communication information indicating a transfer source area, and stores the information that is responsible for processing the area to which the one division transfer unit belongs in association with the division transfer unit as first communication information indicating a transfer source area. a first device identifier that uniquely identifies a processing device is stored in association with the division and transfer unit as second communication information indicating the information processing device of the transfer source, a second division identifier that uniquely identifies the area to which the other division and transfer unit belongs is stored in association with the division and transfer unit as third communication information indicating the area to which the other division and transfer unit belongs, a second device identifier that uniquely identifies the information processing device responsible for processing the area to which the other division and transfer unit belongs is stored in association with the division and transfer unit as fourth communication information indicating the information processing device of the transfer destination, and a type of communication between the information processing devices isThe information processing device is characterized in that the information processing device is connected to the division and transfer unit and stored as fifth communication information that specifies transmission or reception, and the execution function refers to the first communication information to the fifth communication information as a process before executing the process of the division and transfer unit, and establishes a communication connection between the information processing devices. The information processing device further includes one or more information processing devices for processing information, the information processing device having a division function for dividing processing information indicating a processing procedure for data from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition, a determination function for uniquely determining the responsibility of each piece of the processing information divided by the division function to one of the information processing devices, and an execution function for executing processing in the information processing device determined by the determination function, the processing information including a plurality of processing units arranged between each of the starting points and the end point and sequentially processing data from at least one of the respective starting points, and one or more transfer units arranged between any of the processing units among the plurality of processing units and transferring data between the processing units, the division function, when dividing the processing information, dividing a range in which the processing information is recorded into a plurality of areas based on the transfer units, and dividing the transfer units into a plurality of divided transfer units, and allocating each of the divided divided transfer units to each of the areas, One of the multiple split transfer units is connected to the processing unit of the source that belongs to the same area as the one split transfer unit, and the other of the multiple split transfer units is connected to the processing unit of the destination that belongs to the same area as the other split transfer unit, each of the areas includes the starting point, the processing unit, and the split transfer unit, and the execution function, when each of the information processing devices determined by the determination function is responsible for the processing of each of the starting points and the processing units as the processing of each area, inputs the data from the starting point and processes the input data, and when each of the information processing devices determined by the determination function is responsible for the processing of the split transfer unit as the processing of each area, executes data transfer between the information processing devices determined by the determination function, and the information processing devices have an execution process that shares the processing of the starting point and the processing unit as the processing of the execution function, and a transfer process that shares the processing of the split transfer unit as the processing of the execution function. The information processing device further includes one or more information processing devices for processing information, the information processing device having a division function for dividing processing information indicating a processing procedure for data from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition, a determination function for uniquely determining the responsibility of each piece of the processing information divided by the division function to one of the information processing devices, and an execution function for executing the processing in the information processing device determined by the determination function, the processing information including a plurality of processing units arranged between each of the starting points and the end point, and sequentially processing data from at least one of the respective starting points, and a plurality of transfer candidate units arranged between any of the processing units, the information processing device further having a division position recommendation function for calculating at least one of a CPU processing time and a data output amount of the plurality of processing units based on sample data corresponding to the processing information, the division position recommendation function being configured to recommend a division position based on the division position of the plurality of processing units belonging to the sample data, the processing information is updated based on at least one of the calculated total value of the CPU processing time and the data transfer amount by the transfer candidate parts from each of the combinations of transfer candidate parts, the processing information is updated based on at least one of the calculated total value of the CPU processing time and the data transfer amount by the transfer candidate parts from each of the combinations of transfer candidate parts, the processing information is updated based on at least one of the calculated total value of the CPU processing time and the data transfer amount by the transfer candidate parts from each of the combinations of transfer candidate parts, the processing information is updated based on at least one of the calculated total value of the CPU processing time and the data transfer amount by the transfer candidate parts from each of the combinations of transfer candidate parts from each of the combinations of transfer candidate parts, the processing information is updated based on at least one of the calculated total value of the CPU processing time and the data transfer amount by the transfer candidate parts from each of the combinations of transfer candidate parts from each of the combinations of transfer candidate parts from each of the combinations of transfer candidate parts from each of the combinations of transfer candidate parts from each of the processing parts from each of the transfer candidate parts to each of the processing parts from each of the transfer candidate ... Effect of the Invention

[0008] According to the present invention, in a distributed environment, it is possible to realize parallel processing of data from multiple starting points to one or more end points and to merge the data after parallel processing, thereby making it possible to speed up data processing. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram illustrating an example of a schematic configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a configuration diagram illustrating an example of a data flow according to the first embodiment. [Diagram 3] 1 is a configuration diagram illustrating an example of a hardware configuration of an information processing device according to a first embodiment. [Figure 4] 2 is a configuration diagram showing an example of the configuration of functions and data of the support device according to the first embodiment; FIG. [Diagram 5] 2 is a configuration diagram showing an example of the configuration of functions and data of a management device according to the first embodiment; FIG. [Figure 6] 2 is a configuration diagram showing an example of the function and data configuration of the data flow processing device according to the first embodiment; FIG. [Figure 7] 2 is a configuration diagram showing an example of the function and data configuration of the data storage device according to the first embodiment; FIG. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of area responsibility information according to the first embodiment. [Figure 9] FIG. 4 is a diagram illustrating a configuration example of transfer step information according to the first embodiment. [Figure 10] FIG. 11 is a sequence diagram illustrating an example of a distributed execution sequence of a data flow according to the first embodiment. [Figure 11] FIG. 2 is a configuration diagram showing an example of dividing a data flow according to the first embodiment. [Figure 12] FIG. 11 is a sequence diagram illustrating an example of a sequence of a covering area determination process according to the first embodiment. [Figure 13] 1 is a flowchart illustrating an example of a communication connection establishment process according to the first embodiment. [Figure 14] FIG. 4 is a sequence diagram illustrating an example of a communication connection setting sequence according to the first embodiment. [Figure 15] FIG. 11 is a sequence diagram illustrating an example of a transfer sequence of intermediate data between data flow processing devices according to the first embodiment. [Figure 16] FIG. 11 is a configuration diagram illustrating an example of a configuration of an information processing system according to a second embodiment. [Figure 17] FIG. 11 is a configuration diagram showing an example of division of a data flow according to a second embodiment. [Figure 18] FIG. 11 is a configuration diagram illustrating an example of a configuration of an information processing system according to a third embodiment. [Figure 19] FIG. 11 is a configuration diagram illustrating an example of a configuration of a data flow processing device according to a fourth embodiment. [Figure 20] FIG. 13 is a diagram illustrating an example of a configuration of transfer step information according to the fourth embodiment. [Figure 21] FIG. 13 is a diagram illustrating an example of the configuration of connection information according to the fourth embodiment. [Figure 22] FIG. 13 is a sequence diagram illustrating an example of a communication connection setting sequence according to the fourth embodiment. [Diagram 23] FIG. 13 is a sequence diagram illustrating an example of a transfer sequence of intermediate data according to the fourth embodiment. [Figure 24] FIG. 13 is a configuration diagram showing an example in which a candidate for a transfer step insertion position is inserted into a data flow according to a fifth embodiment. [Diagram 25] 13 is a flowchart illustrating an example of a process of a division position recommendation function according to the fifth embodiment. [Figure 26] FIG. 13 is a configuration diagram showing an example in which a transfer step insertion position is presented in a data flow according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0011] A first embodiment will be described with reference to Figures 1 to 15. In this embodiment, an example will be described in which a data flow having a plurality of starting points is executed in an information processing system 101 having a plurality of information processing devices.

[0012] Fig. 1 is a schematic diagram showing an example of the schematic configuration of an information processing system according to a first embodiment. In Fig. 1, the information processing system 101 includes, for example, a support device 103, a management device 104, and a plurality of data flow processing devices 105, and is connected to a user terminal 102 and a plurality of data storage devices 106 via a network 107. The user terminal 102, the support device 103, the management device 104, each data flow processing device 105, and each data storage device 106 are connected to each other via the communication network 107. The information processing system 101 distributes data flow information received from the user terminal 102 to each data flow processing device 105 and executes processing.

[0013] The communication network 107 is configured by a wired or wireless communication infrastructure such as a local area network (LAN), a wide area network (WAN), the Internet, an intranet, a dedicated line, a mobile phone network, or optical fiber.

[0014] The user terminal 102 is composed of an information processing device with a data flow design function equipped with a GUI (Graphical User Interface), and generates data flow information that describes a series of processes from data input to output based on user operations, and transmits the generated data flow information to the network 107.

[0015] When the support device 103 receives data flow information transmitted from the user terminal 102 via the network 107, it accepts the registration of the received information and distributes the accepted data flow information to each data flow processing device 105.

[0016] When a data flow is divided by each data flow processing device 105, the management device 104 is used to assign a data flow processing device 105 that executes the processing of each divided data flow. This assignment process will be described later.

[0017] When each dataflow processing device 105 receives dataflow information distributed from the support device 103 via the network 107, it divides the dataflow into multiple areas based on the transfer steps recorded in the dataflow, determines the dataflow processing device 105 responsible for executing each area for each area, and stores information indicating the combination of each area and the dataflow processing device 105 responsible for each area (i.e., area responsibility information) as information shared by each dataflow processing device 105. After that, each dataflow processing device 105 executes the process belonging to the area it is responsible for. Note that an area is the range in which dataflow information is recorded, and is defined as the range executed by one dataflow processing device 105.

[0018] Each data storage device 106 is, for example, a general database, records various data, and reads the data in response to a read request from various information processing devices. Note that each data storage device 106 is not necessarily a database as long as it has a data storage function and a data reading function. Also, it may be included in the information system 101.

[0019] The user terminal 102, the support device 103, the management device 104, each data flow processing device 105, and each data storage device 106 are information processing devices (e.g., servers), and may be configured using virtual information processing resources such as a cloud server or a container in a cloud system, or may be physical information processing devices. Furthermore, the number of these devices is not limited to the number shown in the figure, and may be one, or two or more. Furthermore, one device may have the functions of another device.

[0020] Fig. 2 is a configuration diagram showing an example of a data flow according to the embodiment 1. In Fig. 2, the user terminal 102 creates information of a data flow 201 that describes a series of data processing from data input to data output based on a user operation. The data flow 201 is information generated by the user terminal 102 based on the user's operation, and includes a first data input (i.e., "data input 1") 211, a second data input (i.e., "data input 2") 212, and a third data input (i.e., "data input 3") 213 as data input units that are the starting points for inputting data, a first process (i.e., "process 1") 221 to a fifth process (i.e., "process 5") 225 as processing units that sequentially execute data processing such as statistical processing of data and anonymization processing of data, a data output 231 as a data output unit that is the end point for outputting data, and a first transfer step (i.e., "transfer 1") 241 and a second transfer step (i.e., "transfer 2") 242 as transfer units that are located between any of the processing units and transfer data between the processing units, which are switching points of the data flow processing device 105. In this case, the information of the data flow 201 is configured as processing information that indicates the data processing procedure from multiple starting points to one end point.

[0021] The first transfer step (i.e., "Transfer 1") 241 is inserted by user operation as a transfer step connecting the first process (i.e., "Process 1") 221 and the third process (i.e., "Process 3") 223, and the second transfer step (i.e., "Transfer 2") 242 is inserted by user operation as a transfer step connecting the fifth process (i.e., "Process 5") 225 and the fourth process (i.e., "Process 4") 224.

[0022] In the data flow 201, data input to the first data input (i.e., "data input 1") 211 is subjected to statistical processing, etc. in the first process (i.e., "process 1") 221, and then transferred to the third process (i.e., "process 3") 223 via the first transfer step (i.e., "transfer 1") 241. Data input to the second data input (i.e., "data input 2") 212 is subjected to statistical processing, etc. in the second process (i.e., "process 2") 222, and then output to the third process (i.e., "process 3") 223. In the third process (i.e., "process 3") 223, the data transferred from the first transfer step (i.e., "transfer 1") 241 and the data input from the second process (i.e., "process 2") 222 are collectively subjected to statistical processing, etc., and then output to the fourth process (i.e., "process 4") 224. The data input to the third data input (i.e., "data input 3") 213 is subjected to statistical processing etc. in a fifth process (i.e., "process 5") 225, and then transferred to a fourth process (i.e., "process 4") 224 via a second transfer step (i.e., "transfer 2") 242. In the fourth process (i.e., "process 4") 224, the data transferred from the second transfer step (i.e., "transfer 2") 242 and the data input from the third process (i.e., "process 3") 223 are collectively subjected to statistical processing etc., and then output to a data output 231.

[0023] Fig. 3 is a configuration diagram showing an example of a hardware configuration of an information processing device according to the first embodiment. In Fig. 3, the information processing device 300 can be used as each of the user terminal 102, the support device 103, the management device 104, the data flow execution device 105, and the data storage device 106. In this case, the information processing device 300 includes, for example, a processor 301, a main storage device 302, an auxiliary storage device 303, an input device 304, an output device 305, and a communication device 306. These are connected to each other so as to be able to communicate with each other via a communication means such as a bus (not shown).

[0024] The processor 301 is configured using, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor 301 reads and executes a computer program stored in the main memory device 302, thereby realizing various functions of the user terminal 102, the support device 103, the management device 104, the data flow processing device 105, and the data storage device 106.

[0025] The main memory device 302 is a device that stores computer programs and data, and is, for example, a read only memory (ROM), a random access memory (RAM), or a non-volatile semiconductor memory.

[0026] The auxiliary storage device 303 is, for example, a hard disk drive, a solid state drive (SSD), an optical storage medium (i.e., a compact disc (CD) and a digital versatile disc (DVD)), a storage system, an integrated circuit card (IC card), a storage device for reading / writing a recording medium such as an SD memory card, a storage area of ​​a cloud server, etc. The computer programs and data stored in the auxiliary storage device 303 are read into the main storage device 302 as needed.

[0027] The input device 304 is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device, etc. The output device 305 is a user interface that provides the user with various information such as the progress of processing and the results of processing. The output device 305 is, for example, a screen display device (i.e., a liquid crystal monitor, LCD (Liquid Crystal Display), or a graphic card), a voice output device (i.e., a speaker, etc.), or a printer, etc. For example, the information processing device 300 may input and output information between itself and another device via the communication device 306.

[0028] The communication device 306 is a wired or wireless communication interface that realizes communication with other devices via a communication means such as a LAN, the Internet, etc. The communication device 306 is, for example, a network interface card (NIC), a wireless communication module, a universal serial bus (USB) module, or a serial communication module.

[0029] FIG. 4 is a configuration diagram showing an example of the configuration of functions and data of the support device according to the first embodiment. In FIG. 4, the support device 103 includes a data flow registration function 401 and a data flow distribution function 402 stored in the auxiliary storage device 303, and records resource information 403 and data flow information 404 in the auxiliary storage device 303. The resource information 403 records information (e.g., IP (Internet Protocol) address, location information, etc.) of each data flow processing device 105 constituting the information processing system 101. When the data flow registration function 401 receives data flow information transmitted from the user terminal 102, it registers the received data flow information in the data flow information 404. The data flow distribution function 402 distributes all or part of the information registered in the resource information 403 and the data flow information 404 to each data flow processing device 105.

[0030] FIG. 5 is a configuration diagram showing an example of the configuration of functions and data of the management device according to the first embodiment. In FIG. 5, the management device 104 has a response function 501 stored in the auxiliary storage device 303, and records area assignment information 502 in the auxiliary storage device 303. When the response function 501 receives an assignment registration request including area ID (Identification) information from each data flow processing device 105, it checks whether the area ID attached to the received assignment registration request has been registered in the area assignment information (i.e., area assignment information for management device) 502, and if the received area ID has not been registered, it records the area ID and the IP address of the data flow processing device 105 that is the sender of the assignment registration request in the area assignment information 502, and returns a response of registration OK to the data flow processing device 105 that is the sender of the assignment registration request. Also, if the received area ID has been registered, the response function 501 returns a response of registration NG to the data flow processing device 105 that is the sender of the assignment registration request. Furthermore, when the response function 501 receives a request for area assignment information from each data flow processing device 105, it returns the area ID recorded in the area assignment information 502 and the IP address of the data flow processing device 105 that is in charge of the area ID.

[0031] Fig. 6 is a configuration diagram showing an example of the configuration of functions and data of the data flow processing device according to the first embodiment. In Fig. 6, the data flow processing device 105 includes a data flow receiving function 601, a data flow dividing function 602, a transfer step preprocessing function 603, a data flow execution function 604, and a responsible area adjustment function 605 as functions stored in the auxiliary storage device 303, and records data flow information 606, transfer step information 607, and area responsible information 608 in the auxiliary storage device 303. When the data flow receiving function 601 receives data flow information transmitted from the support device 103, it records the received data flow information in the data flow information 606. The data flow dividing function 602 divides the data flow information (i.e., processing information) received by the data flow receiving function 601 into multiple pieces using parallel processing by the data flow processing device 105 as a division condition. In this case, when dividing the data flow information (i.e., processing information) into a plurality of parts, the data flow division function 602 is configured as a division function or division unit that divides the range in which the processing information is recorded into a plurality of areas, assigns an ID to each divided area, and records the ID in the data flow information 606. The assigned area adjustment function 605 is configured as a determination function or determination unit that cooperates with the management device 104 to determine the data flow processing device 105 that will be in charge of each divided area, and records the determined content in the area assignment information 608. The transfer step pre-processing function 603 is a function that controls communication between the data flow processing devices, and is configured as a communication function or communication unit that establishes a communication connection before transferring data between the data flow processing devices that are in charge of each divided area, and records information on the established communication connection in the transfer step information 607. The data flow execution function 604 is configured as an execution function or execution unit that executes the data flow of the area in charge.

[0032] FIG. 7 is a configuration diagram showing an example of the configuration of the functions and data of the data storage device according to the first embodiment. In FIG. 7, the data storage device 106 has a data recording function 701 and a data reading function 702 as functions stored in the auxiliary storage device 303, and records various data in data 703 as a data storage unit or a data storage area. The data recording function 701 and the data reading function 702 record and read data in the data 703 in response to a request from each data flow processing device 105. The data storage device 106 may receive and execute a request to record and read data from an information processing device (not shown) via the network 107 or a network (not shown). The data storage device 106 may or may not be included in the information processing system 101.

[0033] FIG. 8 is a configuration diagram showing an example of the configuration of area assignment information according to the first embodiment. Each piece of data recorded by the support device 103, the management device 104, and the dataflow processing device 105 is recorded, for example, in a table format. In this case, the area assignment information 502 stores a record having each item of an area ID 801 and a dataflow processing device IP address 802, as shown in FIG. 8. In the area ID 801, an area identifier (for example, "Area-1") assigned to each dataflow after division is recorded. In the dataflow processing device IP address 802, an IP address (for example, "aaa") of the dataflow processing device 105 responsible for processing the area stored in the area ID 801 is recorded. The area assignment information 608 also has a similar configuration.

[0034] FIG. 9 is a configuration diagram showing a configuration example of the transfer step information according to the first embodiment. In FIG. 9, the transfer step information 607 stores a record having each item of a transfer step ID 901, a transfer source area ID 902, a transfer source IP address 903, a transfer destination area ID 904, a transfer destination IP address 905, a transfer step type 906, a connection ID 907 with a connection destination, and an intermediate data format 908. The transfer step ID 901 stores identification information (e.g., "Transfer 1") of a transfer step included in a data flow. The transfer source area ID 902 records identification information (e.g., "Area-1") of an area including a process up to the previous stage of a transfer step. The transfer source IP address 903 records an IP address (e.g., "aaa") of the data flow processing device 105 in charge of the area stored in the transfer source area ID 902. The transfer destination area ID 904 records identification information (e.g., "Area-2") of an area including a process from the latter stage of a transfer step. The destination IP address 905 records the IP address (e.g., "bbb") of the data flow processing device 105 responsible for the area recorded in the destination area ID 904. The transfer step type 906 records whether the role of the transfer step in the data flow of the area responsible for the data flow processing device 105 that holds this transfer step information 607 is sending or receiving. The connection ID 907 with the connection destination records the identification information (e.g., "S01") of the communication connection established with the destination data flow processing device 105. The intermediate data format 908 records the format of the data transferred between data flow processing devices.

[0035] Fig. 10 is a sequence diagram showing an example of a distributed execution sequence of a dataflow according to the first embodiment. In Fig. 10, when a user creates a dataflow by operating the user terminal 102, information on the created dataflow is transmitted from the user terminal 102 to the support device 103 (S1001). The dataflow registration function 401 of the support device 103 registers the received dataflow information in the dataflow information 404 (S1002). The dataflow distribution function 402 of the support device 103 distributes the dataflow information registered in the dataflow information 404 to each dataflow processing device 105 in the information processing system 101 (S1003). In addition, if the dataflow registration function 401 has a portal site for instructing the registration and execution of a dataflow, the user may access the portal site via the user terminal 102 to register a dataflow and instruct the execution of the dataflow.

[0036] Each dataflow processing device 105 in the distributed environment records the received dataflow information in the dataflow information 606 (S1004). Next, each dataflow processing device 105 divides the received dataflow information into a plurality of areas (S1005), and determines the dataflow processing device 105 in charge of each area in cooperation with the management device 104 (S1006). After that, the dataflow processing device 105 in charge of each area establishes a communication connection with the dataflow processing device 105 to which data is to be transferred (S1007). After that, the dataflow processing device 105 in charge of each area executes the processing of the dataflow in its area (S1008).

[0037] Fig. 11 is a configuration diagram showing an example of dividing a data flow according to the embodiment 1. In Fig. 11, a data flow 201 is configured as processing information indicating a data processing procedure from multiple starting points to one end point, similar to the data flow 201 shown in Fig. 2.

[0038] At this time, the data flow division function 602 of the data flow processing device 105 divides the data flow 201 into three areas 251 to 253 based on the first transfer step (i.e., "Transfer 1") 241 to the second transfer step (i.e., "Transfer 2") 242, and assigns area IDs including sequential numbers (e.g., "Area-1" to "Area-3") to each of the areas 251 to 253. In addition, the data flow division function 602 divides the first transfer step (i.e., "Transfer 1") 241 as a transfer unit into first transfer steps (i.e., "Transfer 1") 241-1 and 241-2, and divides the second transfer step (i.e., "Transfer 2") 242 as a transfer unit into second transfer steps 242-1 and 242-2, and manages the first transfer step (i.e., "Transfer 1") 241-1 and 241-2 and the second transfer steps 242-1 and 242-2 as divided transfer units, respectively.

[0039] In this case, the data flow division function 602 connects the first process (i.e., "Process 1") 221 to the first transfer step (i.e., "Transfer 1") 241-1, connects the first transfer step (i.e., "Transfer 1") 241-2 to the third process (i.e., "Process 3") 223, connects the fourth process (i.e., "Process 4") 224 to the second transfer step (i.e., "Transfer 2") 242-2, and connects the fifth process (i.e., "Process 5") 225 to the second transfer step (i.e., "Transfer 2") 242-1.

[0040] The area 251 in the divided data flow 202 is composed of the first data input (i.e., "data input 1") 211, the first process (i.e., "process 1") 221, and the first transfer step (i.e., "transfer 1") 241-1. The area 252 is composed of the first transfer step 241-2, the second data input (i.e., "data input 2") 212, the second process (i.e., "process 2") 222 to the fourth process (i.e., "process 4") 224, the data output 231, and the second transfer step (i.e., "transfer 2") 242-2. The area 253 is composed of the third data input (i.e., "data input 3") 213, the fifth process (i.e., "process 5") 225, and the second transfer step (i.e., "transfer 2") 242-1. This allows the information of the data flow 202 to be accurately distributed to each of the areas 251 to 253.

[0041] The first transfer steps 241-1 and 242-2 are processes for transferring the processing result (hereinafter referred to as intermediate data) of "Process 1" which is an earlier stage than "Transfer 1" to "Process 3" which is a later stage than "Transfer 1", and the second transfer steps 242-1 and 242-2 are processes for transferring intermediate data which is the processing result of "Process 5" which is an earlier stage than "Transfer 2" to "Process 4" which is a later stage than "Transfer 2". Therefore, when a communication connection is established between the data flow processing device 105 which handles the area 251 and the data flow processing device 105 which handles the area 252, intermediate data transfer is executed between them as the processing of the first transfer steps 241-1 and 242-2. Also, when a communication connection is established between the data flow processing device 105 which handles the area 252 and the data flow processing device 105 which handles the area 253, intermediate data transfer is executed between them as the processing of the second transfer steps 242-1 and 242-2.

[0042] In this case, the first transfer step ("Transfer 1") 241-1 becomes a transfer step belonging to area 251 (i.e. Area-1) of the source of the intermediate data, and the first transfer step ("Transfer 1") 241-2 becomes a transfer step belonging to area 252 (i.e. Area-2) of the destination of the intermediate data. In addition, "From:Area-1" is added to the first transfer steps ("Transfer 1") 241-1 and 241-2 as area information indicating the source of the intermediate data, and "To:Area-2" is added as area information indicating the destination of the intermediate data.

[0043] The second transfer step ("Transfer 2") 242-1 is a transfer step belonging to the area 253 (i.e., Area-3) of the source of the intermediate data, and the second transfer step ("Transfer 2") 242-2 is a transfer step belonging to the area 252 (i.e., Area-2) of the destination of the intermediate data. In addition, "From:Area-3" is assigned to the second transfer steps ("Transfer 2") 242-1 and 242-2 as information indicating the area from which the intermediate data is transferred, and "To:Area-2" is assigned as information indicating the area to which the intermediate data is transferred. The relationship between these destination area IDs and transfer steps is managed by the data flow division function 602.

[0044] The dataflow division function 602 of each dataflow processing device 105 performs the dataflow division process (step S1005 in FIG. 10) using the same division program. As a result, all dataflow processing devices 105 obtain the same division result. In this embodiment, the dataflow division process (step S1005 in FIG. 10) is performed in each dataflow processing device 105, but the support device 103 may have the dataflow division function 602, divide the dataflow in the support device 103, and distribute information on the dataflow after division to each dataflow processing device 105.

[0045] In this case, for example, the support device 103, which transmits and receives information to and from the user terminal 102 and each information processing device 105, can be provided with a division function for the support device which, when receiving processing information from the user terminal 102, divides the range in which the received processing information is recorded and which is to be executed by the dataflow processing device 105 into multiple areas using parallel processing by the dataflow processing device 105 as a division condition, a determination function for the support device which uniquely determines which of the dataflow processing devices 105 will be in charge of processing each of the areas divided by the division function for the support device, and a notification function for the support device which notifies (i.e., transmits) to each of the dataflow processing devices 105 determined by the determination function for the support device that the processing person has been determined, information indicating the processing content of each area, and information on the dataflow processing device 105 in charge of each area. This allows each dataflow processing device 105 to execute the processing of each area based on the notification from the support device 103.

[0046] Fig. 12 is a sequence diagram showing an example of the sequence of the covering area determination process according to the first embodiment. In Fig. 12, the covering area adjustment function 605 of the data flow processing device 105 can use a leader selection algorithm when adjusting the covering area. The covering area adjustment function 605 selects, for example, one of the three areas 251 to 253 as the covering desired area (S1201), and transmits a registration request (i.e., a covering registration request) including the area ID of the covering desired area and the IP address of the request sender (i.e., the data flow processing device 105) to the management device 104 (S1202).

[0047] The response function 501 of the management device 104 that has received the registration request refers to the area assignment information 502 based on the received registration request, and checks whether the area ID attached to the received registration request has already been registered in the assignment area information 502 (S1203). If the assignment of the received area ID is not registered (No in S1203), the response is OK (S1204), and the received area ID and the IP address of the data flow processing device 105 that sent the request are recorded (i.e., registered) in the area assignment information 502 (S1206). On the other hand, if the received ID has already been registered (Yes in S1203), the response function 501 returns NG (S1205). After this, the response function 501 transmits response information to the data flow processing device 105 (S1207).

[0048] The responsible area adjustment function 605 that received the response determines whether the received response is OK (S1208), and if the received response is OK (Yes in S1208), transmits a responsible area information request to the management device 104 (S1212). The response function 501 of the management device 104 that received the responsible area information request transmits to the data flow processing device 105 a pair of an area ID registered in the responsible area information 502 and the IP address of the data flow processing device 105 that is responsible for that area (S1213).

[0049] Next, the assigned area adjustment function 605 of the data flow processing device 105 determines whether there are any areas for which responsibility has not yet been determined (S1214), and if there are any areas for which responsibility has not yet been determined (Yes in S1214), it waits for a predetermined time (S1215) and then sends a request for assigned area information to the management device 104 again (S1212), and if there are no areas for which responsibility has not yet been determined (No in S1214), it links each area ID with the IP address of the data flow processing device 105 responsible for each area and records them in the area assignment information 608 (S1216).

[0050] On the other hand, when the determination result in step S1208 is No, the responsible area adjustment function 605 inquires of the management device 104 about responsible area information (S1209), checks whether there is an area for which responsibility has not been determined (S1210), and if there is an area for which responsibility has not been determined (Yes in S1210), selects one desired area for which responsibility has not been determined from the areas for which responsibility has not been determined, and repeats the processing from S1201 onwards. If there is no area for which responsibility has not been determined in step S1210 (No in S1210), the responsible area adjustment function 605 enters a standby state (S1211). Note that in step S1209, the responsible area adjustment function 605 transmits and receives information to and from the management device 104, similarly to steps S1212 and S1213.

[0051] This processing sequence uniquely determines which data flow processing device 105 is responsible for each area, and each data flow processing device 105 determines which area it is responsible for and can obtain information on the combination of each area and the data flow processing device 105 that executes the data flow of each area from the area responsibility information 608.

[0052] In steps S1212 to S1215, the number of executions of the process may be recorded, and when the number of executions exceeds a preset threshold, an error may be detected and the process may be terminated. Also, one of the dataflow processing devices 105 may be selected as a leader (for example, the dataflow processing device 105 with the smallest IP address may be the leader), and the responsible area adjustment function 605 of the dataflow processing device 105 that has become the leader may sequentially select one dataflow processing device 105 from the dataflow processing devices 105, sequentially assign the selected one dataflow processing device 105 as the person in charge of each area, record each area and one dataflow processing device 105 in the area responsible information 608 in association with each area, and distribute the recorded area responsible information 608 to each dataflow processing device 105. In this way, the responsible area adjustment function 605 that becomes the leader may uniquely determine the dataflow processing device 105 that is in charge of each area. Also, each dataflow processing device 105 may hold the area responsible information 608 as information shared by each device.

[0053] 13 is a flowchart illustrating an example of a communication connection establishment process according to the embodiment 1. After the data flow is divided, data can be transferred between the data flow processing devices via the communication connection.

[0054] 13, the transfer step pre-processing function 603 of the dataflow processing device 105 extracts all transfer steps (i.e., transfer steps 241-1 to 242-2) from the area of ​​the dataflow 202 shown in FIG. 11 that it is responsible for (S1301), and records the transfer step ID 901, the source area ID 902, the source IP address 903, the destination area 904, the destination IP address 905, and the transfer step type 906 for all the extracted transfer steps 241-1 to 242-2 in the transfer step information 607 (S1302). In the dataflow processing device 105 responsible for Area-1, the transfer step 241-1, in the dataflow processing device 105 responsible for Area-2, the transfer steps 241-2 and 242-2, and in the dataflow processing device 105 responsible for Area-3, the transfer step 242-1 is extracted and recorded.

[0055] At this time, when the transfer step 241-1 is extracted, the transfer step pre-processing function 603 refers to the area handling information 608 shown in FIG. 8, and records "Transfer 1" as information for identifying the transfer step 241-1 in the transfer step ID 901 of the transfer step information 607 shown in FIG. 9, records "Area-1" as a first division identifier that uniquely identifies the area 251 to which the transfer step 241-1 belongs (i.e., first communication information indicating the area of ​​the transfer source) in the transfer source area ID 902, and records "Area-2" as a first device identifier that uniquely identifies the data flow processing device 105 in charge of processing the area 251 to which the transfer step 241-1 belongs (i.e., the data flow processing In the transfer destination area 904, "Area-2" is recorded as a second division identifier (i.e., third communication information indicating the area of ​​the transfer destination) that uniquely identifies the area 252 to which the transfer step 241-2 belongs, in the transfer destination IP address 905, "bbb" is recorded as a second device identifier (i.e., fourth communication information indicating the data flow processing device of the transfer destination) that uniquely identifies the data flow processing device 105 that is in charge of processing the area 252 to which the transfer step 241-2 belongs, and in the transfer step type 906, "send" is recorded as the communication type between the data flow processing devices (i.e., fifth communication information specifying the communication type as send or receive). The transfer source IP address 903 and the transfer destination IP address 905 can be specified from the transfer source area ID 902 and the transfer destination area ID 904, respectively, based on the area assignment information 608.

[0056] Next, the transfer step pre-processing function 603 selects one transfer step (e.g., "Transfer 1") identified by the transfer step ID 901 from the transfer step information 607 (S1303), and performs a communication connection setting process (S1304) for the selected transfer step. After that, the transfer step pre-processing function 603 repeats steps S1303 to S1305, and performs the communication connection setting process (S1304) for all transfer steps 241-1 to 242-2.

[0057] When the communication connection setting process is performed for all the transfer steps 241-1 to 242-2, the transfer step pre-processing function 603 sets the re-processing execution counter i to 0 (S1306) and checks whether there is data registered in the re-execution list (S1307). The re-execution list is a list in which information on the transfer steps in which the communication connection setting has failed is recorded, and is created during the communication connection setting process described below, and the ID of the transfer step is recorded. At this time, if there is no data in the re-execution list (No in S1307), the transfer step pre-processing function 603 ends the process (S1308), and if there is data in the re-execution list (Yes in S1307), the transfer step pre-processing function 603 copies the re-execution list to a temporary list and then erases the data in the re-execution list to clear the re-execution list (S1309).

[0058] Thereafter, the transfer step pre-processing function 603 increments the re-processing execution counter i by one (S1310), compares the value of the re-processing execution counter i with a preset threshold value (S1311), and if i<threshold value (Yes in S1311), waits for a predetermined time (S1313), selects one transfer step from the temporary list (S1314), and re-executes the communication connection setting process for the selected transfer step (S1315). After this, the transfer step pre-processing function 603 determines whether the communication connection setting process has been performed for all transfer steps in the temporary list (S1316), and if not all have been performed (No in S1316), repeats the processes from step S1314 to S1316, and if all have been performed (Yes in S1316), returns to step S1307 and checks whether there is data in the re-execution list.

[0059] If the value of the re-processing execution counter i does not satisfy the condition i<threshold in step S1311, the transfer step pre-processing function 603 determines that an error has occurred and ends the processing of this routine (S1312).

[0060] Fig. 14 is a sequence diagram showing an example of a communication connection setting sequence according to the first embodiment. This communication connection setting sequence is an example of the specific contents of the communication connection setting process (S1304) in Fig. 13, and the communication connection setting process (S1304) is executed for each transfer step. In Fig. 14, the transfer step pre-processing function 603 checks whether the type of the transfer step is transmission or reception (S1401), and when the transfer type is reception, ends the process (S1409), and when the transfer step type is transmission, transmits a connection establishment request including the transfer step ID and the IP address of the data flow processing device of the transmission source to the data flow processing device 105 of the connection destination (i.e., the transfer destination) (S1402).

[0061] When the destination data flow processing device 105 receives the connection request, the transfer step pre-processing function 603 refers to the transfer step information 607 based on the transfer step ID and the IP address of the data flow processing device 105 that is the source of the connection request, and checks whether there is connection information (transfer step ID, transfer source IP address, type of transfer step) corresponding to the transfer step information 607 (S1403). The corresponding information is information in which the transfer step ID 901 and the transfer source IP address 903 match the received transfer step ID and the source IP address, respectively, and the type of transfer step is reception. If there is no corresponding information in the transfer step information 607, the destination transfer step pre-processing function 603 returns information of connection NG to the source data flow processing device 105 (S1404), and then ends the processing in this routine (S1405). If there is corresponding information in the transfer step information 607, it sends information of connection OK to the source data flow processing device 105 (S1406).

[0062] When the transfer step pre-processing function 603 of the connection request source receives connection OK information from the destination data flow processing device 105 (Yes in S1407), it establishes a communication connection with the destination data flow processing device 105 (S1410) and records the connection ID 907 with the destination in the transfer step information 607 (S1411). The recording of the connection ID 907 with the destination of the transfer step information 607 is also performed in the destination data flow processing device 105 (S1412). When the connection is NG (No in S1407), the transfer step pre-processing function 603 of the connection request source adds the transfer step ID to the re-execution list (S1408), and then ends the processing in this routine (S1409).

[0063] In step S1408, for the transfer step added to the re-execution list, the communication connection establishment process is re-executed by the processes of steps S1307 to S1316 shown in Fig. 13. The re-execution of the communication connection establishment process (S1315 in Fig. 13) is the process after step S1402 in Fig. 14.

[0064] 10, information about the dataflow processing devices 105 required for parallel processing of data flows and merging is shared between the dataflow processing devices 105, and communication connections required for transferring intermediate data are established. After these processes are performed, that is, after communication connections between the dataflow processing devices are established, each dataflow processing device 105 executes processing of the dataflow in its own area (step S1008 in FIG. 10).

[0065] 15 is a sequence diagram showing an example of a transfer sequence of intermediate data between data flow processing devices according to Example 1. The transfer of intermediate data is executed in a transfer step when the data flow execution function 604 of the data flow processing device 105 executes a data flow.

[0066] In transferring intermediate data, the data flow processing device 105 to which the intermediate data is to be transferred needs to understand the format of the intermediate data. Therefore, the data flow processing device 105 from which the intermediate data is to be transferred notifies the data flow processing device 105 to which the intermediate data is to be transferred of the format of the intermediate data before the first transfer of the intermediate data.

[0067] 15, when the dataflow execution function 604 of the dataflow processing device 105 of the data transfer source receives intermediate data from the previous process in the transfer step 241-1 (S1501), it judges whether it is the first transfer or not (S1502), and if it is the first transfer (Yes in S1502), it transmits information indicating the data format of the intermediate data to the dataflow processing device 105 of the data transfer destination (S1503). The dataflow execution function 604 of the dataflow processing device 105 of the data transfer destination records the information indicating the data format of the intermediate data in the transfer step information 607 (S1504).

[0068] The data flow processing device 105 of the data transfer source converts the intermediate data into a format for transmission (for example, serializes it) (S1505) and transmits the converted data in the transmission format to the data flow processing device 105 of the data transfer destination (S1506). When the data flow processing device 105 of the data transfer destination receives data in the transmission format, it converts the received data in the transmission format into the data format of intermediate data (S1507) and transmits the converted intermediate data to a subsequent processing stage (S1508).

[0069] The transmission of the information indicating the data format of the intermediate data and the transmission of the intermediate data are performed using each communication connection established for each transfer step by the communication connection establishment process. Moreover, the processes other than the transfer step in which data transfer is performed are the same as the execution of the data flow performed within one data flow processing device 105.

[0070] In addition, if an error occurs in any of the processes of dividing the data flow (step S1005 in FIG. 10), determining the area of ​​responsibility (step S1006 in FIG. 10), or establishing a communication connection (step S1007 in FIG. 10), the function in charge of each process may notify the support device 103 of the error, and the support device 103 may notify the user terminal 102 of the error.

[0071] In this embodiment, when each dataflow processing device 105 is in charge of processing the data input unit (i.e., "Data Input 1" 211 to "Data Input 3" 213) and the processing unit (i.e., "Processing 1" 221 to "Processing 5" 225) as the processing of each area, the dataflow execution function 604 inputs data from the start point and executes processing of the input data (e.g., statistical processing and anonymization processing), and when each dataflow processing device 105 is in charge of processing the division and transfer unit (i.e., "Transfer 1" 241-1, 241-2, "Transfer 2" 242-1, and 242-2) as the processing of each area, the dataflow execution function 604 executes data transfer between information processing devices. This allows the processing in each area to be reliably executed by each dataflow processing device 105.

[0072] Furthermore, in this embodiment, the data format is notified (i.e., the information on the data format is transferred) before the first transfer of intermediate data, but the timing of notifying the data format is not limited to this and may be performed when the communication connection is established. Furthermore, in cases where the format of intermediate data differs for each piece of intermediate data, the format of the intermediate data may be notified each time before the intermediate data is transferred. Furthermore, when transferring intermediate data, it is not always necessary to notify the data format in advance; for example, by using the JSON format, the data type can be understood by the data flow processing device 105 at the transfer destination without notifying the data format in advance.

[0073] Furthermore, in this embodiment, a communication connection for transferring intermediate data is established, but it is not always necessary to establish a communication connection, and intermediate data may be transferred by connectionless communication.

[0074] Furthermore, when intermediate data has multiple records, for example, it may be transferred one record at a time, several records at a time, or all at once. The method of transferring intermediate data may be specified by the user when creating the data flow, or a default method may be specified. Also, the transfer method may be different for transfer steps included in the same data flow.

[0075] In addition, although the present embodiment shows an example in which one dataflow processing device 105 is responsible for one area, one dataflow processing device 105 may be responsible for multiple areas. In this case, the dataflow processing device 105 may execute the dataflow of each area individually, or may execute the dataflows of the multiple areas as one dataflow.

[0076] In this embodiment, as shown in Fig. 12, each dataflow processing device 105 determines the area in an autonomous distributed manner, but for example, the support device 103 may divide the dataflow and determine the dataflow processing device 105 in charge of each divided area. In this case, the support device 103 may randomly assign each area to each dataflow processing device 105. When the dataflow processing devices 105 are installed at locations distant from each other, each dataflow processing device 105 or the support device 103 may create multiple combinations of the source dataflow processing device 105 and the destination dataflow processing device 105 for each transfer step, and assign the dataflow processing device 105 in charge of each area so that the network distance between the source dataflow processing device 105 and the destination dataflow processing device 105 is close. When there are two or more transfer steps, the dataflow processing device 105 in charge of each area may be assigned so that the total network distance is small.

[0077] For example, the assigned area adjustment function 605 of each dataflow processing device 105 creates a predetermined number of combinations of dataflow processing devices 105 including the dataflow processing devices 105 for the number of areas. In each created combination, the dataflow processing devices 105 included in each combination are uniquely assigned to each area of ​​the dataflow, distance information indicating the total distance between the dataflow processing devices is stored in correspondence with each combination, and a combination in which the total distance between the dataflow processing devices is the smallest is identified based on the stored distance information, and the dataflow processing device 105 included in the identified combination can be determined as the processor of the area 251 or 253 including the source transfer step (i.e., division and transfer unit) 241-1 or 242-1 among the multiple areas, or the processor of the area 252 including the destination transfer step (i.e., division and transfer unit) 241-2 and 242-2. This allows data transfer between dataflow processing devices with the smallest total distance, making it possible to minimize the time required for data transfer. For example, it is also possible to select an optimal combination of dataflow processing devices 105 by machine learning so that the total distance between the dataflow processing devices 105 is minimized.

[0078] In addition, in this embodiment, a table notation is used as an example of each data, but this does not limit the recording method of each data to a table type, and various methods such as a list or a chain may be used to record the data. Furthermore, the recorded elements may be expressed in various forms, such as numbers, symbols, or mathematical expressions.

[0079] Further, the support device 103, the management device 104, the dataflow processing device 105, and the data storage device 106 may each be a virtual machine or a container. For example, the support device 103 may have a function for starting a virtual machine or a container, and may start the virtual machine or the container as the dataflow processing device 105 upon receiving a dataflow registration request from the user terminal 102. For example, the dataflow processing device 105 may have an execution container that shares the processing of the data input unit (i.e., "Data Input 1" 211 to "Data Input 3" 213) and the processing unit (i.e., "Process 1" 221 to "Process 5" 225) as the processing of the dataflow execution function 604, and a transfer container that shares the processing of the division and transfer unit (i.e., "Transfer 1" 241-1, 241-2, "Transfer 2" 242-1, 242-2). This allows a part of the dataflow processing device 105 to be virtualized.

[0080] According to this embodiment, which is configured in this manner, in a distributed environment, it is possible to realize parallel processing of data from multiple starting points to one or more end points and to merge the data after parallel processing, thereby making it possible to speed up data processing.

[0081] It is possible to easily realize parallel processing of data flows based on the information of transfer steps inserted by user operation, without the user explicitly specifying the data flow processing device 105 that executes the data flow of each area. EXAMPLES

[0082] This embodiment shows an example of data flow that is executed across two or more bases, such as countries or regions, multiple companies, multiple departments within the same company, etc. When handling data managed by different sources, it is necessary to comply with the laws and regulations of each country or region and the security policies of the companies in order to transfer data between bases.

[0083] FIG. 16 is a configuration diagram showing an example of the configuration of an information processing system according to a second embodiment. In FIG. 16, an information processing system 1601 includes a support device 103 and two or more bases 1602 to 1603, and each of the bases 1602 to 1603 includes a management device 104, a plurality of data flow processing devices 105, and a plurality of data storage devices 106. The user terminal 102, the support device 103, and each of the bases 1602 to 1603 are connected to each other via a network 107. In this case, the base 1602 is managed as a base name "base A", and the base 1603 is managed as a base name "base B". In this embodiment and the following embodiments, each of the devices 102 to 106 has the same functions as those in the first embodiment, so that the following description will be focused on the differences from the first embodiment.

[0084] Fig. 17 is a configuration diagram showing an example of division of a data flow according to the second embodiment. In Fig. 17, a data flow 1701 is information generated by the user terminal 102 based on a user's operation, and includes a first data input (i.e., "data input 1") 211, a second data input (i.e., data input 2) 212, and a third data input (i.e., "data input 3") 213 as a starting point for inputting data, a first process (i.e., "process 1") 221 to a fifth process (i.e., "process 5") 225 as processes for performing statistical processing of data, anonymization processing of data, etc., a first transfer step (i.e., "transfer 1") 261 and a second transfer step (i.e., "transfer 2") 262 as transfer steps that are switching points of the data flow processing device 105, and a data output 231 as an end point for outputting data.

[0085] At this time, the user inserts a transfer step at the switching point of the data flow processing device 105 for executing parallel processing as in the first embodiment, and also inserts a transfer step at the switching points of the bases 1602 and 1603. Furthermore, for each transfer step, the user specifies the execution base of the processing before the transfer step and the execution base of the processing after the transfer step. For example, the user inserts a first transfer step (i.e., "Transfer 1") 261 as a transfer step at the switching point of the data flow processing device 105. This first transfer step (i.e., "Transfer 1") 261 is given "From: base A" and "To: base A" as information (i.e., execution base identifier) ​​instructing that the processing before the transfer step and the processing after the transfer step are both executed at the base A. Furthermore, the user inserts a second transfer step (i.e., "Transfer 2") 262 as a transfer step at the switching points of the bases 1602 and 1603. This second transfer step (i.e., "Transfer 2") 262 is assigned "From: Site B" and "To: Site A" as information (i.e., execution site identifier) ​​indicating that the processing before the transfer step is executed at site B and the processing thereafter is executed at site A. The execution site indication is displayed as a configurable property, for example, by double-clicking each transfer step 261 or 262 when the user designs a data flow using a GUI.

[0086] The data flow division function 602, which has received information on the data flow 1701, divides the data flow 1701 into three regions 271-273 based on the first transfer step (i.e., "Transfer 1") 261 and the second transfer step (i.e., "Transfer 2") 262, and assigns region IDs including sequential numbers (e.g., "A1", "A2", "B1") to each of the regions 271-273. The data flow division function 602 also divides the first transfer step ("Transfer 1") 261, which is a transfer section, into first transfer steps (i.e., "Transfer 1") 261-1 and 261-2, which are divided transfer sections, and divides the second transfer step (i.e., "Transfer 2") 262, which is a transfer section, into second transfer steps ("i.e., "Transfer 2") 262-1 and 262-2, which are divided transfer sections. In this case, the area 271 of the divided data flow 1702 is composed of the first data input (i.e., "Data Input 1") 211, the first process (i.e., "Process 1") 221, and the first transfer step (i.e., "Transfer 1") 261-1, the area 272 is composed of the first transfer step (i.e., "Transfer 1") 261-2, the second process (i.e., "Process 2") 221 to the fourth process (i.e., "Process 4") 224, the data output 231, and the second transfer step (i.e., "Transfer 2") 262-2, and the area 273 is composed of the third data input (i.e., "Data Input 3") 213, the fifth process (i.e., "Process 5") 225, and the second transfer step (i.e., "Transfer 2") 262-1.

[0087] The data flow division function 602 assigns an area ID to each of the divided areas 271 to 273. At this time, the data flow division function 602 converts the information of the processing execution location instructed in each transfer step 261 and 262 from the base name to an area ID. The area ID is composed of a combination of the base name and a sequential number within the base. For example, the first transfer step (i.e., "Transfer 1") 261-1 and 261-2 are assigned "From: A1" as area information indicating the transfer source of the intermediate data, and "To: A2" as area information indicating the transfer destination of the intermediate data. In addition, the second transfer step (i.e., "Transfer 2") 262-1 and 262-2 are assigned "From: B1" as area information indicating the transfer source of the intermediate data, and "To: A2" as area information indicating the transfer destination of the intermediate data.

[0088] Each dataflow processing device 105 holds the base name to which it belongs as an environment variable, for example. The sequence of determining the area in charge is the same as in the first embodiment. However, when selecting the desired area to be in charge, the area in charge adjustment function 605 selects one of the areas instructed to be executed at the base to which it belongs (i.e., in this embodiment, an area whose area ID includes the base name of the dataflow processing device 105). Furthermore, the management device 104 periodically transmits area responsibility information 608 in the base to which it belongs to the management device 104 of the other base, and shares information with each other. Furthermore, in response to a request for area information from the dataflow processing device 105 (i.e., step S1212 in FIG. 12), the management device 104 transmits the area responsibility information 608 of the base to which it belongs and the area responsibility information 502 obtained from the other management device 104 to the dataflow processing device 105 (i.e., step S1213 in FIG. 12).

[0089] As a result, each dataflow processing device 105 can obtain information on the dataflow processing device 105 in charge of each area of ​​a dataflow executed at two or more different bases 1602-1603, and can execute a dataflow across bases. In this case, when each dataflow processing device 105 is arranged at a plurality of different bases 1602-1603, if an execution base identifier (e.g., From: Base A) that uniquely identifies the execution base of the dataflow processing device 105 is assigned to the transfer steps 261 and 262 of the dataflow 1701, the dataflow division function 602 of each dataflow processing device 105 converts the execution base identifier into a division identifier (e.g., area ID=A1) that uniquely identifies each area, and assigns the converted division identifier to each area. The assigned area adjustment function 605 of each dataflow processing device 105 uniquely determines the dataflow processing device 105 that will be in charge of processing each area based on the division identifier assigned to each area.

[0090] According to this embodiment, when each data flow processing device 105 is arranged at a plurality of different bases 1602 to 1603, it is possible to execute a data flow across the bases. Also, according to this embodiment, it is possible to specify the execution base of each process of the data flow, and it is possible to execute anonymization processing for each input data of one data flow, convert each input data into transferable data, for example, convert into data from which items that cannot be transferred are deleted, and aggregate and process the converted data. Also, it is possible to specify parallel processing of the data flow even without knowing the resource configuration of each base, and high-speed processing of the data flow can be easily realized. By inserting a transfer step and specifying a base by the user, high-speed processing of the data flow is possible.

[0091] The management device 104 does not necessarily have to exist at each of the bases 1602-1603, and may be one in the information processing system 1601, or may exist at some of the bases (for example, the base 1602). The different bases 1602-1603 may be geographically different bases, or may be logically different bases, for example, bases that are the same geographically but indicate departments in a company. Although an example of two bases is shown in this embodiment, the number of bases may be more than two. EXAMPLES

[0092] In this embodiment, a method of specifying a processing execution base and a method of adjusting a coverage area in a case where bases have a hierarchical structure will be described.

[0093] FIG. 18 is a configuration diagram showing an example of the configuration of an information processing system according to a third embodiment. In FIG. 18, an information processing system 1801 includes a support device 103 and three or more upper hierarchical bases 1810, 1820, 1830, the base 1810 includes a plurality of lower hierarchical bases 1811, 1812, . . . the base 1820 includes a plurality of lower hierarchical bases 1821, 1822, . . . the base 1830 includes a plurality of lower hierarchical bases 1831, 1832, . . . the lower hierarchical bases 1811-1832 include a management device 104, one or more data flow processing devices 105, and one or more data storage devices 106. The user terminal 102, the support device 103, the upper hierarchical bases 1810, 1820, 1830, and the lower hierarchical bases 1811-1832 are connected to each other via a network 107. It is not necessary for each of the bases 1811 to 1832 to include at least one data storage device 106, but the data storage device 106 may be included in the higher-level bases 1810 to 1830 or the information processing system 1801 and may be accessible by each of the bases 1811 to 1832. The number of higher-level bases 1810 to 1830 may be more or less than three.

[0094] In this case, the location 1810 is managed with the location name "jp", the location 1820 is managed with the location name "us", and the location 1830 is managed with the location name "eu". Furthermore, the locations 1811, 1812, ... belonging to the location 1810 are managed with the location name "jp-i.jp", the locations 1821, 1822, ... belonging to the location 1820 are managed with the location name "us-i.us", and the locations 1831, 1832, ... belonging to the location 1830 are managed with the location name "eu-i.eu".

[0095] In the case where the bases have a hierarchical structure, when processing is to be executed at one of the bases 1811, 1812, ..., in the base 1810 in the upper layer, the user may specify the base name using a wildcard such as *.jp in the transfer steps 261 and 262 in Fig. 17 when creating the data flow, or may simply specify the base name as jp. For example, when specifying a base by *.jp, in the desired area selection step S1201 of the area determination sequence shown in Fig. 12, the data flow processing device 105 belonging to each base in jp selects an area to which it wishes to be in charge from among the area specified for execution at the base (jp-x.jp) to which it belongs and the area specified for execution at the base (*.jp) 1810 in the upper layer. Also, in the determination of the presence or absence of an area to which it is not yet assigned (step S1210 in Fig. 12), it is determined whether or not there is an area to which it is not yet assigned between the base (jp-x.jp) 1811 to which it belongs and the area specified for execution at the base (*.jp) 1810 in the upper layer.

[0096] There may be one or more management devices 104 in the information processing system 1801, and they may be present at each of the lower hierarchical bases 1811 to 1832, or at each of the upper hierarchical bases 1810, 1820, and 1830. The management devices 104 cooperate with each other to uniquely determine the data flow processing device 105 in charge of each area.

[0097] According to this embodiment, even in the case where the processing resource amount differs for each of the bases 1811 to 1832, the dataflow processing device 105 at each base can uniquely determine the dataflow processing device 105 in charge of each area and execute the dataflow without the user having to consider the resource configuration of each base. This makes it possible to efficiently utilize resources within the range in which data can be transferred according to the confidentiality of the data being handled by converting each input data into transferable data. For example, even if the bases 1811, 1812, ... include a mixture of those with weak resource amounts and those with large resource amounts, by specifying the base 1810 in the upper hierarchy, it becomes possible to execute processing suited to the resource amount in the dataflow processing device 105 at any of the bases 1811, 1812, ... belonging to the base 1810.

[0098] In this embodiment, an example in which there are two hierarchical levels is shown, but the number of hierarchical levels is not limited to two. The number of hierarchical levels may be different for each upper hierarchical level. The hierarchical levels may be geographical or logical (e.g., based on the departmental structure of a company). EXAMPLES

[0099] In this embodiment, an example will be shown in which in each data flow processing device 105, the execution of the data flow and the transfer of intermediate data between the data flow processing devices 105 are executed by different processes.

[0100] Fig. 19 is a configuration diagram showing an example of the configuration of a dataflow processing device according to Example 4. For ease of understanding, in Fig. 19, each function provided in the dataflow processing device 105 and data to be recorded are shown associated with a dataflow execution process 1901 that executes a dataflow and an intermediate data transfer process 1902 that transfers intermediate data.

[0101] In FIG. 19 , the data flow execution process 1901 includes a data flow receiving function 601, a data flow division function 602, a transfer step preprocessing function 1903, a data flow execution function 604, data flow information 606, and transfer step information 1906, and the intermediate data transfer process 1902 includes a responsible area adjustment function 605, a connection setting function 1904, a data transfer function 1905, area responsible information 608, and connection information 1907.

[0102] In this case, the transfer step pre-processing function 603 shown in the first embodiment is divided into a transfer step pre-processing function 1903 and a connection setting function 1904. Also, the transfer step information 607 is divided into transfer step information 1906 and connection information 1907. The data transfer function 1905 transmits and receives intermediate data between the data flow execution process 1901 and the intermediate data transfer process 1902, and transfers intermediate data between the data flow processing devices 105.

[0103] Fig. 20 is a diagram showing an example of the configuration of transfer step information according to Example 4. In Fig. 20, the transfer step information 1906 records a transfer step ID 2001, a transfer source area ID 2002, a transfer destination area ID 2003, a transfer step type 2004, a connection ID 2005 with the intermediate data transfer process, and a format 2006 of the intermediate data.

[0104] In this embodiment, when the data flow processing device 105 receives information on the data flow 201 and executes the process of establishing a communication connection shown in Fig. 13, in step S1302, the transfer step ID 2001, the source area ID 2002, the destination area ID 2003, and the transfer step type 2004 of the transfer step information 1906 are recorded with information similar to the information recorded in the transfer step ID 901, the source area ID 902, the destination area ID 904, and the transfer step type 906 of the transfer step information 607 shown in Fig. 9. In the connection ID 2005 with the intermediate data transfer process, for example, information "t01" is recorded.

[0105] Fig. 21 is a diagram showing an example of the configuration of connection information related to Example 4. In Fig. 21, the connection information 1907 records a transfer step ID 2101, a connection ID 2102 with a data flow execution process, a connection ID 2103 with a connection destination, a transfer source IP address 2104, a transfer destination IP address 2105, and a transfer step type 2016.

[0106] The connection ID 2103 with the connection destination is an identifier of the communication connection between the data flow processing device 105 of the source or destination of the intermediate data, and this communication connection is established by executing the process of establishing the communication connection shown in Fig. 13, as in the first embodiment. Note that the transfer step ID 2001, the transfer source IP address 2104, the transfer destination IP address 2105, and the transfer step type 2106 of the connection information 1907 record information similar to the information recorded in the transfer step ID 901, the transfer source IP address 903, the transfer destination IP address 905, and the transfer step type 906 of the transfer step information 607 shown in Fig. 9.

[0107] 22 is a sequence diagram showing an example of a communication connection setting sequence according to the fourth embodiment. In this embodiment, in a plurality of dataflow processing devices, dataflow execution and intermediate data transfer are performed in different processes. To achieve this, a communication connection is established between a dataflow processing device that executes a dataflow execution process 1901 and a dataflow processing device that executes an intermediate data transfer process 1902.

[0108] In FIG. 22, a transfer step pre-processing function (function separated from the transfer step pre-processing function 603) 1903 of a data flow processing device 105 executing a data flow execution process 1901 sends a connection establishment request including a transfer step ID, a source area ID, a destination area ID, and a transfer step type to an intermediate data transfer process 1902 of the same data flow processing device 105 (S2201), and establishes a connection between the data flow execution process 1901 and the intermediate data transfer process 1902 (S2202).

[0109] Next, the transfer step pre-processing function 1903 judges whether the connection has been successfully established (S2203), and if the connection has been successfully established (yes in S2203), it updates the transfer step information 1906 (S2204). Specifically, the transfer step pre-processing function 1903 records information such as "t01" in the connection ID 2005 with the intermediate data transfer process in the transfer step information 1906. If the connection has not been established (no in S2203), the transfer step pre-processing function 1903 waits for a predetermined time (S2205), and then again transmits a connection connection request to the intermediate data transfer process 1902 of the same data flow processing device 105 (S2201).

[0110] When the connection setting function 1904 establishes a connection (S2202), it registers the transfer step ID 2101, the connection ID 2102 with the data flow execution process 1901, the source IP address 2104, the destination IP address 2105, and the transfer step type 2106 in the connection information 1907 (S2206). The source IP address 2104 and the destination IP address 2105 are identified from the source area ID and the destination area ID included in the connection connection request, respectively, based on the area responsibility information 608.

[0111] Next, the connection setting function 1904 sets up a communication connection with the destination data flow processing device 105 (S2207). The communication connection is set up in the same manner as in the communication connection setting sequence shown in FIG. 14. However, in step S1411 in FIG. 14, the connection ID with the destination is set not in the transfer step information 1906 but in the connection ID 2103 with the destination of the connection information 1907, for example, "s01 " is recorded.

[0112] 23 is a sequence diagram showing an example of a transfer sequence of intermediate data according to the embodiment 4. The transfer of intermediate data and the notification of the data format of the intermediate data are performed in a transfer step when the data flow execution function 604 executes a data flow. In addition, these communications are performed using each communication connection established by the communication connection setting sequence.

[0113] In FIG. 23, when the data flow execution process 1901 of the data flow processing device 105 of the data transfer source receives information of a data flow 202, for example, as processing in the transfer step 241-1 of the data transfer source, when intermediate data is received from the previous step (S2301), it determines whether or not this is the first transfer (S2302), and if it is the first transfer (yes at S2302), it transmits information regarding the format of the intermediate data to the data flow processing device 105 of the data transfer destination via the intermediate data transfer process 1902 (S2303 to S2304).

[0114] The intermediate data transfer process 1902 in the data flow processing device 105 of the data transfer destination transfers information regarding the format of the received intermediate data to the data flow execution process 1901 in the data flow processing device 105 of the data transfer destination (S2305). The data flow execution process 1901 records the information regarding the data format of the intermediate data in the transfer step information 1906 (S2306).

[0115] Next, the data flow execution process 1901 of the data flow processing device 105 of the data transfer source converts the intermediate data into data in a format for transmission (S2307), for example, into serialized information, and transfers the converted data to the data flow processing device 105 of the transfer destination via the intermediate data transfer process 1902 of the data transfer source (S2308-S2309).

[0116] The intermediate data transfer process 1902 in the destination data flow processing device 105 transfers the received data in the transmission format to the data flow execution process 1901 in the destination data flow processing device 105 (S2310). The destination data flow execution process 1901 converts the received data in the transmission format into the intermediate data format (S2311) and transmits the data to the subsequent step (S2312).

[0117] Incidentally, processes other than the data flow transfer step are executed as usual by the data flow execution function 604 of each data flow processing device 105.

[0118] It is not always necessary to set up a communication connection, and connectionless communication may be used. Also, the timing of notification of the intermediate data format is not limited to that shown in this embodiment.

[0119] According to this embodiment, since the intermediate data transfer process 1902 is responsible for the transfer of intermediate data, the data flow execution process 1901 executes data transfer with the intermediate data transfer process 1902 in the transfer step, thereby realizing data transfer between the data flow processing devices 105. Also, according to this embodiment, as the processing of the data flow execution function 604, the processing of the data input section (i.e., "Data Input 1" to "Data Input 3") and the processing section (i.e., "Process 1" to "Process 4") of the data flow 202 can be shared by the data flow execution process (i.e., execution process) 1901, and the processing of the division and transfer section (i.e., "Transfer 1" to "Transfer 2") can be shared by the intermediate data transfer process (i.e., transfer process) 1902, thereby making it possible to reduce the load on the data flow execution function 604.

[0120] It is considered that data flows are executed in various languages ​​and applications. By separating the data flow processing function from the intermediate data transfer processing function, it becomes possible to execute one data flow in cooperation with multiple data flow processing devices 105 in various languages ​​and applications by adding data transfer processing with the intermediate data transfer process. Data transfer between the intermediate data transfer process 1902 and the data flow processing process 1901 can be realized by a general method such as WebSocket communication.

[0121] Also, by using container technology, it is possible to execute the data flow execution process 1901 and the intermediate data transfer process 1902 in separate containers. For example, in Kubernetes (registered trademark), one or more containers can be collectively handled as a unit called a pod, and the data flow processing device 105 can be realized by a pod including a container for data flow execution and a container for intermediate data transfer. EXAMPLES

[0122] In the first embodiment, the user inserts a transfer step into the data flow. In this embodiment, a method for recommending the position where the transfer step is to be inserted will be described.

[0123] A user designs a data flow using an information processing device that has a data flow design function with a GUI, a data flow execution function, and a division position recommendation function.

[0124] Fig. 24 is a configuration diagram showing an example in which a candidate for a transfer step insertion position is inserted into a data flow according to the fifth embodiment. In Fig. 24, a data flow 2401 is information generated by the user terminal 102 based on a user operation, and includes a first data input (i.e., "data input 1") 211, a second data input (i.e., "data input 2") 212, and a third data input (i.e., "data input 3") 213 as start points for inputting data, a first process (i.e., "process 1") 221 to a seventh process (i.e., "process 7") 227 as processes for performing statistical processing of data, anonymization processing of data, and the like, and includes a first data output (i.e., "data output 1") to a second data output (i.e., "data output 2") 232, 233 as end points for outputting data.

[0125] In this case, the user terminal 102, which is an information processing device having a data flow design function, a data flow execution function, and a division position recommendation function, inserts candidate locations 281 to 286, which are the junctions between processes and the branching points between processes and are candidates for inserting a transfer step, as transfer candidate parts in the process of generating information about the data flow 2401. Candidate location 281 is inserted between the first process (i.e., "process 1") 221 and the third process (i.e., "process 3") 223, candidate location 282 is inserted between the second process (i.e., "process 2") 222 and the third process (i.e., "process 3") 223, candidate location 283 is inserted between the third process (i.e., "process 3") 223 and the fourth process (i.e., "process 4") 224, candidate location 284 is inserted between the fifth process (i.e., "process 5") 225 and the fourth process (i.e., "process 4") 224, candidate location 285 is inserted between the fourth process (i.e., "process 4") 224 and the sixth process (i.e., "process 6") 226, and candidate location 286 is inserted between the fourth process (i.e., "process 4") 224 and the seventh process (i.e., "process 7") 227.

[0126] Fig. 25 is a flowchart showing an example of processing of the division position recommendation function according to the fifth embodiment. In Fig. 25, the data flow execution function of the user terminal 102 executes each processing (i.e., "Process 1" to "Process 7") recorded in the data flow 2401 using sample data (i.e., sample data corresponding to "Input 1" 211 to "Input 3" 213) generated based on a user's operation (S2501), and calculates the CPU processing time of each processing (S2502). After that, the division position recommendation function of the user terminal 102 extracts candidate positions 281 to 286 that are candidates for inserting a transfer step based on the information of the data flow 2401 (S2503).

[0127] Next, the division position recommendation function of the user terminal 102 creates combinations of insertion positions of transfer steps based on the number of areas specified in advance (S2504), and calculates the CPU processing time for each area for each combination (S2505). Specifically, the division position recommendation function of the user terminal 102 calculates the total value of the CPU processing time for each process included in each area.

[0128] Next, the division position recommendation function of the user terminal 102 calculates the variance of the CPU processing time for each area (S2506), selects the combination that minimizes the calculated variance, and presents the selected combination as the recommended insertion position for the transfer step (S2507).

[0129] The division position recommendation function of the user terminal 102 may, in step S2502, calculate the data transfer volume at the candidate locations 281-286 for inserting a transfer step in addition to the CPU processing time, and, in step S2507, present a combination that minimizes the weighted sum of the variance of the CPU processing time for each area and the total value of the data transfer volume.

[0130] In addition, one or more thresholds for the CPU processing time for each area and an upper limit on the number of areas may be set in advance, and the processing of steps S2504 to S2507 may be repeated while increasing the number of areas in the combination selected in step S2507 until the average or maximum value of the CPU processing time for each area falls below the set threshold or the number of areas reaches the upper limit, and the finally selected combination of transfer step insertion positions may be presented to the user.

[0131] In addition, in presenting the recommended positions in the transfer step (step S2507), the division position recommendation function may present one or more top combinations so that the user can select one.

[0132] In the creation of the combination of the insertion positions of the transfer steps in step S2504, the insertion of a transfer step may be prohibited in some or all of the input lines to the joining position of one process (e.g., "Process 3"). Similarly, the insertion of a transfer step may be prohibited in some or all of the output lines from one process.

[0133] In this embodiment, each dataflow processing device 105 may be configured to have a division position recommendation function. In this case, each dataflow processing device 105 further has a division position recommendation function that calculates the CPU processing times of multiple processing units (i.e., "process 1" to "process 7") based on sample data corresponding to the dataflow 2401. This division position recommendation function extracts each of a plurality of transfer candidate parts (i.e., candidate locations 281 to 286) belonging to the data flow 2401, creates a plurality of combinations of one or more transfer candidate parts based on each extracted transfer candidate part and the number of regions set as the number by which the data flow 2401 is divided into a plurality of regions, calculates a CPU processing time for each region obtained by dividing the data flow 2401 based on the transfer candidate parts belonging to each created set, selects from each combination of transfer candidate parts a combination for which the calculated CPU processing time is equal to or less than a set value, for example a combination for which the variance of the calculated CPU processing time is minimum, updates the data flow 2401 with each transfer candidate part belonging to the selected set as a transfer part (i.e., a transfer step) that transfers data between processing units, and outputs the updated data flow 2401 to the data flow division function 602. The data flow division function 602 divides the range recorded in the updated data flow 2401 into multiple areas based on the transfer parts, and divides the transfer parts belonging to the updated data flow 2401 into multiple divided transfer parts, and allocates each of the divided divided transfer parts to each area.

[0134] As a result, even if the data flow 2401 does not have a transfer step, each data flow processing device 105 can update the data flow 2401 with each transfer candidate part selected from among multiple transfer candidate parts (i.e., candidate locations 281-286) present in the data flow 2401 as a transfer part (i.e., transfer step) that transfers data between processing units, divide the updated data flow 2401 into multiple regions based on the transfer part, divide the transfer part into multiple divided transfer parts, and allocate each divided divided transfer part to each region. As a result, each data flow processing device 105 can automatically divide the data flow 2401 into multiple regions if the data flow 2401 has multiple transfer candidate parts (i.e., candidate locations 281-286) even if the user does not specify the position of the transfer step.

[0135] Fig. 26 is a configuration diagram showing an example in which a transfer step insertion position is presented in a data flow according to the embodiment 5. In Fig. 26, a data flow 2601 is information generated by the user terminal 102 based on a user operation, and is displayed on the screen of the output device 305 by the division position recommendation function of the user terminal 102. This data flow 2601 has a first data input (i.e., "data input 1") 211, a second data input (i.e., "data input 2") 212, and a third data input (i.e., "data input 3") 213 as starting points for inputting data, a first process (i.e., "process 1") 221 to a seventh process (i.e., "process 7") 227 as processes for performing statistical processing of data and anonymization processing of data, a first data output (i.e., "data output 1") 232 to a second data output (i.e., "data output 2") 233 as end points for outputting data, and a first transfer step (i.e., "transfer 1") 291 to a third transfer step (i.e., "transfer 3") 293 as transfer steps that are switching points of the data flow processing device.

[0136] In this case, the first transfer step (i.e., "Transfer 1") 291 is inserted as a transfer step (i.e., a transfer section) into the candidate location 281 shown in the data flow 2401 of FIG. 24, the second transfer step (i.e., "Transfer 2") 292 is inserted as a transfer step (i.e., a transfer section) into the candidate location 284 shown in the data flow 2401 of FIG. 24, and the third transfer step (i.e., "Transfer 3") 293 is inserted as a transfer step (i.e., a transfer section) into the candidate location 285 shown in the data flow 2401 of FIG. 24.

[0137] The division position recommendation function of the user terminal 102 may display the CPU processing time and output data amount of each process when displaying the transfer step insertion position.

[0138] According to this embodiment, it is possible to display the recommended position for inserting a transfer step in the data flow. Also, it is possible to select a transfer step insertion position that meets the conditions (for example, the CPU load of each area is equalized) from one or more combinations of the transfer step insertion positions.

[0139] Although the present invention has been specifically described based on the embodiment, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the gist of the present invention. For example, the above embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, other configurations can be added, deleted, or replaced with part of the configuration of the above embodiment.

[0140] In addition, the control lines and information lines in each diagram are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines in the implementation. For example, it may be considered that almost all components are actually connected to each other.

[0141] Furthermore, each of the above-described embodiments may be implemented alone, or may be implemented in partial or complete combination.

[0142] In the above description, the components (e.g., each function, database, and element step) are not necessarily essential unless otherwise specified or considered to be clearly essential. Some components (e.g., the support device 103) may also have the functions of other components (e.g., the management device 104).

[0143] The data recorded by each information processing device may include more or fewer items than those shown in the embodiment. The number of components (e.g., user terminal 102, support device 103, management device 104, data flow processing device 105, data storage device 106, etc.) is not limited to the number shown in the embodiment, and may be more or less. [Explanation of symbols]

[0144] 101: Information processing system, 102: User terminal, 103: Support device, 104: Management device, 105: Data flow processing device, 106: Data storage device, 201: Data flow

Claims

1. A device comprising one or more information processing devices for processing information, The information processing device includes: A division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition; a determination function for uniquely determining which of the information processing devices is to be assigned to each of the pieces of processing information divided by the division function; an execution function for executing a process in the information processing device determined by the determination function, The processing information is a plurality of processing units disposed between the respective start points and the respective end points, sequentially processing data from at least one of the respective start points; and one or more transfer units disposed between any of the plurality of processing units, transferring data between the processing units; The division function is When dividing the processing information, a range in which the processing information is recorded is divided into a plurality of regions based on the transfer section, and the transfer section is divided into a plurality of divided transfer sections, and each of the divided divided transfer sections is allocated to each of the regions, and one of the plurality of divided transfer sections is connected to the processing section of the transfer source which belongs to the same region as the one divided transfer section, and the other of the plurality of divided transfer sections is connected to the processing section of the transfer destination which belongs to the same region as the other divided transfer section, One of the decision functions belonging to the information processing device is An information processing system characterized by sequentially selecting one information processing device from the information processing devices, sequentially assigning the selected one information processing device to be in charge of each of the areas, correlating each of the areas with the one information processing device and recording the recorded area assignment information, and distributing the recorded area assignment information to each of the information processing devices.

2. A device comprising: one or more information processing devices for processing information; The information processing device includes: A division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition; a determination function for uniquely determining which of the information processing devices is to be assigned to each of the pieces of processing information divided by the division function; an execution function for executing a process in the information processing device determined by the determination function, The processing information is a plurality of processing units disposed between the respective start points and the respective end points, sequentially processing data from at least one of the respective start points; and one or more transfer units disposed between any of the plurality of processing units, transferring data between the processing units; The division function is When dividing the processing information, a range in which the processing information is recorded is divided into a plurality of regions based on the transfer section, and the transfer section is divided into a plurality of divided transfer sections, and each of the divided divided transfer sections is allocated to each of the regions, and one of the plurality of divided transfer sections is connected to the processing section of the transfer source which belongs to the same region as the one divided transfer section, and the other of the plurality of divided transfer sections is connected to the processing section of the transfer destination which belongs to the same region as the other divided transfer section, The information processing device further includes a management device that transmits and receives information to and from the information processing device, The management device includes: a management device area assignment information for recording a division identifier for uniquely identifying each of the areas and a device identifier for uniquely identifying each of the information processing devices in charge of each of the areas in association with each other; The decision function belonging to each of the information processing devices is transmitting a registration request to the management device for allocating a specified area among the areas to the information processing device to which the determination function belongs; The management device includes: When the registration request is received, the area assignment information for the management device is referenced based on the received registration request, and, on condition that the assignment of the specified area is not registered based on the result of the reference, the division identifier of the specified area is associated with the device identifier of the requester, and registered in the area assignment information for the management device, and a response indicating that the registration as the person in charge of the specified area has been made to the determination function is sent, and if the person in charge of the specified area has already been registered, a response indicating that the registration as the person in charge of the specified area is not possible is sent to the determination function. The determination function that has received the response registers the specified area as a responsible area of ​​the information processing device to which the determination function belongs, on condition that the response indicates that the designated area has been registered as the responsible area of ​​the information processing device to which the determination function belongs; An information processing system characterized in that a person in charge of the specified area is uniquely determined.

3. A device comprising: one or more information processing devices for processing information; The information processing device includes: A division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition; a determination function for uniquely determining which of the information processing devices is to be assigned to each of the pieces of processing information divided by the division function; an execution function for executing a process in the information processing device determined by the determination function, The processing information is a plurality of processing units disposed between the respective start points and the respective end points, sequentially processing data from at least one of the respective start points; and one or more transfer units disposed between any of the plurality of processing units, transferring data between the processing units; The division function is When dividing the processing information, a range in which the processing information is recorded is divided into a plurality of regions based on the transfer section, and the transfer section is divided into a plurality of divided transfer sections, and each of the divided divided transfer sections is allocated to each of the regions, and one of the plurality of divided transfer sections is connected to the processing section of the transfer source which belongs to the same region as the one divided transfer section, and the other of the plurality of divided transfer sections is connected to the processing section of the transfer destination which belongs to the same region as the other divided transfer section, The information processing devices are disposed at different locations, The division function is When an execution location identifier that uniquely identifies an execution location of the information processing device is assigned to the transfer section in the processing information, the execution location identifier is converted into a division identifier that uniquely identifies each of the areas, and the converted division identifier is assigned to each of the areas; The decision function is an information processing system, characterized in that the information processing device in charge of each of the areas is uniquely determined based on the division identifier assigned to each of the areas.

4. A device comprising: one or more information processing devices for processing information; The information processing device includes: A division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition; a determination function for uniquely determining which of the information processing devices is to be assigned to each of the pieces of processing information divided by the division function; an execution function for executing a process in the information processing device determined by the determination function, The processing information is a plurality of processing units disposed between the respective start points and the respective end points, sequentially processing data from at least one of the respective start points; and one or more transfer units disposed between any of the plurality of processing units, transferring data between the processing units; The division function is When dividing the processing information, a range in which the processing information is recorded is divided into a plurality of regions based on the transfer section, and the transfer section is divided into a plurality of divided transfer sections, and each of the divided divided transfer sections is allocated to each of the regions, and one of the plurality of divided transfer sections is connected to the processing section of the transfer source which belongs to the same region as the one divided transfer section, and the other of the plurality of divided transfer sections is connected to the processing section of the transfer destination which belongs to the same region as the other divided transfer section, The information processing device includes: a communication function for controlling communication between the information processing devices determined by the determination function; The division function is assigning a division identifier to each of the regions so as to uniquely identify each of the regions; The communication function includes: a first division identifier that uniquely identifies the area to which the one of the division and transfer units belongs is stored in association with the division and transfer unit as first communication information indicating the area of ​​the transfer source, a first device identifier that uniquely identifies the information processing device responsible for processing the area to which the one of the division and transfer units belongs is stored in association with the division and transfer unit as second communication information indicating the information processing device of the transfer source, a second division identifier that uniquely identifies the area to which the other of the division and transfer units belongs is stored in association with the division and transfer unit as third communication information indicating the area of ​​the transfer destination, a second device identifier that uniquely identifies the information processing device responsible for processing the area to which the other of the division and transfer units belongs is stored in association with the division and transfer unit as fourth communication information indicating the information processing device of the transfer destination, and a type of communication between the information processing devices is stored in association with the division and transfer unit as fifth communication information specifying transmission or reception; An information processing system characterized in that the execution function establishes a communication connection between the information processing devices by referring to the first communication information to the fifth communication information as a process before executing the processing of the division and transfer unit.

5. A device comprising: one or more information processing devices for processing information; The information processing device includes: A division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition; a determination function for uniquely determining which of the information processing devices is to be assigned to each of the pieces of processing information divided by the division function; an execution function for executing a process in the information processing device determined by the determination function, The processing information is a plurality of processing units disposed between the respective start points and the respective end points, sequentially processing data from at least one of the respective start points; and one or more transfer units disposed between any of the plurality of processing units, transferring data between the processing units; The division function is When dividing the processing information, a range in which the processing information is recorded is divided into a plurality of regions based on the transfer section, and the transfer section is divided into a plurality of divided transfer sections, and each of the divided divided transfer sections is allocated to each of the regions, and one of the plurality of divided transfer sections is connected to the processing section of the transfer source which belongs to the same region as the one divided transfer section, and the other of the plurality of divided transfer sections is connected to the processing section of the transfer destination which belongs to the same region as the other divided transfer section, Each of the regions is The start points, the processing unit, and the division and transfer unit are included, The executive function is When each of the information processing devices determined by the determination function is responsible for processing each of the start points and the processing units as processing for each of the regions, the data is input from the start point and the input data is processed; when each of the information processing devices determined by the determination function is responsible for processing the division and transfer units as processing for each of the regions, data transfer is performed between the information processing devices determined by the determination function; The information processing device includes: An information processing system comprising: an execution process that shares the processing of the starting point and the processing unit as processing of the execution function; and a transfer process that shares the processing of the division and transfer unit as processing of the execution function.

6. A device comprising: one or more information processing devices for processing information; The information processing device includes: A division function for dividing processing information indicating a data processing procedure from a plurality of starting points to one or more end points into a plurality of pieces using parallel processing by the information processing device as a division condition; a determination function for uniquely determining which of the information processing devices is to be assigned to each of the pieces of processing information divided by the division function; an execution function for executing a process in the information processing device determined by the determination function, The processing information is a plurality of processing units arranged between the start points and the end points, sequentially processing data from at least one of the start points; and a plurality of transfer candidate units arranged between any of the plurality of processing units, The information processing device includes: a division position recommendation function for calculating at least one of a CPU processing time and a data output amount of the plurality of processing units based on sample data corresponding to the processing information; The division position recommendation function includes: extracting each of the multiple transfer candidate parts belonging to the sample data, creating one or more combinations of one or more transfer candidate parts from each of the extracted transfer candidate parts, dividing the processing information into a plurality of areas based on the transfer candidate parts belonging to the combination of transfer candidate parts for each combination of transfer candidate parts, calculating the total value of the CPU processing time for each set of areas created, selecting a combination of transfer candidate parts from among the combinations of transfer candidate parts based on at least one of the calculated total value of the CPU processing time and the amount of data transfer by the transfer candidate parts, updating the processing information with each of the transfer candidate parts belonging to the selected combination of transfer candidate parts as a transfer part that transfers data between the processing units, and outputting the updated processing information to the division function; The division function is An information processing system characterized in that the range in which the updated processing information is recorded is divided into a plurality of areas based on the transfer section, and the transfer section is divided into a plurality of divided transfer sections, and each of the divided divided transfer sections is allocated to each of the areas.

Images