Data flow control device, data flow control method, and data flow control program

Abstract

A data flow control device (10) calculates a third data flow among data flows in which some of a first data flow currently in operation in a system that processes data and an inputted second data flow are commonized and integrated, the third data flow being such that the use amount of resources when operated in the system satisfies a prescribed condition. The data flow control device (10) indicates to the system that the first data flow be switched to the third data flow.

Description

[Technical field]

[0001] The present invention relates to a data flow control device, a data flow control method, and a data flow control program. [Background technology]

[0002] Conventionally, a method called "Kafon" is known that divides functions related to data transfer into a plurality of logical layers and manages them based on metadata of a workflow that performs data processing (see, for example, Non-Patent Document 1).

[0003] The technique described in Non-Patent Document 1 manages each function of the system by dividing it into four layers: application logic, transfer route management, transmission between nodes, and middleware implementation, and aims to improve the efficiency of data flow by standardizing transfer processing, etc. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Cyber ​​Agent, "Large-scale data processing", https: / / www.cyberagent.co.jp / techinfo / labo / tech / detail / id=23934 [Non-Patent Document 2] Ichiro Fukuda et al., "CyberAgent's R&D System and Initiatives for Utilizing Data," Japanese Society for Artificial Intelligence, Vol. 30, No. 3, pp. 318-324. (https: / / jsai.ixsq.nii.ac.jp / ej / ?action=repository_action_common_download&item_id=1892&item_no=1&attribute_id=22&file_no=1) Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional methods have a problem that it may be difficult to improve the efficiency of data flow in the entire system. For example, the method described in Non-Patent Document 1 does not take into account the capabilities of the computing units to which tasks are assigned or data transfer between the computing units, and does not necessarily improve the performance of the entire system. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the data flow control device is characterized by having a calculation unit that calculates a third data flow, which is a data flow obtained by commonizing and integrating at least a portion of a first data flow running in a data processing system and an input second data flow, and which calculates a third data flow whose resource usage when run in the system satisfies a predetermined condition, and an instruction unit that instructs the system to switch from the first data flow to the third data flow. Effect of the Invention

[0007] According to the present invention, data flow can be made more efficient throughout the entire system. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a data flow control device according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a data flow. [Diagram 3] FIG. 3 is a diagram illustrating switching of data flows. [Figure 4] FIG. 4 is a flowchart showing a process flow of the data flow control device according to the first embodiment. [Diagram 5] FIG. 5 illustrates an example of a computer that executes a data flow control program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of a data flow control device, a data flow control method, and a data flow control program according to the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0010] [Configuration of the first embodiment] First, the configuration of the data flow control device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the data flow control device according to the first embodiment.

[0011] 1, a dataflow control device 10 accepts input of a dataflow and outputs an instruction to change the dataflow to a data processing system 20. A user inputs dataflow definition information for adding, updating, deleting, etc. a dataflow to the dataflow control device 10. The dataflow control device 10 has an interface unit 11, a storage unit 12, and a control unit 13. The data processing system 20 is an example of a system that processes data.

[0012] The interface unit 11 is an interface for inputting and outputting data and communicating data. For example, the interface unit 11 accepts data input from an input device such as a keyboard and a mouse. Also, for example, the interface unit 11 outputs data to an output device such as a display and a speaker. Also, for example, the interface unit 11 may be a NIC (Network Interface Card).

[0013] The storage unit 12 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk. The storage unit 12 may be a data-rewritable semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 12 stores an OS (Operating System) and various programs executed by the data flow control device 10. The storage unit 12 also stores a flow information management DB 121, a computational resource allocation management DB 122, and a line information DB 123.

[0014] The flow information management DB 121 holds information about data flows running in the data processing system 20. The flow information management DB 121 holds the data acquisition source, processing method, processing order, etc. of each process included in the data flow. In addition, the flow information management DB 121 holds various metadata related to the attributes of the data collected by the data processing system 20 and the users of the data flows.

[0015] The computational resource allocation management DB 122 holds information on each computational resource, such as executable processes and whether or not the computational resource is allocated to each process. Furthermore, the computational resource allocation management DB 122 holds information on whether or not the computational resource is permitted to be allocated to each user, i.e., information on access control.

[0016] The line information DB 123 holds information on delays and bandwidths of networks between servers, between users and servers, between data collection sources and servers, and between data centers. Note that the servers here are, for example, multiple servers that make up the data processing system 20.

[0017] The control unit 13 controls the entire data flow control device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 13 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. The control unit 13 also functions as various processing units by the operation of various programs. For example, the control unit 13 has a confirmation unit 131, an extraction unit 132, a calculation unit 133, an instruction unit 134, and an update unit 135.

[0018] The confirmation unit 131 refers to the flow information management DB 121 and confirms whether the attribute information of the user corresponding to the input data flow permits sharing of the data flow or not. Furthermore, the confirmation unit 131 confirms whether there is an overlap between the input data flow and a data flow in operation.

[0019] For example, when the flow information management DB 121 indicates that resources such as computing units and network resources are being used by a data flow owned by a user with priority use authority, the resource is occupied by the user. Therefore, the confirmation unit 131 determines that integration of the data flow owned by the user with priority use authority and the input data flow is not permitted.

[0020] Here, the data flow will be described with reference to Fig. 2. Fig. 2 is a schematic diagram showing an example of a data flow. The existing data flow in Fig. 2 is a data flow currently in operation in the data processing system 20. The additional data flow is a data flow input to the data flow control device 10 by a user.

[0021] As shown in Fig. 2, the existing data flow and the additional data flow overlap in the portion "Camera → Process P1." Therefore, in the example of Fig. 2, the confirmation unit 131 confirms that there is an overlapping portion between the input data flow and the running data flow.

[0022] The extraction unit 132 extracts sharable data flows from among the existing data flows. For example, the extraction unit 132 extracts data flows that are permitted to be shared according to user attribute information and that have overlapping portions.

[0023] The calculation unit 133 calculates a new data flow from among the data flows obtained by commonizing and integrating at least a portion of the data flow extracted by the extraction unit 132 and the input data flow, such that the resource usage when operated in the data processing system 20 satisfies a predetermined condition.

[0024] Here, the data flow extracted by the extraction unit 132 is an example of a first data flow running in the system that processes data. The input data flow is an example of a second data flow. The new data flow is an example of a third data flow.

[0025] First, the calculation unit 133 creates a combination of available computing resources and network resources capable of realizing each data flow for each data flow extracted by the extraction unit 132. At this time, the calculation unit 133 can create the combination by referring to the computational resource allocation management DB 122 and the line information DB 123.

[0026] Furthermore, the calculation unit 133 calculates the resource usage, i.e., the cost, for each combination created, and selects, for example, the combination with the smallest cost. The calculation unit 133 calculates the cost based on the data transfer delay time between the computation resources of each combination, between the data provider and the computation resource, and between the data output destination and the computation resource, and the network NW bandwidth information.

[0027] The computational resources are, for example, computational devices of information processing devices constituting the data processing system 20. Data sources are, for example, cameras and sensors. Data output destinations are, for example, other information processing devices and output devices such as displays.

[0028] As described above, the confirmation unit 131 confirms whether or not sharing of the running data flows is permitted based on attribute information preset for the user of the running data flows. Then, when the confirmation unit 131 confirms that sharing of the running data flows is permitted, the calculation unit 133 calculates the shared data flow.

[0029] The instruction unit 134 instructs the system to switch the data flow extracted by the extraction unit 132 to the new data flow calculated by the calculation unit 133.

[0030] Fig. 3 is a diagram for explaining switching of data flows As shown in Fig. 3, in the data flows before switching, when a user U1 executes an application AP1, a process P1 is executed by a computing unit a.

[0031] Here, it is assumed that execution of application AP1 by user U2 and a data flow related to execution of application AP1 by user U2 are added. The data flow before the switching and the two added data flows overlap in the part where process P1 is executed. Therefore, the calculation unit 133 calculates a data flow of "Camera → process P1 (operating unit b) → user U1 AP1" which is an integration of the data flow before the switching and the added data flow.

[0032] Then, when the calculation unit 133 selects the data flow "Camera → Processing P1 (Calculator b) → User U1 AP1", the instruction unit 134 instructs the data processing system 20 to switch the existing data flow "Camera → Processing P1 (Calculator a) → User U1 AP1" to the selected data flow "Camera → Processing P1 (Calculator b) → User U1 AP1".

[0033] Furthermore, the computing unit that executes the process P1 changes before and after the switch. For example, the computing unit b stores the execution result of the process P1 in a queue provided between the process P2 and the process P3.

[0034] In this way, the calculation unit 133 can calculate an integrated data flow in which the result of process P1, which is common to the existing data flow and the input data flow, is stored in a queue, and processes P2 and P3, which use the result of process P1 in the input data flow, acquire data from the queue. In other words, the calculator c and the calculator d acquire the execution result of process P1 from the queue in the process of executing processes P2 and P3, respectively.

[0035] In this case, the data processing system 20 can obtain the execution result from the queue when executing the processes P2 and P3, without executing the process P1. The execution result of the process P1 is, for example, a list of data obtained by a search.

[0036] In addition, if the calculation unit 133 determines that the overall cost will increase as a result of changing the execution entity of the process P1 from the calculator a to the calculator b, it can decide not to select the switching of the data flow as shown in FIG. 3.

[0037] The update unit 135 updates the information in each DB in accordance with the switching between the old and new data flows. Here, the data flow before the switching is called the old data flow, and the data flow after the switching is called the new data flow.

[0038] First, before switching between the old and new data flows, the update unit 135 secures computation resources for the new data flow and updates the computational resource allocation management DB 122. Then, after switching between the old and new data flows, the update unit 135 adds information about the new data flow to the flow information management DB 121 and updates information about the old data flow.

[0039] Then, the update unit 135 releases the computing resources used in the old data flow and updates the computing resource allocation management DB 122. Furthermore, the update unit 135 updates the flow information management DB 121 of the old data flow.

[0040] [Processing of the first embodiment] 4 is a flowchart showing the flow of processing of the data flow control device according to the first embodiment. First, the data flow control device 10 accepts input of data flow information (step S101).

[0041] Next, the data flow control device 10 checks whether the data flow is to be shared based on the user information (step S102). If the data flow is not to be shared (step S103, No), the data flow control device 10 proceeds to step S107. On the other hand, if the data flow is to be shared (step S103, Yes), the data flow control device 10 proceeds to step S104.

[0042] Next, the dataflow control device 10 judges whether the input dataflow has an overlapping portion with an existing dataflow (step S104). If the input dataflow has no overlapping portion with an existing dataflow (step S104, No), the dataflow control device 10 proceeds to step S107. On the other hand, if the input dataflow has an overlapping portion with an existing dataflow (step S104, Yes), the dataflow control device 10 proceeds to step S105.

[0043] The data flow control device 10 extracts existing data flows that can be shared (step S105). Then, the data flow control device 10 calculates a data flow that is an integration of the input data flow and the extracted data flow (step S106).

[0044] The data flow control device 10 creates a combination of available computing resources and NW (network) resources that can realize the calculated data flow (step S107). Then, the data flow control device 10 selects a combination based on the cost from input to output (step S108).

[0045] Here, the data flow controller 10 secures the computation resources and updates the computation resource allocation management DB 122 (step S109).Then, the data flow controller 10 instructs the data processing system 20 to switch between the old and new data flows (step S110).

[0046] The dataflow control device 10 also adds information about the new dataflow to the flow information management DB 121 and updates information about the old dataflow (step S111). Furthermore, the dataflow control device 10 releases the computation resources used by the old dataflow and updates the computation resource allocation management DB 122 (step S112). Then, the dataflow control device 10 updates the flow information management DB 121 for the old dataflow (step S113).

[0047] [Advantages of the First Embodiment] As described above, the data flow control device 10 calculates a third data flow that satisfies a predetermined condition in terms of resource usage when operated in the system, among data flows that are obtained by sharing a part of the first data flow running in the data processing system and the input second data flow. The data flow control device 10 instructs the system to switch the first data flow to the third data flow. In this way, the data flow control device 10 not only shares a part of the new and old data flows, but also switches them in consideration of the resource status of the system. As a result, according to this embodiment, the data flow can be made more efficient in the entire system. Furthermore, according to this embodiment, it is possible to replace pluggable processing of data flows.

[0048] The data flow control device 10 stores the result of a first process common to the first and second data flows in a queue, and calculates a third data flow so that a second process in the second data flow, which uses the result of the first process, retrieves data from the queue. As a result, according to this embodiment, the processing efficiency of the entire system can be improved.

[0049] The data flow control device 10 checks whether sharing of the first data flow is permitted based on attribute information preset for the user of the first data flow. When the confirmation unit 131 confirms that sharing of the first data flow is permitted, the data flow control device 10 calculates the third data flow. There are multi-tenant systems in which data is output to multiple users. In such systems, sharing of data flows may not be permitted depending on the attributes of the users. In this embodiment, the efficiency of processing of the entire system can be improved by taking multi-tenancy into consideration.

[0050] [System configuration, etc.] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0051] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0052] [program] As an embodiment, the data flow control device 10 can be implemented by installing a data flow control program that executes the above data flow control process as package software or online software on a desired computer. For example, by executing the above data flow control program on an information processing device, the information processing device can function as the data flow control device 10. The information processing device here includes desktop and notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and even slate terminals such as PDA (Personal Digital Assistant), etc.

[0053] The data flow control device 10 can also be implemented as a data flow control server device that provides a service related to the above-mentioned data flow control processing to a client, the client being a terminal device used by a user. For example, the data flow control server device is implemented as a server device that provides a data flow control service in which a data flow is input and an instruction to change the data flow is output. In this case, the data flow control server device may be implemented as a Web server, or may be implemented as a cloud that provides a service related to the above-mentioned data flow control processing by outsourcing.

[0054] 5 is a diagram showing an example of a computer that executes a data flow control program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0055] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0056] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the data flow control device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the data flow control device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0057] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0058] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0059] 10 Data flow control device 11 Interface section 12 Storage section 13 Control section 20 Data Processing System 121 Flow Information Management DB 122 Computing Resource Allocation Management DB 123 Line Information DB 131 Verification Department 132 Extraction part 133 Calculation section 134 Instruction section 135 Update Department

Claims

1. a confirmation unit that confirms whether or not sharing of a first data flow is permitted based on attribute information that is preset for a user of the first data flow that is running in the data processing system; a calculation unit that stores a result of a first process common to the first data flow and an input second data flow in a queue, and calculates a third data flow in which a second process that uses a result of the first process in the second data flow acquires data from the queue when it is confirmed by the confirmation unit that sharing of the first data flow is permitted; an instruction unit that instructs the system to switch from the first data flow to the third data flow; A data flow control device comprising:

2. A data flow control method executed by a data flow control device, comprising: a confirmation step of confirming whether or not sharing of a first data flow is permitted based on attribute information preset for a user of the first data flow running in the data processing system; a calculation step of storing a result of a first process common to the first data flow and an input second data flow in a queue, and calculating a third data flow in which a second process in the second data flow, which uses the result of the first process, obtains data from the queue, when it is confirmed in the confirmation step that sharing of the first data flow is permitted; an instruction step of instructing the system to switch from the first data flow to the third data flow; 13. A data flow control method comprising:

3. A data flow control program for causing a computer to function as the data flow control device according to claim 1.

Images