Data processing method and data processing system

Abstract

To reduce man hours required to construct a pipeline and to suppress error occurrences that occur in manual work by reproducing a part or the whole of a past created pipeline in an arbitrary environment to be able to quickly and simply construct the pipeline suitable for a new environment.SOLUTION: Information of defining a configuration of a past created pipeline is acquired in advance and stored in a configuration management part 31. When an engineer inputs a pipeline name of a copy source and environment information of a copy destination, a deployment script 32 of a horizontal deployment tool 30 creates a new pipeline in an environment of the copy destination on the basis of information of defining the pipeline of the copy destination replicated by the input and the environment information of the copy destination.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a data processing method and a data processing system for constructing a pipeline.

Background Art

[0002] Conventionally, there is known a data utilization platform that supports easily constructing a pipeline for collecting, managing, storing, aggregating, etc. data and outputting data suitable for visualization. Data utilization platforms for constructing pipelines are currently released from a plurality of companies including major cloud vendors.

[0003] A pipeline constructed by a data utilization platform is a series of processes in which a plurality of data processing components are combined according to needs and arranged so that a plurality of data processes are sequentially executed. When data is supplied, it is configured to execute a series of processes defined as a pipeline to output desired data.

[0004] Pipelines are applicable to data processing in various fields. For example, an analysis pipeline constructed for a company that provides services to customers is configured to collect, format, aggregate, etc. the services provided to customers and the data held by customers, and output data suitable for visualization, thereby enabling support for solving known problems and discovering new values.

[0005] When building a pipeline on a data utilization platform, a pipeline can be created by combining pre-prepared resources (template components, source code execution functions, databases, etc.) using a dedicated GUI (Graphical User Interface) tool. For example, Patent Document 1 below describes a system that automatically generates analytical pipelines for various execution engines based on analytical solution module metadata and user customization. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-201936 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, if the data handled in a particular customer's environment (the source environment) differs even slightly from that of another customer (the destination environment), or if customization is required to meet the needs of another customer, it is not possible to simply apply a pre-existing pipeline provided to one customer to another customer.

[0008] Therefore, when replicating pipelines in existing data utilization platforms, engineers need to manually replicate the pipelines using a GUI. However, this process is time-consuming and prone to errors due to manual work.

[0009] The present invention aims to provide a data processing method and a data processing system that enable the rapid and easy construction of new pipelines using previously created pipelines. [Means for solving the problem]

[0010] The data processing method according to the present invention is a data processing method performed by a computer that, in order to achieve the above objective, creates a new second pipeline using a first pipeline that has already been created, A storage step for storing information that defines the configuration of the first pipeline, An input reception step that receives identification information for the first pipeline and environmental information for the replication destination of the second pipeline, The invention is characterized by a creation step of creating the second pipeline based on information defining the configuration of the first pipeline identified by the identification information and environmental information for the replication destination of the second pipeline.

[0011] According to the method described above, a first pipeline created in the past can be replicated to any environment, allowing for the rapid and easy construction of a second pipeline adapted to the new environment. This reduces the development effort and time required for pipeline replication. Furthermore, engineers only need to input the identification information of the source first pipeline and the environment information of the target environment, significantly reducing the effort required to create the pipeline and suppressing errors that may occur when replicating pipelines manually.

[0012] The data processing method according to the present invention may, in the storage step, acquire configuration information for each resource of the first pipeline and connection information between each resource, and store them as a configuration information definition file that defines the configuration of the first pipeline.

[0013] According to the method described above, the configuration information of each resource and the connection information between each resource, which can be easily extracted from the first pipeline, can be used as information to define the configuration of the first pipeline created in the past.

[0014] The data processing method according to the present invention may include a configuration information definition file in which the configuration information and connection information for each component constituting the first pipeline are described on a component-by-component basis.

[0015] According to the method described above, the first pipeline can be replicated as individual components of the second pipeline.

[0016] In the data processing method according to the present invention, the creation of the second pipeline may be performed in the creation step by executing a plurality of deployment scripts that have been prepared in advance for the purpose of replicating the pipeline.

[0017] According to the method described above, by executing multiple pre-created deployment scripts as needed, the creation of the second pipeline can be carried out in stages, as if assembling a pipeline in a new environment.

[0018] The data processing method according to the present invention may be created by combining commands used in an existing data utilization platform in the deployment script.

[0019] According to the method described above, various deployment scripts can be created using existing data utilization platforms.

[0020] In the data processing method according to the present invention, in the creation step, a database resource for storing data in the second pipeline may be created based on the configuration information of each resource in the first pipeline, and then a resource for performing data processing in the second pipeline may be created and connected to the database resource.

[0021] According to the above method, the creation of the second pipeline can be divided into the stage of creating the resources of the database and the stage of creating the resources for data processing and connecting them to the resources of the database. By checking for errors at each of the above stages, the resources can be deployed while gradually testing whether they are compatible with the environment of the replication destination.

[0022] The data processing method according to the present invention may have a correction step of correcting the second pipeline after creation.

[0023] According to the above method, the newly created second pipeline can be corrected to conform to the environment of the replication destination, etc.

[0024] The data processing method according to the present invention may obtain the configuration information of each resource of the second pipeline corrected in the correction step and the connection information between each resource, and store it as a configuration information definition file for defining the configuration of the second pipeline.

[0025] According to the above method, the corrected second pipeline can be used as the pipeline of the replication source.

[0026] The data processing method according to the present invention may, in the creation step, create only the specific parts of the second pipeline based on the information defining the configuration of the specific parts included in the first pipeline.

[0027] According to the above method, only the specific parts of the first pipeline created in the past can be diverted to quickly and easily construct a second pipeline compatible with the new environment.

[0028] Furthermore, in order to achieve the above objective, the data processing system according to the present invention is a data processing system that creates a new second pipeline using a first pipeline that has already been created, and is composed of a remote terminal that can be operated by an engineer and a server device located on a network that can be accessed by the remote terminal. The aforementioned server device A storage unit for storing information defining the configuration of the first pipeline, An input receiving unit that receives identification information of the first pipeline and environment information of the replication destination of the second pipeline from the remote terminal via the network, The system is characterized by having a creation unit that creates the second pipeline based on information defining the configuration of the first pipeline identified by the identification information and environmental information for the replication destination of the second pipeline.

[0029] With the above configuration, the first pipeline created in the past can be replicated to any environment, and a second pipeline adapted to the new environment can be quickly and easily constructed, reducing the development effort and time required for pipeline replication. Furthermore, engineers only need to access the server device from a remote terminal and input the identification information of the source first pipeline and the environment information of the replication destination, significantly reducing the effort required to create the pipeline and suppressing the occurrence of errors that may occur when replicating pipelines manually. [Effects of the Invention]

[0030] According to the present invention, it becomes possible to quickly and easily construct a pipeline adapted to a new environment by replicating part or all of a previously created pipeline in any environment, thereby reducing the man-hours required to construct the pipeline and suppressing errors that occur in manual work. [Brief explanation of the drawing]

[0031] [Figure 1]This figure shows an example of a pipeline handled in an embodiment of the present invention. [Figure 2] This figure shows an example of the configuration of an analysis pipeline for collecting, formatting, and aggregating user behavior logs of a chatbot, as part of an embodiment of the present invention. [Figure 3] This figure shows an overview of pipeline lateral deployment using the lateral deployment tool in an embodiment of the present invention. [Figure 4] This figure shows the preliminary preparations before replicating the pipeline using the lateral expansion tool in an embodiment of the present invention. [Figure 5] This flowchart shows an example of the process for acquiring configuration information and connection information related to the preliminary preparations before replicating a pipeline using the lateral deployment tool in an embodiment of the present invention. [Figure 6] This figure shows an example of a configuration information definition file obtained from a pre-created pipeline in an embodiment of the present invention. [Figure 7] This figure shows the process of creating a new pipeline using the lateral expansion tool in an embodiment of the present invention. [Figure 8] This flowchart shows an example of the processing of a deployment script for a horizontal expansion tool in an embodiment of the present invention. [Figure 9] This figure illustrates a process for modifying a replicated pipeline in an embodiment of the present invention. [Figure 10] This figure illustrates an embodiment of the present invention that illustrates a process of replicating only a portion of a pipeline created in the past. [Figure 11] This figure shows an example of the configuration of a data processing system in an embodiment of the present invention. [Figure 12] This figure shows an example of the configuration of a server device in an embodiment of the present invention. [Modes for carrying out the invention]

[0032] The data processing method and data processing system according to embodiments of the present invention will be described below with reference to the drawings.

[0033] (Pipeline) First, an example of a pipeline handled in an embodiment of the present invention will be described with reference to Figure 1. Figure 1 is a diagram showing an example of a pipeline handled in an embodiment of the present invention.

[0034] The pipelines handled in the embodiments of the present invention are composed of a combination of multiple data processing components (hereinafter sometimes referred to as "components") as appropriate to meet the needs. The pipelines can be built using cloud-based data utilization platforms such as Microsoft Azure®, Amazon Web Services®, and Google Cloud Platform®.

[0035] Figure 1 illustrates, as an example, pipeline 20A provided to customer A, who is receiving service A, and pipeline 20B provided to customer B, who is receiving service B.

[0036] In customer A's environment (customer environment) 10A, data related to service A is stored. The pipeline 20A provided to customer A is an analytical pipeline for data analysis, consisting of a collection pipeline 21A that performs processing to collect the data stored in customer environment 10A, a formatting pipeline 22A that formats the data collected by collection pipeline 21A, and an aggregation pipeline 23A that aggregates the data formatted by formatting pipeline 22A.

[0037] In pipeline 20A for customer A, the collection pipeline 21A retrieves data accumulated in the customer environment 10A via an API (Application Programming Interface), then the formatting pipeline 22A formats the data, and the aggregation pipeline 23A aggregates the data, outputting data including the aggregation results. The data output by pipeline 20A is visualized and output as a visualization report such as graphs and tables, enabling various analyses related to service A.

[0038] Furthermore, the pipeline can be customized to incorporate necessary components according to customer needs. For example, in Figure 1, the pipeline 20B provided to customer B is an analysis pipeline consisting only of a collection pipeline 21B that performs the process of collecting data accumulated in the customer environment 10B.

[0039] In pipeline 20B for customer B, anonymized data accumulated in customer A's environment by collection pipeline 21B is retrieved via API, and the retrieved data is output. The data output by pipeline 20B is visualized, similar to pipeline 20A for customer A, and output as a visualization report such as graphs and tables, allowing for various analyses related to service B.

[0040] A pipeline is constructed by setting the configuration information of each resource that makes up the pipeline, as well as the connection information between each resource. For example, pipeline 20B for customer B in Figure 1 consists only of the collection pipeline 21B. By setting a component that collects data via an API (API call) as a template component, and setting storage to store the collected data as object storage, pipeline 20B including the collection pipeline 21B can be created. Furthermore, by setting parameters in the pipeline that match the customer environment and the desired output data, data processing is performed in a way that is tailored to each customer's environment.

[0041] Referring to Figure 2, a specific example of a pipeline in an embodiment of the present invention will be described. Figure 2 is a diagram showing an example of the configuration of an analysis pipeline in an embodiment of the present invention that collects, formats, and aggregates the behavior logs of chatbot users.

[0042] Figure 2 illustrates a concrete example where Customer A, as shown in Figure 1 above, is a company that provides a chatbot service (Service A), and shows the pipeline 20A provided to this company.

[0043] In customer environment 10A, user activity logs of the chatbot are stored as accumulated data. This accumulated data consists of the user activity logs stored as they are, and for example, specific items are stored as a large file in CSV (Comma Separated Values) format.

[0044] Pipeline 20A is designed to collect, format, and aggregate behavioral logs that are stored as unorganized data in the customer environment 10A, and to output data suitable for visualization.

[0045] Pipeline 20A is created by combining pre-prepared resources, and is composed of components arranged in the order of collection pipeline 21A, formatting pipeline 22A, and aggregation pipeline 23A. In addition, pipeline 20A is configured with the ID (identification information) of the application to be logged, the API access key, and the period of logs to be collected, which are necessary information for calling the API when reading the stored data accumulated in the customer environment 10A. Furthermore, the storage locations for the collected data collected by collection pipeline 21A, the formatted data created by formatting pipeline 22A, and the aggregated data created by aggregation pipeline 23A are also configured.

[0046] The collection pipeline 21A is configured to collect logs for a set period from the accumulated data (chatbot user activity logs) stored in the customer environment 10A via an API. The collected data collected by the collection pipeline 21A is stored in the configured data storage location.

[0047] The formatting pipeline 22A is configured to retrieve the collected data from the collection pipeline 21A and format it into a format that is easy to aggregate. For example, the formatting pipeline 22A is configured to format the user behavior logs of a chatbot on a conversational basis. The formatted data is data that has been formatted to represent how a user behaved in a single conversation with the chatbot, and the formatted data includes information such as the flow of the conversation, whether a conversion (CV) was achieved, and the number of times the user re-entered information. The formatted data created by the formatting pipeline 22A is stored in the configured formatted data storage location.

[0048] The aggregation pipeline 23A is configured to retrieve the formatted data collected by the formatting pipeline 22A and to aggregate various information obtained from the formatted data. For example, the aggregation pipeline 23A aggregates the formatted data created from the chatbot user's behavior log at regular intervals. The aggregated data includes the aggregated results of the chatbot user's behavior log and contains various information such as object data, conversion rate / exit rate / stop rate, average conversion rate / exit time, number of executions per conversation, number of conversations, number of unique users, and conversation transition summaries. The aggregated data created by the aggregation pipeline 23A is stored in the configured storage location for aggregated data.

[0049] In pipeline 20A, the collected data, formatted data, and aggregated data stored in each storage location can be read and used from each storage location as needed. In particular, the aggregated data, which includes the aggregated results of chatbot user behavior logs, organizes various information that could not be easily grasped from the accumulated data simply stored in customer environment 10A. By applying visualization processing to the aggregated data and creating reports that summarize various information in graphs and tables, it becomes possible to make visible statistical values ​​of chatbot user behavior, issues inherent in the chatbot service, and directions for improving the efficiency of the service, thereby supporting the discovery of areas for improvement of the service (issues and solutions to solve those issues) and new service methods.

[0050] (Challenges in pipeline replication) The pipeline is built to suit the specific customer environment. For example, Figure 2 shows a pipeline 20A that is adapted to customer environment 10A, which provides a chatbot service.

[0051] When a pipeline created for a specific customer is to be rolled out to another customer, if it is possible to simply replicate the pipeline created for that customer as a pipeline for the other customer, the time and cost required to create a pipeline for the other customer can be reduced.

[0052] However, there is a problem in that if the data handled in the source environment and the data handled in the destination environment differ even slightly, or if the pipeline needs to be customized to meet the requirements of another customer, a pre-existing pipeline created for a specific customer (the source pipeline) cannot be used as is for another customer. For this reason, when replicating pipelines in existing data utilization platforms, engineers need to manually verify the pipeline using a GUI. However, manual replication is time-consuming and prone to errors due to being a human process. Therefore, there has been a need for a new method to quickly and easily build new pipelines using previously created pipelines.

[0053] This invention was made in view of the above-mentioned situation, and by introducing the horizontal deployment tool described below, pipelines can be replicated quickly and easily. This reduces the development effort and time required for pipelines, and also reduces the occurrence of errors caused by human work.

[0054] (Overview of the horizontal expansion tool) The lateral expansion tool in an embodiment of the present invention will be described below with reference to Figure 3. Figure 3 is a diagram showing an overview of pipeline lateral expansion using the lateral expansion tool in an embodiment of the present invention.

[0055] Figure 3 schematically illustrates the process of creating a pipeline 20X for another customer by duplicating a pipeline 20A previously created based on a specific customer environment 10A to suit a different customer environment 10X, using the horizontal expansion tool 30. In this embodiment of the present invention, instead of simply duplicating the created pipeline 20A to provide to another customer environment 10X, the horizontal expansion tool 30 is used to create a pipeline 20X that is suitable for a different customer environment 10X based on the created pipeline 20A.

[0056] The lateral deployment tool 30 is generally composed of a configuration manager 31 and a deployment script 32. The configuration manager 31 stores configuration information and connection information for pipelines 20A (in Figure 3, an analysis pipeline consisting of a collection pipeline 21A, a shaping pipeline 22A, and an aggregation pipeline 23A) that have been created in the past. The engineer inputs environment information of the replication destination (lateral deployment destination) into the lateral deployment tool 30 and executes the deployment script 32. The environment information here refers to the type and format of data to be handled, data processing in each component of the pipeline, and parameters referenced during data processing, and includes information customized to the environment of the replication destination and the requirements of the replication destination customer.

[0057] The deployment script is executed in the horizontal deployment tool 30, which replicates the previously created pipeline 20A to a new customer environment 10X, enabling the rapid and easy creation of a pipeline 20X (in Figure 3, an analysis pipeline consisting of a collection pipeline 21X, a shaping pipeline 22X, and an aggregation pipeline 23X) adapted to the new customer environment 10X. While there are no particular limitations on inputting environment information for the replication target and executing the deployment script 32 to the horizontal deployment tool 30, it is preferable to use, for example, a CI (Continuous Integration) tool.

[0058] (Preparation for the horizontal expansion tool) Configuration and connection information for the source pipeline is stored in the configuration management 31 before the lateral deployment tool 30 is executed. The following describes the preparations made before the lateral deployment tool 30 replicates the pipeline, with reference to Figures 4 and 5. Figure 4 is a diagram showing the preparations made before the lateral deployment tool replicates the pipeline in an embodiment of the present invention.

[0059] Here, it is assumed that a pipeline 20A (the source pipeline) to be provided to a specific customer A has been created in advance using a dedicated GUI tool of the data utilization platform. The source pipeline 20A is, for example, an analysis pipeline created to suit service A provided by a specific customer A, and consists of three components: a data collection pipeline 21A, a data shaping pipeline 22A, and an aggregation pipeline 23A.

[0060] When creating a new pipeline using an existing pipeline, it is necessary to retrieve the configuration and connection information from the existing pipeline and store it in advance in the configuration management 31 of the horizontal deployment tool 30. For example, by operating a dedicated GUI tool of the data utilization platform, engineers can easily retrieve the configuration and connection information of various pipelines created in the past, including pipeline 20A shown in Figure 4, and store it in the configuration management 31 of the horizontal deployment tool 30.

[0061] Referring to Figure 5, the process for acquiring configuration information and connection information of a created pipeline will be explained. Figure 5 is a flowchart showing an example of the process for acquiring configuration information and connection information related to the preliminary preparations before duplicating a pipeline using the lateral deployment tool in an embodiment of the present invention.

[0062] First, the engineer inputs the identification information (e.g., pipeline name) of the pipeline from which they want to obtain configuration and connection information, for example, using a dedicated GUI tool on the data utilization platform. The data utilization platform identifies the pipeline from the entered pipeline identification information (step S11).

[0063] Next, for the pipeline identified in step S11, the configuration information of each resource constituting the pipeline and the connection information between each resource are obtained (step S12). Then, the configuration information of each resource and the connection information between each resource obtained in step S12 are output as a configuration information definition file, and the configuration information definition file is stored in the storage location (configuration management database) of the configuration management 31 (step S13). The processing in steps S11 to S13 above corresponds to the storage step of the present invention.

[0064] The following describes an example of a configuration information definition file. Figure 6 shows an example of a configuration information definition file obtained from a pre-created pipeline in an embodiment of the present invention.

[0065] The configuration information definition file is a file that contains information about the resources of a created pipeline (configuration information for each resource and connection information between each resource). Figure 6 illustrates, as an example, a configuration information definition file obtained from pipeline 20A (pipeline A), which was developed to suit service A provided by a specific customer A.

[0066] The configuration information definition file shown in Figure 6 contains information about each resource of pipeline A, which is composed of the collection pipeline, shaping pipeline, and aggregation pipeline components, described on a component-by-component basis.

[0067] Specifically, the configuration information definition file describes resource A-1, which is a template component for API calls, and resource A-2, which is the object storage where the collected data created by the collection pipeline is stored, as information about the resources of the collection pipeline. Resource A-1 describes information such as the endpoint and the storage location of the data to be acquired.

[0068] Furthermore, the configuration information definition file includes information about the data formatting pipeline resources, specifically Resource A-3, which is a template component for data formatting; Resource A-4, which is source code 1 that defines the processing procedure of the data formatting pipeline; and Resource A-5, which is object storage where the formatted data created by the data formatting pipeline is stored.

[0069] Furthermore, the configuration information definition file includes information about the resources of the aggregation pipeline, specifically resource A-6, which is a template component for data aggregation; resource A-7, which is source code 2 that defines the processing procedure of the aggregation pipeline; and resource A-8, which is the database (NoSQL) where the formatted data created by the formatting pipeline is stored.

[0070] Thus, the configuration information definition file describes information about the resources that make up the pipeline. In this embodiment, it is possible to obtain resource information from various pipelines created in the past, and multiple configuration information definition files related to various pipelines can be stored in the configuration management 31 of the horizontal deployment tool 30.

[0071] Furthermore, various pipelines created in the past have parameters set for each resource that are referenced during pipeline processing, along with information about the resources that make up the pipeline. In this embodiment, the setting parameters for each resource can also be stored in the configuration management 31 of the horizontal deployment tool 30. The setting parameters for each resource may be obtained using a dedicated GUI tool of the data utilization platform and stored in the configuration management 31 of the horizontal deployment tool 30, similar to the information about the resources that make up the pipeline, or they may be stored in the configuration management 31 of the horizontal deployment tool 30 manually by an engineer, for example.

[0072] (Processing by the deployment script) Next, the process of creating a new pipeline using the lateral expansion tool will be described with reference to Figures 7 and 8. Figure 7 is a diagram showing the process of creating a new pipeline using the lateral expansion tool in an embodiment of the present invention.

[0073] In this embodiment, when creating a new pipeline, a configuration information definition file for a previously created pipeline, which is stored in the configuration management 31 of the horizontal expansion tool 30, is used as a template.

[0074] As described above, the lateral deployment tool 30 is generally composed of a configuration management system 31 and a deployment script 32.

[0075] Configuration management 31 is a set of information managed and stored in a database format (configuration management database) in a predetermined data storage device. Configuration management 31 stores pipeline management definition information (pipeline configuration definition) 31C, along with pipeline management definition information (pipeline configuration definition) 31A for managing pipelines stored in configuration management 31, and deployment authentication information 31B for obtaining access rights to the data utilization platform, as well as pipeline configuration definition information (pipeline configuration definition).

[0076] The pipeline configuration definition information 31C contains information that defines the configuration of the pipeline, and includes, for example, configuration information definition files obtained from each pipeline created in the past, and the setting parameters for each pipeline. As an example, Figure 7 illustrates the configuration information definition file (pipeline A configuration definition) 31D and the setting parameters for pipeline A (pipeline A parameter sheet) 31E obtained from pipeline A, which was created in the past, and the configuration information definition file (pipeline B configuration definition) 31F and the setting parameters for pipeline B (pipeline B parameter sheet) 31G obtained from pipeline B, which was created in the past.

[0077] Note that Figure 7 shows that information on two pipelines, A and B, created in the past, is stored in the configuration management system 31. However, there is no particular limit to the number of pipelines that can be stored in the configuration management system 31.

[0078] Deployment script 32 is a program that performs the processing necessary to expand the pipeline laterally.

[0079] For example, existing data utilization platforms provide multiple commands that allow you to deploy each resource based on configuration information. By combining the various commands available in the data utilization platform, you can develop various deployment scripts 32 that can be used when expanding the pipeline horizontally.

[0080] Figure 7 illustrates an example of a deployment script 32 used in this embodiment, including a deployment script 32A for overall management of the pipeline, a deployment script 32B for setting parameters, a deployment script 32C for creating pipeline storage, a deployment script 32D for creating authentication information, a deployment script 32E for creating each component of the pipeline, a deployment script 32F for creating a database to store data processed by the pipeline, a deployment script 32G for grouping resources, a deployment script 32H for setting processing scripts in the pipeline, and a deployment script 32I for deleting all resources.

[0081] In this embodiment, the pipeline can be quickly and easily deployed by creating these deployment scripts 32 in advance. Note that the deployment scripts 32 are not limited to those described above; it is possible to create a desired deployment script 32 by appropriately combining various commands available on the data utilization platform.

[0082] When replicating a pipeline to a target environment using the lateral deployment tool 30, it is preferable to sequentially build and test the pipeline using, for example, a CI tool. For example, it is preferable to store a deployment script 32 in a CI server repository or the like that performs various processes such as creating the entire pipeline and each component within the pipeline (collection pipeline 21A, shaping pipeline 22A, aggregation pipeline 23A, etc.), creating and connecting resources for each pipeline, deleting resources, and setting pipeline parameters. Based on the configuration information definition file of the source pipeline and the target environment information entered by the engineer, it is preferable to enable the deployment script 32 to appropriately execute processes related to pipeline creation. The processing unit realized by the execution of the deployment script 32 constitutes the creation unit of the present invention.

[0083] Furthermore, in the pipeline creation process, it is preferable to check for errors at each stage of resource creation and connection, and to delete all resources being created and already created if an error occurs. By replicating the pipeline in this way, stage by stage, it is possible to ensure reliable pipeline replication and, as a result, reduce the man-hours and effort required to build the pipeline.

[0084] When an engineer uses the horizontal deployment tool 30 to duplicate a pipeline, they input identification information (e.g., pipeline name) to identify the source pipeline. This identifies the source pipeline. Furthermore, by inputting additional parameters that are environment information for the destination and appropriately issuing instructions to execute the deployment script 32, a new pipeline is created in the destination environment.

[0085] The following describes an example of processing by the deployment script 32 of the horizontal expansion tool 30. Figure 8 is a flowchart showing an example of processing by the deployment script of the horizontal expansion tool in an embodiment of the present invention. Note that the processing by the deployment script in Figure 8 is merely an example, and the present invention is not limited thereto.

[0086] The engineer inputs the source pipeline name and destination environment information into the lateral deployment tool 30 (step S21). Upon receiving the source pipeline name and destination environment information, the lateral deployment tool 30 executes a predetermined deployment script 32 (or a deployment script 32 instructed by the engineer). The process in step S21 corresponds to the input acceptance step of the present invention.

[0087] Upon execution of the deployment script 32, the lateral deployment tool 30 obtains the configuration information definition file of the source pipeline, identified by the source pipeline name, from the configuration management 31 (step S22), and adds the destination environment to the obtained destination pipeline configuration information definition file based on the input destination environment information (step S23). Next, the lateral deployment tool 30 creates database resources to store input data, output data, and intermediate data based on the configuration information definition file with the destination environment added (step S24). These processes are executed, for example, by a deployment script (for parameter input) 32B for setting parameters, a deployment script (for storage creation) 32C for creating pipeline storage, a deployment script (for pipeline creation) 32E for creating each component of the pipeline, and a deployment script (for database creation) 32F for creating a database to store the data processed by the pipeline.

[0088] If an error occurs during resource creation in step S24 (YES in step S25), the lateral deployment tool 30 deletes all resources being created and already created (step S31) and returns an error as output (step S32). The error is displayed, for example, on a display visible to the engineer. The process of deleting resources is performed, for example, by a deployment script 32I for deleting all resources (for deleting all resources).

[0089] On the other hand, if no error occurred during resource creation in step S24 (NO in step S25), the lateral deployment tool 30 creates a resource for data processing based on the configuration information definition file with the destination environment added, and connects it to the database resource created in step S24 (step S26). These processes are executed, for example, by a deployment script (for pipeline creation) 32E to create each component of the pipeline, and a deployment script (for processing script input) 32H to set processing scripts in the pipeline.

[0090] If an error occurs during resource creation or connection in step S26 (YES in step S27), all resources being created and already created are deleted (step S31), and an error is returned as output (step S32). The error is displayed, for example, on a display visible to the engineer. The process of deleting resources is performed, for example, by a deployment script 32I for deleting all resources.

[0091] On the other hand, if no errors occur during resource creation and connection in step S26 (NO in step S27), the pipeline is replicated to the destination environment, and the pipeline replication process is completed (step S28). The pipeline created in the destination environment is based on the configuration information definition file of the source pipeline and is adapted to the destination environment. The processes in steps S21 to S28 described above correspond to the creation steps of the present invention.

[0092] In the flowchart shown in Figure 8, by inputting the identification information of the source pipeline (for example, the name of the source pipeline) and the environment information of the destination, the deployment script 32 can easily create a pipeline that is suitable for the destination environment by using the configuration information definition file of the source pipeline that is pre-stored in the configuration management 31.

[0093] Furthermore, in the flowchart shown in Figure 8, after creating a database resource to store input data, output data, and intermediate data (step S24), a resource for data processing is created and connected to the database resource created in step S24 (step S26). By dividing the process into creating a database resource and creating a resource for data processing and connecting to the database resource, and performing the new pipeline creation process in this order, it is possible to reliably create a new pipeline that is compatible with the target environment while testing at each stage.

[0094] (Modifying the duplicated pipeline) Figure 9 is a diagram illustrating the process of modifying a replicated pipeline in an embodiment of the present invention. Figure 9 schematically illustrates the process of modifying a pipeline 20X for customer environment 10X, which has been replicated using pipeline 20A provided to customer environment 10A.

[0095] As shown in Figure 9, the replicated pipeline 20X can be further modified, for example, by customizing a portion of it (the shaded collection pipeline in the example in Figure 9) on the data utilization platform to meet the requirements of customer X.

[0096] Furthermore, as shown in Figure 9, the configuration information of the modified pipeline 20X may be obtained as a configuration information definition file and stored in the configuration management 31. This allows the pipeline to be recreated using the configuration information definition file of the modified pipeline 20X. Alternatively, a new deployment script 32 tailored to the modified pipeline 20X may be created.

[0097] (Partial replication of the pipeline) Figure 10 is a diagram illustrating a process for replicating only a portion of a previously created pipeline in an embodiment of the present invention. The pipeline 20Y shown in Figure 10 is an analysis pipeline that includes the components of a collection pipeline 21Y, a shaping pipeline 22Y, and an aggregation pipeline 23Y. Figure 10 schematically illustrates a process for partially replicating only a portion of pipeline 20A, which is provided to customer environment 10A, into pipeline 20Y for customer environment 10Y.

[0098] In this embodiment, the configuration information definition file obtained from a pre-created pipeline contains configuration information for each resource constituting the pipeline, as well as connection information between each resource, described on a component basis. As a result, as shown in Figure 10, it is possible to extract components from a pre-created pipeline and reuse them as part of another pipeline, such as reusing only the components of the collection pipeline 21A of pipeline 20A in the collection pipeline 21Y of pipeline 20Y (shaded in Figure 10).

[0099] (Example of a data processing system configuration) The configuration of the data processing system in an embodiment of the present invention will be described below with reference to Figures 11 and 12. Figure 11 is a diagram showing an example of the configuration of the data processing system in an embodiment of the present invention.

[0100] As shown in Figure 11, in one embodiment of the present invention, the data processing system 100 is configured such that a remote terminal 200 that can be operated by an engineer and a server device 300 that stores various data related to the pipeline and performs processing related to the pipeline can communicate with each other via a network 400 such as the Internet.

[0101] The remote terminal 200 is a general-purpose computer such as a PC (personal computer) that can be operated by an engineer, and is configured to access the server device 300 and use the various services provided by the server device 300. Although only one remote terminal 200 is shown in Figure 11, multiple remote terminals 200, each operable by multiple engineers, may be deployed.

[0102] The server device 300 is equipped with functions related to the present invention and is configured to execute processing related to each function in response to operation input from a remote terminal 200 accessed via the network 400. The server device 300 is equipped with functions such as providing a data utilization platform and providing a CI tool, and is configured to be able to execute the horizontal deployment tool according to the present invention, which is composed of a combination of these functions.

[0103] For the sake of simplicity, Figure 11 shows only one server device 300, and this single server device 300 has functions related to the present invention implemented on it. However, it is also possible that multiple server devices 300, each equipped with various functions such as a data utilization platform, a CI tool, a deployment tool, and a database management function for storing data, are distributed and arranged in a coordinated manner.

[0104] Figure 12 shows an example of the configuration of a server device in an embodiment of the present invention. The server device according to the present invention is not limited to the configuration shown in Figure 12.

[0105] As shown in Figure 12, the server device 300, for example, includes a processor 311, memory 312, communication unit 313, operation input unit 314, monitor 315, and storage 316, with each component connected via a bus 310.

[0106] The processor 311 has functions for performing data calculations and processing control. The processor 311 may consist of, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) that performs data processing specialized for a specific purpose.

[0107] Memory 312 is volatile memory that temporarily stores programs executed by the server device 300 and data to be processed. Memory 312 is a main memory such as RAM (Random Access Memory).

[0108] The communication unit 313 has a function for the server device 300 to communicate with other devices. As shown in Figure 11, the server device 300 is connected to the network 400 and is configured to execute processing related to each function in response to operation input from a remote terminal 200 accessed via the network 400. The communication unit 313 constitutes the input receiving unit of the present invention.

[0109] The operation input unit 314 has the function of receiving information input from the user and represents input devices such as a mouse and keyboard. The monitor 315 has the function of outputting visual information to the user and is equipped with a display screen. The server device 300 is basically configured to process based on operation input from the remote terminal 200, and the operation input unit 314 and monitor 315 are used as appropriate, for example, when an operator on the server device 300 side performs maintenance.

[0110] Storage 316 is an auxiliary storage device that can be implemented using, for example, magnetic disks such as HDDs (Hard Disk Drives), semiconductor memory such as SSDs (Solid State Drives), magneto-optical disks, and optical disks.

[0111] The storage 316 may be located in a network storage device or database management device separate from the server device 300, for example. In this embodiment, the storage 316 encompasses any data storage device accessible by the server device 300.

[0112] The storage 316 is capable of storing programs and data in which desired processing procedures are described as program instructions. For example, as shown in Figure 12, the storage 316 stores various programs such as the data utilization platform execution program 316A which realizes the data utilization platform, the CI tool execution program 316B which realizes the CI tool, and the lateral expansion tool execution program 316C which realizes the lateral expansion tool 30. The processor 311 reads the various programs from the storage 316 as appropriate, loads them into the memory 312, and executes program instructions, thereby enabling the server device 300 to realize the functions corresponding to each program.

[0113] Although Figure 12 schematically illustrates the various programs according to the present invention as independently existing programs, these programs may be configured to run in cooperation with each other. This allows the various functions realized by the execution of the various programs to work in coordination with each other. Furthermore, the various programs may be configured to run on separate server devices 300, and as described above, functions such as the function that provides a data utilization platform, the function that provides CI tools, the function that provides the lateral deployment tool 30, and the database management function that stores data may be distributed across different server devices 300.

[0114] Furthermore, the storage 316 stores multiple deployment scripts 32 that are executed when the pipeline is replicated. As described above, the deployment scripts 32 can be created by combining various commands available on the data utilization platform, for example. The deployment scripts 32 are stored in a repository managed by a CI tool, which is implemented by executing the CI tool execution program 316B, for example, and the pipeline can be built in the replication destination environment using the functions of the CI tool. The deployment scripts 32 are not particularly limited, but examples include the deployment scripts 32A to 32I shown in Figure 7. The processing unit implemented in the server device 300 by the execution of the deployment scripts 32 constitutes the creation unit of the present invention.

[0115] Furthermore, the storage 316 is capable of storing various types of data according to the present invention. The storage 316 functions as a configuration management database that stores the configuration management 31, for example, as shown in Figure 12. The configuration management 31 stores various types of information 31A to 31G, as shown in Figure 7. The storage 316 and the configuration management constitute the storage unit of the present invention.

[0116] Storage 316 may store various programs and data other than those mentioned above. For example, storage 316 may store various programs such as an operating system program necessary for the operation of the server device 300, and a server program that manages access from the remote terminal 200.

[0117] In the data processing system 100 shown in Figures 11 and 12, an engineer can access and operate the server device 300 from a remote terminal 200 to extract configuration information for each resource and connection information between each resource from the created pipeline, and store it in the configuration management 31 as a configuration information definition for the created pipeline.

[0118] Furthermore, engineers can access and operate the server device 300 from a remote terminal 200, for example, by combining various commands available on the data utilization platform to create a deployment script 32 necessary for replicating the pipeline.

[0119] In the data processing system 100 shown in Figures 11 and 12, an engineer can access and operate the server device 300 from a remote terminal 200. By inputting identification information of the source pipeline (e.g., pipeline name) and destination environment information into the lateral deployment tool 30 implemented in the server device 300, the system can use the existing pipeline (source pipeline) to create a new pipeline adapted to the destination environment through the processing of the deployment script 32 created for pipeline replication.

[0120] In the data processing system 100 shown in Figures 11 and 12, engineers can access and operate the server device 300 from a remote terminal 200 to modify parts of the pipeline created at the replication destination, extract configuration information for each resource and connection information between each resource from the modified pipeline, and store it in the configuration management 31 as a configuration information definition for the created pipeline.

[0121] In the data processing system 100 shown in Figures 11 and 12, an engineer can access and operate the server device 300 from a remote terminal 200, allowing only a portion of a pre-created pipeline to be reused in another pipeline.

[0122] Thus, in this embodiment, it is possible to quickly and easily build a pipeline adapted to a new environment by replicating part or all of a previously created pipeline to any environment. This reduces the development effort and time required for pipeline replication. Furthermore, engineers only need to input the identification information of the source pipeline and the environment information of the destination, significantly reducing the effort required to create a pipeline and suppressing the occurrence of errors that may occur when replicating pipelines manually.

[0123] Furthermore, in this embodiment, by executing the deployment script 32 created as the horizontal deployment tool 30, the database resources are placed first, followed by the placement of data processing resources and their connection to the database resources. This allows for the deployment of resources while performing step-by-step tests to ensure compatibility with the target environment. This ensures the creation of a pipeline that functions correctly in the new environment. Additionally, by executing multiple pre-created deployment scripts as needed, the creation of a pipeline compatible with the target environment can be performed step-by-step, much like assembling a pipeline in a new environment.

[0124] The operation of embodiments of the present invention will be described.

[0125] The data processing method in this embodiment involves creating a new second pipeline (destination pipeline) using the already created first pipeline (source pipeline).

[0126] The data processing method in this embodiment includes a storage step for storing information defining the configuration of the first pipeline, an input reception step for receiving input of identification information for the first pipeline and environmental information for providing the second pipeline, and a creation step for creating the second pipeline based on the information defining the configuration of the first pipeline identified by the identification information and the environmental information for the environment in which the second pipeline is created. The storage step corresponds to the processing of steps S11 to S13 shown in Figure 5. The input reception step corresponds to the processing of step S21 shown in Figure 8, and the creation step corresponds to the processing of steps S22 to S28 shown in Figure 8.

[0127] This allows for the rapid and easy creation of a second pipeline adapted to the new environment by replicating a previously created first pipeline to any environment, thereby reducing the development effort and time required for pipeline replication. Furthermore, engineers only need to input the identification information of the source first pipeline and the environment information of the destination, significantly reducing the effort required to create the pipeline and suppressing the occurrence of errors that can occur when replicating pipelines manually.

[0128] Furthermore, in the storage step described above, configuration information for each resource in the first pipeline and connection information between each resource may be obtained and stored as a configuration information definition file that defines the configuration of the first pipeline.

[0129] This allows you to use the configuration information of each resource and the connection information between each resource, which can be easily extracted from the first pipeline, as information to define the configuration of the first pipeline created in the past.

[0130] Furthermore, the configuration information definition file may also contain configuration information and connection information for each component that constitutes the first pipeline, on a component-by-component basis.

[0131] This allows the first pipeline to be replicated component by component as each component of the second pipeline.

[0132] Furthermore, in the creation step described above, the second pipeline may be created by executing multiple deployment scripts 32 that have been prepared in advance to perform the pipeline replication process.

[0133] This allows for the step-by-step creation of the second pipeline, where the pipeline is assembled in a new environment, by executing multiple pre-created deployment scripts 32 as needed.

[0134] Furthermore, the deployment script 32 may be created by combining commands used in an existing data utilization platform.

[0135] This allows for the creation of various deployment scripts using existing data utilization platforms.

[0136] Alternatively, in the creation step described above, after creating a database resource for storing data in the second pipeline based on the configuration information of each resource in the first pipeline, a resource for processing data in the second pipeline may be created and connected to the database resource.

[0137] This allows for the creation of a second pipeline, divided into two stages: creating database resources and creating resources for data processing and connecting them to the database resources. By checking for errors at each of these stages, resources can be deployed while progressively testing whether they are compatible with the target environment.

[0138] Furthermore, the data processing method in this embodiment may include a modification step for modifying the second pipeline after its creation.

[0139] This allows you to modify the newly created second pipeline to suit the target environment, etc.

[0140] Alternatively, the configuration information of each resource in the modified second pipeline and the connection information between each resource may be obtained in the modification step described above and stored as a configuration information definition file that defines the configuration of the second pipeline.

[0141] This allows the modified second pipeline to be used as the original pipeline for replication.

[0142] Furthermore, in the creation step described above, only specific components of the second pipeline may be created based on information defining the configuration of specific components included in the first pipeline.

[0143] This allows for the rapid and easy construction of a second pipeline adapted to the new environment by reusing only specific components from the first pipeline created in the past.

[0144] The data processing system 100 in this embodiment creates a new second pipeline (destination pipeline) using a previously created first pipeline (source pipeline). The data processing system 100 in this embodiment consists of a remote terminal 200 that can be operated by an engineer and a server device 300 located on a network 400 that can be accessed by the remote terminal 200.

[0145] The server device 300 is characterized by having a storage unit (configuration management 31 shown in Figure 7, storage 316 and configuration management 31 shown in Figure 12) that stores information defining the configuration of the first pipeline, an input receiving unit (communication unit 313 shown in Figure 12) that receives input of identification information for the first pipeline and environmental information for providing the second pipeline from a remote terminal 200 via the network 400, and a creation unit (a processing unit implemented by executing the deployment script 32 shown in Figures 7 and 12) that creates the second pipeline based on the information defining the configuration of the first pipeline identified by the identification information and the environmental information in the environment in which the second pipeline is created.

[0146] This allows for the rapid and easy creation of a second pipeline adapted to the new environment by replicating a previously created first pipeline to any environment, thereby reducing the development effort and time required for pipeline replication. Furthermore, engineers only need to access the server device 300 from a remote terminal 200 and input the identification information of the source first pipeline and the environment information of the destination, significantly reducing the effort required to create a pipeline and suppressing the occurrence of errors that may occur when replicating pipelines manually.

[0147] The present invention is not limited to the embodiments described above, and its technical scope includes various modifications and design changes, etc., without departing from the technical spirit of the present invention. [Explanation of Symbols]

[0148] 10A, 10B, 10X, 10Y customer environment 20A, 20B, 20X, 20Y pipeline 21A, 21B, 21X, 21Y Collection Pipeline 22A, 22X, 22Y shaping pipelines 23A, 23X, 23Y aggregation pipeline 30 Horizontal Expansion Tools 31 Configuration Management 31A Pipeline Management Definition Information (Pipeline Management Definition) 31B Deployment credentials 31C Pipeline Configuration Definition Information (Pipeline Configuration Definition) 31D Configuration Information Definition File (Pipeline A Configuration Definition) 31E Configuration Parameters (Pipeline A Parameter Sheet) 31F Configuration Information Definition File (Pipeline B Configuration Definition) 31G Configuration Parameters (Pipeline B Parameter Sheet) 32, 32A~32I Deployment Scripts 100 Data Processing Systems 200 remote terminals 300 Server Devices 310 Bus 311 Processors 312 memory 313 Communications Department 314 Operation Input Section 315 Monitor 316 storage 316A Data Utilization Platform Execution Program 316B CI Tool Executor Program 316C Lateral Expansion Tool Executor Program 400 Networks

Claims

1. A computer-based data processing method that creates a new second pipeline using a first pipeline that has already been created, A storage step for storing information that defines the configuration of the first pipeline, An input reception step that receives identification information for the first pipeline and environmental information for the replication destination of the second pipeline, A data processing method characterized by comprising: a creation step of creating a second pipeline based on information defining the configuration of the first pipeline identified by the identification information and environmental information for the replication destination of the second pipeline.

2. The data processing method according to claim 1, characterized in that, in the storage step, configuration information of each resource in the first pipeline and connection information between each resource are acquired and stored as a configuration information definition file that defines the configuration of the first pipeline.

3. The data processing method according to claim 2, characterized in that the configuration information definition file contains the configuration information and connection information for each component constituting the first pipeline, on a component-by-component basis.

4. The data processing method according to claim 1, characterized in that the creation of the second pipeline is performed by executing a plurality of deployment scripts that have been prepared in advance in order to perform pipeline replication in the creation step.

5. The data processing method according to claim 4, characterized in that the deployment script is created by combining commands used in an existing data utilization platform.

6. The data processing method according to claim 1, characterized in that, in the creation step, after creating a database resource for storing data in the second pipeline based on the configuration information of each resource in the first pipeline, a resource for performing data processing in the second pipeline is created and connected to the database resource.

7. The data processing method according to claim 1, further comprising a modification step for modifying the second pipeline after its creation.

8. The data processing method according to claim 7, characterized in that it obtains the configuration information of each resource in the second pipeline that has been modified in the modification step and the connection information between each resource, and stores it as a configuration information definition file that defines the configuration of the second pipeline.

9. The data processing method according to claim 1, characterized in that, in the creation step, only the specific component of the second pipeline is created based on information defining the configuration of the specific component included in the first pipeline.

10. A data processing system that creates a new second pipeline using a first pipeline that has already been created, comprising a remote terminal that can be operated by an engineer and a server device located on a network and accessible by the remote terminal, The aforementioned server device A storage unit for storing information defining the configuration of the first pipeline, An input receiving unit that receives identification information of the first pipeline and environment information of the replication destination of the second pipeline from the remote terminal via the network, A data processing system characterized by having a creation unit that creates the second pipeline based on information defining the configuration of the first pipeline identified by the identification information and environmental information for the replication destination of the second pipeline.

Images