Automatic correction of computer programs

Abstract

A method for updating a computer program includes receiving a computer program configured to be hosted on and executed by a first computing system. The method includes analyzing the computer program to obtain a characterization of the lineage, architecture, and behavior of the computer program. The lineage includes relationships between elements of the computer program, the architecture includes characteristics of data sources, characteristics of data targets, and characteristics of one or more processors configured to process data contained in data records, and the operations include processes performed to process data from the data records. The method includes receiving a characterization of an update to be made to the computer program, where when the computer program is modified according to the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system, and modifying the computer program to implement the update and generate a modified computer program.

Description

[Technical field]

[0001] (Priority Claim) This international patent application claims priority under 35 U.S.C. §119(e) to U.S. patent application Ser. No. 63 / 253,851, filed October 8, 2021, which claims priority to U.S. patent application Ser. No. 17 / 704,469, filed March 25, 2022. [Background technology]

[0002] A data processing system may include multiple computer programs executable to process data contained in input data records. Within a data processing system, data records may be passed from one computer program to another, resulting in a set of output data records containing the processed data. Summary of the Invention

[0003] overview In one aspect, a method for updating a computer program includes receiving a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output the processed data records to a data target. The method includes analyzing the computer program by one or more processors to obtain a characterization of the lineage, architecture, and behavior of the computer program. The lineage of the computer program includes relationships between elements of the computer program, the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process data contained in the data records, and the behavior of the computer program includes processes of the computer program executed to process data from the data records. The method includes receiving a characterization of an update to be made to the computer program, where when the computer program is modified according to the update, at least a portion of the modified computer program is hosted on and configured to be executed by a second computing system, and modifying the computer program to implement the update and generate a modified computer program, including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the behavior of the computer program.

[0004] Some embodiments may include one or any combination of two or more of the following features.

[0005] The second computing system is a cloud-based system.

[0006] Modifying a computer program involves merging a characterization of the update with a characterization of the computer program's lineage, architecture, and behavior.

[0007] The ancestry of a computer program includes the relationships between the computer program and other computer programs.

[0008] The computer program includes data processing components configured to process values ​​in fields of data records, the data processing components being connected by links representing the flow of the data records. In some cases, modifying the computer program includes modifying values ​​or representations of parameters of the data processing components or links of the computer program. In some cases, modifying the computer program includes adding new data processing components, deleting data processing components, or both. In some cases, modifying the computer program includes adding new links, deleting links, or both.

[0009] Characterizing the update includes characterizing a distribution processing scheme for at least a portion of the computer program. In some cases, modifying the computer program includes modifying a layout of the computer program to implement distribution of processing operations according to the distribution processing scheme. In some cases, modifying the computer program includes generating a specification of a first new data processing component configured to implement a partitioning operation, generating a specification of a second new data processing component configured to implement a gathering operation, inserting the first new data processing component into the computer program upstream of at least a portion of the computer program, and inserting the second new data processing component into the computer program downstream of at least a portion of the computer program. In some cases, the method includes identifying components of the computer program to which the distribution processing scheme is applied based on the characterization of the update.

[0010] The data source includes a first file system or database, and characterizing the updates made to the computer program includes identifying a second file system or database from which the modified computer program receives the data records. In some cases, modifying the computer program includes updating a name of the data source in a specification of a component of the computer program. In some cases, modifying the computer program includes deleting a first data source component of the computer program that corresponds to the first file system or database and inserting a second data source component that corresponds to the second file system or database.

[0011] The data target includes a first file system or database, and characterizing the update to be made to the computer program includes identifying a set of multiple second file systems or databases to which the modified computer program outputs data records. In some cases, at least one of the second file systems or databases is in a different location than a location of at least one other of the second file systems or databases. In some cases, modifying the computer program includes duplicating a flow in the computer program that connects a data processing component of the computer program to a first data target component that corresponds to the first file system or database, and inserting a new component that corresponds to the first one of the second file systems or databases, the duplicated flow connecting the data processing component to the new component. In some cases, the second file system or database is a cloud-based file system or database, and characterizing the update includes identifying a first characteristic of data stored in a non-cloud-based storage location, identifying a second characteristic of data stored in a second cloud-based file system or database, or both. In some cases, analyzing the computer program includes performing a data lineage analysis and identifying a first component configured to receive or output data records having a first characteristic, a second component configured to receive or output data records having a second characteristic, or both based on the data lineage analysis. In some cases, the first characteristic includes personally identifiable information (PII). In some cases, modifying the computer program includes modifying a specification of a first data processing component that outputs data having the first characteristic, modifying a specification of a second data processing component that outputs data having the second characteristic, or both.

[0012] Modifying the computer program includes identifying a data processing component of the computer program that implements a first type of file transfer protocol and modifying a specification of the data processing component according to a second type of file transfer protocol, and characterizing the update includes indicating a change from the first type of file transfer protocol to the second type of file transfer protocol. In some cases, modifying the specification of the data processing component includes changing a value or representation for one or more parameters of the data processing component.

[0013] The characterization of the update includes requirements for anonymization of personally identifiable information (PII). In some cases, analyzing the computer program includes identifying a data processing component configured to receive a first data record having one or more fields including PII. In some cases, modifying the computer program includes adding a component configured to implement a tokenization service upstream of the identified data processing component, the tokenization service configured to receive the first data record and generalize the PII included in the fields of the received record. In some cases, modifying the computer program includes modifying a specification of the identified data processing component to change a definition of a record format of the data records processed by the identified data processing component. In some cases, the second computing system is a cloud-based computing system, and modifying the computer program includes designating a non-cloud-based computing system for execution of the tokenization service.

[0014] The method includes testing at least a portion of the modified computer program, where testing includes providing an input test data record to the at least a portion of the modified computer program and obtaining a first processed data record from the at least a portion of the modified computer program. In some cases, the method includes testing at least a portion of the computer program, where the at least a portion of the computer program corresponds to the tested portion of the modified computer program, where testing at least a portion of the computer program includes providing an input test data record to the at least a portion of the computer program and obtaining a second processed data record from the at least a portion of the computer program, where testing at least a portion of the modified computer program includes comparing the first processed data record and the second processed data record.

[0015] The method includes migrating the modified computer program to a second computing system.

[0016] Modifying the computer program includes identifying data processing components of the computer program having attribute values ​​that match a target attribute value indicated by the characterization of the update, and replacing the identified data processing components with new data processing components. In some cases, identifying data processing components of the computer program having attribute values ​​that match the target attribute value indicated by the characterization of the update includes generating a first set of data records corresponding to each data processing component of the computer program, each data record including an identifier of the respective data processing component and an attribute value of an attribute of the respective data processing component, and filtering the first set of data records based on the attribute values ​​included in the first set of data records to obtain a second set of data records, including filtering out data records of the first set that do not include a value for a particular attribute that matches the target attribute value indicated by the characterization of the update.

[0017] Modifying a computer program to implement updates and generate a modified computer program includes at least modifying the architecture of the computer program.

[0018] Modifying the architecture includes at least adapting characteristics of one or more processors configured to process data included in the data records to characteristics of one or more processors of the second computing system.

[0019] The received computer program is a copy of a computer program hosted on and executable by the first computing system.

[0020] Modifying the computer program includes generating a copy of the received computer program and modifying the generated copy to generate a modified computer program.

[0021] The modifying of one or more of the computer program lineage, the computer program architecture, or the computer program operation is performed while taking into account characteristics of a second computing system, the characteristics of the second computing system being characteristics of hardware used by the second computing system to execute the modified computer program.

[0022] The second computing system is a cloud-based computing system, and modifying the computer program includes adding a component configured to implement a tokenization service that runs on the local computer system and is configured to anonymize PII included in fields of records received from the data source, preferably by replacing the PII with a token, and ensuring that the token is mapped back to the PII via the tokenization system, but that the PII itself is not provided to the cloud-based system where other processing operations of the modified computer program are performed.

[0023] Each of the method operations is performed automatically by a computing system implementing the method.

[0024] The computing system on which the method is performed is a program update system different from each of the first computing system and the second computing system, and / or the second computing system is different from the first computing system.

[0025] In one aspect, a method for updating a dataflow graph includes accessing the dataflow graph, a specification of the dataflow graph defining nodes, at least one of the nodes representing a data processing component defining operations to be performed to process data in one or more fields of data records having a record format, the data records being provided to the data processing components, and one or more links connecting the nodes, each representing a flow of the data records. The method includes generating a first set of data records representing the dataflow graph, generating data records corresponding to each of the data processing components of the dataflow graph, each data record including an identifier of the data processing component and attribute values ​​for attributes of the data processing component. The method includes receiving a characterization of updates to the data flow graph, the update characterization indicating a target value for a particular attribute; filtering a first set of data records based on the target value indicated by the update characterization to obtain a second set of data records, including removing data records that do not include an attribute value for the particular attribute that matches the target value indicated by the update characterization; and, for each data record in the second set of data records, replacing a corresponding component with a new component indicated by the update characterization.

[0026] Some embodiments may include one or both of the following features.

[0027] Replacing a given component with a new component involves updating the flows connected to the given component.

[0028] Replacing a given component with a new component involves deleting the flows connected to the node representing the given component and creating new flows connected to the node representing the new component.

[0029] Aspects described herein enable automatic modification of a computer program to migrate to a new computing system different from the computing system on which the computer program was originally hosted and configured to execute. To prepare the computer program for migration to the new computing system and for its execution by the new computing system, the automatic modification of the computer program takes into account characteristics of the new computing system to ensure proper operation of the modified computer program in the new computing system, such as proper operation with respect to data security, data processing throughput, computing resource consumption, or correct execution of the processes / functions of the computer program. The characteristics of the new computing system that are considered when modifying the computer program may include characteristics related to the hardware of the new computing system and / or the degree of data protection (data security) in the new computing system. These aspects are particularly beneficial, but not limited to, migration of a computer program from a local computing system to a cloud-based computing system. Details of one or more implementations are set forth in the accompanying drawings and the following description. Other features and advantages will be apparent from the specification and drawings, as well as from the claims. [Brief description of the drawings]

[0030] [Figure 1] FIG. [Diagram 2] 1 is a diagram of a system for updating a computer program. [Diagram 3] FIG. 1 is a diagram of a data flow graph. [Figure 4] FIG. 1 is a diagram of a data flow graph. [Diagram 5] FIG. 1 is a diagram of a data flow graph. [Figure 6] FIG. 1 is a diagram of a data flow graph. [Figure 7] FIG. 1 is a diagram of a data flow graph. [Figure 8] FIG. 1 is a diagram of a data flow graph. [Figure 9] FIG. 1 is a diagram of a data flow graph. [Figure 10A] 1A and 1B are diagrams of a portion of an original and modified dataflow graph, respectively. [Figure 10B] 1A and 1B are diagrams of a portion of an original and modified dataflow graph, respectively. [Figure 11] FIG. [Figure 12] FIG. [Figure 13] FIG. 1 is a diagram of a computing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Described herein are techniques, preferably automated techniques, for modifying a computer program, such as a data flow graph (other types of programs are possible), such that at least a portion of the modified computer program may be hosted on and executed by a computing system different from that of the original computer program. For example, the modification may configure the modified computer program to be hosted on and executed by a cloud-based computing system. The automated modification process includes an automated analysis of the computer program to characterize the lineage, architecture, and behavior of the computer program. One or more of the lineage, architecture, and behavior are then automatically modified to implement an update, such as an update specified by a user or a computing system. An update may be, for example, the introduction of a distributed processing scheme into the modified computer program, the modification of locations for retrieving input data records or storing output data records, or the introduction of a process for masking personally identifiable information (PII), for example to comply with privacy regulations.

[0032] With reference to FIG. 1 , a computer program 100 is hosted on and executed by a first computing system 102. In one example, the first computing system 102 is a computing system that is local to a user running the computer program, e.g., a computing system located at an office site. A user, such as a systems engineer, may wish to update the computer program 100 such that at least a portion of a modified version of the computer program (referred to as a modified computer program 104) is hosted on and executed by a second computing system 106 (e.g., a cloud computing system). For example, the update of the computer program 100 may be performed as part of a process of migrating an application or process from a local environment (e.g., local storage and execution) to a cloud-based environment (e.g., cloud-based storage and / or execution), or to a combination of a local environment and a cloud-based environment. Updating the computer program 100 to generate the modified computer program 104 is an automated process based on input from a user characterizing the purpose of the update, as described in the following paragraphs.

[0033] The computer program 100 may be automatically analyzed to obtain various characteristics of the computer program. For example, the computer program 100 may be analyzed to obtain characteristics of the genealogy of the computer program 100, including relationships between elements of the computer program, relationships between the computer program and other computer programs, or both. The characteristics of the genealogy of the computer program include data indicative of those relationships. The elements may be data processing nodes or components, data resources or targets, data flow links, etc. The computer program 100 may be analyzed to obtain characteristics of the architecture of the computer program, including characteristics of data sources, characteristics of data targets, characteristics of one or more processors configured to execute the computer program (e.g., process data contained in fields of data records), or other features of the computer program architecture. The characteristics of the architecture of the computer program include data indicative of those characteristics. The computer program 100 may be analyzed to obtain characteristics of the operation of the computer program, including processes of the computer program executed during execution of the computer program, e.g., processes executed to process data contained in fields of data records received by the computer program. The characteristics of the operation of the computer program include data indicative of those processes.

[0034] A characterization of an update to the computer program 100 is received, for example, as an input from a user. The update, when implemented, for example, when the computer program 100 is modified according to the update, results in a modified computer program 104, at least a portion of which is configured to be hosted on and executed by the second computing system 106. The computer program 100 may be automatically modified to implement the update, thereby generating the modified computer program 104. Implementing the update includes one or more of modifying the lineage of the computer program, modifying the architecture of the computer program, or modifying the operation of the computer program. The modified computer program 104 is configured to be hosted, at least in part, on and executed by the second computing system 106.

[0035] The computer program 100 and the modified computer program 104 are computer programs that, when executed, receive data records from a data source, process data contained in fields of the data records, and output the processed data to a data target. In some examples, the computer program 100 and the modified computer program 104 are executable data flow graphs. An executable data flow graph is a computer program in the form of a graph that includes nodes, which are executable data processing components and data resources, such as data sources and data targets. The nodes can receive data records in the graph, process data contained in the data records, such as values ​​in fields of the data records, and output the processing results in data records, which are forwarded to a destination in the graph, such as a data resource, e.g., a data target. A data resource is a repository of data, such as a data record, e.g., a source of data that is processed or used during execution of the data flow graph, or a destination (target) of the processed data records that are output by the data flow graph. A data resource can be, for example, a file, a database (e.g., a table of a database), a queue, an object, or other type of data source or target. A link connecting two nodes of the graph provides for the flow of information and / or data (such as data records) between the nodes. An executable dataflow graph is configurable to process data contained in fields of data records when executed. A dataflow graph (sometimes called a graph) may be a data processing graph or plan that controls the execution of one or more graphs. In some examples, one or more data processing components of a dataflow graph are subgraphs.

[0036] 2 is a schematic diagram of a computing system 230 for updating a computer program 200 to generate a modified computer program 204. The computing system 230 includes one or more processors and a memory. The computer programs 200, 204 are executable data flow graphs that can be configured to process data contained in fields of data records, for example. The original computer program 200 is hosted on and executed by a first computing system 201, such as a computing system local to a business or entity interested in the computer program 200. The modified computer program 204 is hosted on and executed by a second computing system 203, such as a cloud-based computing system, that is different from the first computing system 201. In some examples, a portion of the modified computer program 204 is hosted on and / or executed by the first computing system 201, and another portion is hosted on and / or executed by the second computing system 203.

[0037] The system 230 receives a copy of the program 200 from the system 201, or the system 230 receives the program 200 that is configured to be hosted on the system 201. The system 230 includes a program analysis module 210 that analyzes (a copy of) the received computer program 200 to obtain a characterization 211 of the lineage of the computer program 200, the architecture of the computer program 200, the operation of the computer program 200, or a combination of any two or more of them. This characterization 211 is passed to the modification module 220.

[0038] The lineage of the computer program 200 includes relationships between elements of the computer program (e.g., relationships between nodes, data resources, or both), relationships between the computer program and other computer programs, or both. The lineage of the computer program 200 can identify or be based on static dependencies, runtime dependencies, or both. A static dependency between two elements or programs is a dependency that is defined by values ​​in a previously stored parameter set associated with the computer program. A static dependency between elements of the computer program or between computer programs is identified by static analysis of values ​​in a stored parameter set. A runtime dependency between two elements or computer programs is a dependency that is defined during execution of one of the computer programs, but is not evident from the static analysis. When the computer program 200 is executed to process data records, the execution commands may include parameter values ​​in addition to or instead of parameter values ​​in a previously stored parameter set that define, for example, a static dependency. A runtime log generated during execution of the computer program indicates these parameter values, for example, indicating nodes that were executed or data resources that were accessed. Run-time dependencies between elements of the computer programs or between computer programs are identified by analysis of parameter values ​​indicated in the run-time log.

[0039] Further description of static analysis and runtime analysis can be found in U.S. Patent Application Publication Nos. 2016 / 0019057 and 2016 / 0019057, the contents of both of which are incorporated by reference in their entireties.

[0040] The architecture of the computer program 200 includes characteristics of a data source, characteristics of a data target, characteristics of one or more processors configured to execute the computer program (e.g., process data contained in fields of data records), or other features of the computer program architecture. The characterization of a data source or data target may include a name of the data source or data target, a type of data source or data target (e.g., database, file, queue, etc.), an actual location of the data source or data target (e.g., a physical file or dataset path), or a parameterized location of the data source or data target. A parameterized location is an expression that resolves to an actual data source or data target path, e.g., a physical file or dataset path, upon execution of the computer program. For example, a dataset, such as a data source or data target, may be characterized by a parameterized path / ${FEED} / inv_${DATE}.dat. At execution, the computer program receives values ​​of the FEED and DATE parameters such that the parameterized path can be resolved to a specific physical location. The characterization may include identification or indication of multiple data sources or data targets, e.g., local and / or cloud-based data sources or data targets. The characterization of the one or more processors may include the number of processors, a layout for a distributed processing scheme (e.g., an indication of a partitioning scheme for the processing operations), a location of each of the one or more processors, a target or actual power consumption for each of the one or more processors, or other characteristics.

[0041] The operations of the computer program 200 include computer program processes that are performed when the computer program is executed, such as processes that are performed to process data contained in fields of data records. The computer program processes may include, for example, file transfer operations, sorting operations, filtering operations, joining operations, and other suitable operations.

[0042] The system 230 includes a modification module 220 that receives a characterization 222 of an update to be made to the computer program 200. The characterization 222 of the update may be received from a user via a user interface 224, such as a program development interface. The update is a modification made to the computer program 200 such that, when the computer program 200 is modified according to the update to generate a modified computer program 204, at least a portion of the modified computer program 204 is configured to be hosted on and executed by the second computing system 203. Examples of characterizations of an update include one or more of the following: ● Characterizing the distributed processing scheme implemented for at least a portion of the modified computer program, e.g. characterizing the partitioning (e.g. changing the width of the processing operations for at least a portion of the modified computer program, or changing the rules by which data records are partitioned into each of a number of processing streams (e.g. changing from partitioning based on customer number to partitioning based on product number). ● Changing the data sources, data targets, or both of the modified computer program 204 (e.g., identifying one or more data sources or data targets that are different from the data sources or data targets, respectively, of the computer program 200), or changing the number of data sources, data targets, or both (e.g., changing from local data targets to cloud-based data targets, or to a combination of local targets and cloud-based data targets). ● Changing the type of file transfer protocol implemented by the modified computer program (e.g., from the traditional File Transfer Protocol (FTP) to the Secure File Transfer Protocol (SFTP)). • The introduction of a requirement to mask personally identifiable information (PII) in data records received or processed by the modified computer program 204.

[0043] Modification module 220 implements updates 222 by combining updates 222 with characterization 211 of computer program 200 and applying the result to computer program 200, thereby generating modified computer program 204. Modifying computer program 200 includes modifying the lineage of computer program 200, the architecture of the computer program, the operation of computer program 200, or a combination of any two or more thereof. In some examples, modifying computer program 200 includes directly modifying computer program 200 itself to generate modified computer program 204. In some examples, modifying the computer program includes generating a copy of computer program 200 and modifying the copy to generate modified computer program 204.

[0044] If the computer program 200 is a data flow graph, the modification may include modifying values ​​or representations of parameters of data processing components or links of the data flow graph. The modification may include adding new data processing components or links, deleting new data processing components or links, or a combination thereof, such as to account for characteristics (e.g., hardware characteristics) of the system 203. For example, to replace a first data processing component in the computer program 200 with a second data processing component, the first data processing component and its associated links are deleted and the new data processing component and appropriate links are added.

[0045] In some examples, the modification module 220, upon receiving the characterization of the update from the user, automatically identifies components of the computer program 200 to be modified and performs the updates. For example, the characterization of the update 222 can identify characteristics of components in the original computer program 200 to be modified and can specify how to modify components that meet the characteristics. Based on the characterization 211 of the computer program 200, the modification module 220 performs a search and replace process, described in more detail below, to identify components that meet the characteristics and replace those identified components with appropriately modified replacement components. In a particular example, the characterization of the update indicates that all data targets in the original computer program 200 having a particular location parameter (e.g., a path to a local data storage location) are to be modified to have different location parameters (e.g., a path to a cloud-based data storage location). The modification module 222 searches for all data target components having the specified location parameters and replaces them with data target components having the different location parameters.

[0046] In some examples, the characterization 222 of the update provided by the user identifies or indicates components that are modified or affected by the modification, and the modification module 220 implements the update of those components. For example, the characterization 222 of the update can identify a set of processes and can specify how the set of processes is to be modified. Based on the characterization 211 of the computer program 200, the modification module 220 can identify components to be updated. In a particular example, the characterization of the update includes identifying a set of processes to be executed according to a distributed processing scheme in the modified computer program. The modification module 220 identifies components corresponding to those processes based on the characterization 211 of the architecture and operation of the computer program 200, and inserts a partition component upstream of the components identified by the user and a collection component downstream of the components identified by the user, as described in more detail below.

[0047] In some examples, the update characterization 222 can specify that all personally identifiable information (PII) is anonymized in a tokenization process before being provided to the system 203, such as to occupy a less protective data security environment in the second computer system 203. The tokenization process can result in a change in format, for example, from a numeric format of the PII to an alphanumeric format for the resulting token. The modification module 220 can identify components whose record formats should be updated to account for the change in format based on the data lineage characterization of the computer program 200.

[0048] FIG. 3 illustrates an example of a computer program 300 and a modified computer program 304. The computer program 300 does not implement a distributed processing scheme and is modified according to an update to implement a distributed processing scheme in a portion 302 of the modified computer program 304 to account for the characteristics of a second computing system. A distributed processing scheme is an architecture in which multiple processors are used to execute a computer program. In some examples, the multiple processors may be located on different computing devices. For example, one or more of the processors may be located on a local computer, such as one that complies with privacy regulations, and one or more other processors may be cloud-based processors. A distributed computing scheme may be used to, for example, achieve efficient processing, introduce elasticity and scalability, achieve migration of a computer program to a cloud-based system while maintaining compliance with privacy regulations that may prohibit cloud-based processing of sensitive data, such as personally identifiable information (PII), or for other reasons.

[0049] Implementing the distributed processing scheme in the modified computer program 304 may include modifying the layout of the computer program 300 to implement the distribution of processing operations as specified by the distributed processing scheme. The modification includes generating a specification of a first new data processing component 306 that implements a partitioning operation. The partitioning component 306 is inserted in the modified computer program 304 upstream of the portion 302 of the modified computer program 304 that implements the distributed processing, for example between the upstream component 310 and the first component 312 of the distributed processing portion 302. The link connecting the upstream component 310 to the first component 312 in the original computer program 300 is deleted and two new links are generated to connect the partitioning component 306 to the components 310, 312. Furthermore, the modification includes generating a specification of a second new data processing component 308 that implements the gather operation. The collection component 308 is inserted into the modified computer program 304 downstream of a portion 302 of the modified computer program 304 that implements distributed processing, for example, between a final component 314 and a downstream component 316 of the distributed processing portion 302. The link connecting the final component 314 to the downstream component 316 in the original computer program 300 is deleted and two new links are created to connect the collection component 308 to the components 314, 316.

[0050] In some examples, the original computer program already implemented a distributed processing scheme and the update is a change to the partitioning of the distributed processing scheme. The change can be a change to the number of parallel streams (e.g., the original computer program implemented N parallel streams and the update is a change to M parallel streams). The change can be a change to the way the data records are partitioned. For example, the original computer program may implement partitioning of the data records based on a first characteristic of the data contained in the data records (e.g., the data records may be partitioned according to a customer number) and the modified computer program may implement partitioning of the data records based on a different characteristic of the data (e.g., the data records may be partitioned according to a product number). The change can be a change to the location of the distributed processor. For example, both the original computer program and the modified computer program may implement the same number of parallel streams, but the processing by the original computer program is performed by a local processor and the processing by the modified computer program is performed by a cloud-based processor.

[0051] In one example, modification of a computer program 300 to introduce a distributed processing scheme is performed as follows: The lineage, architecture, and operation of the computer program 300 are characterized in an automated analysis process. A user provides input identifying one or more processes that are implemented according to the distributed processing scheme, and input identifying partitioning characteristics, e.g., the number of parallel streams and the location per processor (e.g., path) that executes parts of the distributed processing scheme. The input characterization is combined with a characterization of the lineage, architecture, and operation of the computer program 300 to generate a modified computer program 304. For example, a set of computer program components that together correspond to the one or more processes identified in the user input are identified, and a partition component and a collect component are inserted upstream and downstream, respectively, of the set of identified components.

[0052] 4 illustrates an example of a computer program 400 and a modified computer program 404 that has been modified to change the data source. The computer program 400 receives data records from a first data source (in this example, a database) corresponding to a data source component 402 connected to a processing component 406 of the computer program 400 by a first link 408. An update changes the source of the data records, so that the modified computer program 404 receives data records from a second data source represented by a data source component 410 connected to the processing component 406 by a second link 412. In the example of FIG. 4, the first data source corresponding to the data source component 402 is a database and the second data source is a file system, with the data source component 410 being a component configured to read from a Hadoop Distributed File System (HDFS). Other types of data sources may be used for the computer program 400, the second computer program 404, or both. In some examples, both data sources may be the same type of data source, and attributes such as the name or location (eg, path) of the data source may be changed.

[0053] In some examples, the data source component 410 may be identical to the data source component 402 except for a change in the characteristics of the data source, such as a name or path (e.g., location) of the data source, such as from a local data source to a cloud-based data source. For example, parameters of the data source component 402 may be changed to obtain the data source component 410. In some examples, the modifications to the computer program include deleting the data source component 402 and the first link 408, and adding a data source component 410 and a second link 412 to connect the newly added data source component 410 to the processing component 406, where the newly added data source component 410 represents a data source for the modified computer program 404.

[0054] In one example, modification of computer program 400 to change data sources is performed as follows: The lineage, architecture, and behavior of computer program 400 are characterized in an automated analysis process. A user provides input indicating changes to the data sources, e.g., identification of data source paths in the computer program that are to be changed to different data source paths in the modified computer program. The characterization of the input is combined with the characterization of the lineage, architecture, and behavior of computer program 400 to generate modified computer program 404. For example, a search and replace process is performed to identify data sources that match the data source paths specified in the user input and replace the corresponding data source components with new data source components that represent data sources with different data source paths.

[0055] 5 illustrates examples of a computer program 500 and a modified computer program 504 that has been modified to change a data target. The computer program 500 outputs processed data records to a first data target (in this example, a database) represented by a data target component 502 connected to a processing component 506 of the computer program 500 by a first link 508. The update changes the target of the processed data records, so that the modified computer program 504 outputs the processed data records to a second data target (in this example, a file system) represented by a data target component 510 connected to the processing component 506 by a second link 512. Although the processing component 506 is illustrated as being the same component between the computer program 500 and the modified computer program 504, in some examples the processing component may differ between the two computer programs.

[0056] In some examples, the data target component 510 may be identical to the data target component 502, except for a change in the characteristics of the data target, such as a name or path (e.g., location) of the data target, such as from a local data target to a cloud-based data target. In some examples, the modifications to the computer program include deleting the data target component 502 and the first link 508, and adding a data target component 510 and a second link 512 to connect the newly added data target component 510 to the processing component 506, where the newly added data target component 510 represents a data target of the modified computer program 504.

[0057] In some examples, the data targets may be different types of file transfer protocols, e.g., the data target component 502 may implement a first type of file transfer protocol and the data target component 510 implements a second type of file transfer protocol, e.g., the secure ftp protocol.

[0058] In one example, modification of computer program 500 to change data targets is performed as follows: The lineage, architecture, and behavior of computer program 500 are characterized in an automated analysis process. A user provides input indicating changes to the data targets, e.g., identification of data target paths in the computer program that are to be changed to different data target paths in the modified computer program. The characterization of the input is combined with the characterization of the lineage, architecture, and behavior of computer program 500 to generate modified computer program 504. For example, a search and replace process is performed to identify data targets that match the data target paths specified in the user input and replace the corresponding data target components with new data target components that represent data targets with different data target paths.

[0059] The computer program may be modified to change the data target of the output data records to comply with restrictions on storage of data in cloud-based data storage, such as a cloud-based file system or database. For example, privacy regulations may dictate that certain data having certain characteristics, such as data records containing personally identifiable information (PII), may not be stored in cloud-based data storage, and that other data may be stored in cloud-based data storage.

[0060] Identification of data targets within the computer program 500 that will receive data records containing PII may be performed by data lineage analysis, possibly in combination with semantic discovery analysis.

[0061] Data lineage is information, such as a data flow graph, that describes the life cycle of a data record processed by a computer program. Data lineage information for a given data record may include identifiers of one or more data records on which the given data record depends, one or more downstream data records that depend on the given data record, one or more components of the computer program that process the data record to generate the given data record, and one or more components of the computer program that process the given data record or data records that depend on the given data record. A downstream data record that depends on an upstream data record is one in which the processing of an upstream data record by a computer program directly or indirectly results in the generation of a downstream data record. The generated downstream data record may be a data record that is output from the computer program (sometimes called an output data record) or a data record that is further processed by the computer program (sometimes called an intermediate data record). An upstream data record may be a data record input to the computer program (sometimes called an input data record or a reference data record) or a data record that has already been processed by the computer program (sometimes called an intermediate data record). Data lineage analysis is the analysis of a computer program to identify data records that depend on or on which a given data record depends, and to identify components of the computer program that process the data records to produce the given data record, process the given data record, or process data records that depend on the given data record.

[0062] In certain instances where data records containing PII are ensured not to be stored in or processed by a cloud-based system, a data lineage analysis may be performed to identify components that process data records containing PII or data targets that receive data records containing PII. These data targets may remain local data targets and these components may be implemented using local processors.

[0063] In some examples, data records containing PII are automatically identified in a semantic discovery process. Further description of semantic discovery can be found in U.S. Patent Application Publication No. 2020 / 0380212, the contents of which are incorporated herein by reference in their entirety.

[0064] FIG. 6 illustrates an example of a computer program 600 and a modified computer program 604 that has been modified to output processed data records to multiple data targets. The computer program 600 outputs the processed data records to a first data target (in this example, a database) represented by a data target component 602 connected to a processing component 606 of the computer program 600 by a first link 608. The update removes the first data target and introduces multiple second data targets (in this example, multiple databases) represented by a replication component 614 for replicating the data flow and data target components 610a-610c each connected to the processing component 606 by a respective second link 612a-612c. At least one of the second data targets may be in a location that is different from the location of the other second data targets. In some examples, one of the second data targets is identical to the first data target and additional data targets are added. In some examples, all of the second data targets are different from the first data target. In some examples, the opposite modification may be made, such that a computer program that outputs processed data records to multiple data targets may be modified to only output processed data records to a single data target. Other types of data targets, such as a file system, may be used for the computer program 600, the modified computer program 604, or both.

[0065] In some examples, the modifications to the computer program include deleting the data target component 602 and the first link 608, and adding data target components 610a-610c and second links 612a-612c to connect the newly added data target components 610a-610c to the processing components. For example, one of the data target components (e.g., component 610a) and its corresponding link (e.g., link 612a) are added. Then, component 610a and link 612a are duplicated to generate and connect the other data target components 610b, 610c. Then, parameters of the data target components 610b, 610c are changed to reference the respective data targets.

[0066] 7 illustrates an example of a computer program 700 and a modified computer program 704. A component 706 of the computer program 700 performs processing according to a first parameter set 710. The update modifies the component 706 so that the process is performed according to a second (different) parameter set 712. The modification includes analyzing the computer program 700 to identify a data processing component that includes a particular parameter, parameter expression, or parameter value. In some examples, the specification of the identified component is modified, e.g., by changing the value or expression of one or more parameters in the identified component's parameter set 710, to generate a modified component 708 having a modified parameter set 712. In some examples, the identified component 706 is removed and replaced by a new component 708 having a different parameter set.

[0067] 8 illustrates an example of a computer program 800 and a modified computer program 804. The computer program 800 is configured for execution on a local computer system and the modified computer program 804 is configured such that at least a portion of the execution of the modified computer program 804 occurs on a cloud-based system. In order to comply with restrictions regarding PII in cloud-based systems, such as prohibitions on processing or storing PII in cloud-based systems, the computer program is modified such that all PII in data records received and processed by the computer program 800 is anonymized by the modified computer program 804 before reaching the cloud-based system.

[0068] The modification of the computer program 800 includes analyzing the computer program to identify a data processing component (e.g., component 806) configured to receive data records having one or more fields containing PII from a data source represented by the data source component 810 and process the data records. The analysis can be, for example, a data lineage analysis performed in combination with a semantic discovery analysis. The component 806 is modified such that the modified component 808 of the modified computer program 804 is configured to run on a cloud-based computing system, for example, as described above. In addition, a new component 812 is added between the data source 810 and the component 806, such that the new component receives data records from the data source 810 and outputs data records to the component 808. The component 812 is configured to run on a local (e.g., non-cloud-based) computer system and performs a tokenization service configured to anonymize PII contained in fields of records received from the data source 810, for example, by replacing the PII with a token. The token is remapped to PII via the tokenization system, ensuring that the PII itself is not provided to a cloud-based system where other processing operations of the modified computer program 804 take place.

[0069] In some examples, the token has a different format than the PII. For example, if the PII is a social security number, the PII is a nine-digit number, but the token may be an alphanumeric value of a different number of digits. If the format of the token differs from the format of the PII, component 808 is modified to change the definition of the record format of the data records processed by component 808 so that component 808 is compatible with the format of the token. In some cases, other downstream components are also modified accordingly to process data records of different record formats.

[0070] In some examples, if the data records output from the computer program 800 include PII, the data records output from the modified computer program 804 include respective tokens of the PII. In some examples, a de-tokenization component is added to the modified computer program 804, e.g., immediately upstream of the data target, to reintroduce the PII into the data records in a process executed on the local computer system. The de-tokenization component is added in a process similar to how the tokenization component 812 is added.

[0071] In some examples, components to be updated are automatically identified and replaced in a search and replace process that, when applied to a particular computer program, implements a search function that searches the components of the computer program and obtains their attributes, a filter function that filters components having attributes that match target attributes (e.g., attributes identified in a user's characterization of the update as attributes that will trigger replacement), a reformatting function that describes how to make each replacement, and a replace function that performs the replacement.

[0072] Figure 9 illustrates an example of a dataflow graph 900 implementing a search and replace process. In the example of Figure 9, the search and replace process replaces an input file to a component configurable to read from a Hadoop Distributed File System (HDFS). A first processing component 902 of the dataflow graph 900 receives data records from an input component 904. The data records provide instructions for the replacement process, such as an identification of the computer program (e.g., dataflow graph) in which the replacement is to be performed, attributes of the component to be replaced, and a description of the replacement component.

[0073] The first processing component 902 is a search component that searches all of the components of the computer program and retrieves their attributes. The search component 902 passes a set of data records to a filter component 906, each record corresponding to one component of the computer program for which a substitution is to be performed. Each record includes an identifier for the respective component (e.g., computer program identification, component path, etc.) and a list of attributes for the respective component of the computer program.

[0074] The filter component 906 filters the data records received from the search component 902 to retain records that correspond to components having attributes that match the target attributes (e.g., attributes identified in the user's characterization of the update as attributes that trigger replacement). In this example, the filter component 906 filters the data records to retain only those data records that correspond to components that represent the input file. The retained data records are passed to the reformatting component 908.

[0075] The reformatting component 908 creates records that describe the replacement components that replace each of the components that correspond to the records received from the filter component 906. These records may include some or all of the following data: -Identification of the computer program (e.g., data flow graph) in which the permutation is to be performed. -Identification of the component of the computer program to be replaced (e.g., a path). -An identification for the new replacement component (e.g. a parameterized path). - The name of the new replacement component. - A list of names of ports on the replacement component (e.g. a vector). - Mapping of the names of the ports of the replacement component to the names of the ports of the component being replaced. - A list of names of parameters for the replacement component (e.g. a vector). - A mapping of the names of the parameters of the replacing component to the names of the parameters of the replaced component. - A list (e.g. a vector) of instance values ​​to be replaced.

[0076] The records that are output from the reformatting component 908 characterize the replacements. These records are provided to a replacement component 910 that performs the replacements for each component indicated by the records. A log of the replacement operations is output to a log file 912.

[0077] 10A and 10B show examples of an original dataflow graph 150 and a modified dataflow graph 152 that has been updated using an automated search and replace process to replace an input file represented as component 154 with a READ HDFS component 156.

[0078] In some examples, the modified computer program is tested prior to release to verify the operation of the modified computer program. For example, the modified computer program may be expected to output data records that are identical to the data records output by the original computer program for an identical set of input data records. Testing may include providing an identical input data set to both the original computer program and the modified computer program, and obtaining respective sets of processed data records from both computer programs. The two sets of processed data records are compared. If the two sets of processed data records match, e.g., are identical (e.g., are identical within a threshold), the modified computer program is verified. If the two sets of process data records do not match, an error message is output, preferably accompanied by guidance on how to resolve the error.

[0079] In some examples, only the portion of the computer program that has been modified is tested, e.g., only the portion of the modified computer program that has been changed relative to the original computer program. The portion of the computer program may be tested by using insertions, e.g., test sources and probes, which are objects associated with a flow in a data flow graph. A test source replaces data passing through a flow (e.g., upstream data) with new data, making it unnecessary for the upstream computation to be re-run for each execution of the computer program. For example, a test source may replace a data source such that test data records are provided to the graph from the test source rather than from the data source. A probe may monitor data records passing through a flow as the graph executes, such that the data records are saved for later inspection or reuse. For example, a probe may receive data records that would have otherwise been saved in a data target, such as a database. Insertions may also be introduced at locations in the data flow graph other than the data sources and targets, allowing the graph developer to access the data records as they pass through the graph. By using insertions, input data record sets may be provided to the original and modified computer programs just upstream of the portion of the modified computer program, and processed data records may be retrieved just downstream of the modified portion. The use of insertion can allow for more efficient testing in that less than the entire computer program is executed during testing. Further description of insertion can be found in U.S. Patent No. 10,055,333, entitled "Debugging a Graph," and U.S. Patent No. 9,880,818, entitled "Application Testing," the contents of both of which are incorporated herein by reference in their entireties.

[0080] 11, in an exemplary process for updating a computer program, a computer program is received (150). The computer program is configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output the processed data records to a data target. The computer program may be a data flow graph including data processing components configured to process values ​​in the fields of the data records, the data processing components being connected by links representing the flow of the data records.

[0081] The computer program is analyzed by one or more processors to obtain a characterization of the lineage, architecture, and behavior of the computer program (152). The analysis may include a static analysis of the computer program, a run-time analysis of the computer program, a schedule analysis of the computer program, a data lineage analysis of the computer program, a semantic discovery analysis of the computer program, or other types of analysis. The lineage of the computer program includes relationships between elements of the computer program, and optionally relationships of the computer program to other computer programs. The architecture of the computer program includes characteristics of data sources, characteristics of data targets, and characteristics of one or more processors configured to process data contained in the data records. The behavior of the computer program includes processes of the computer program executed to process data from the data records.

[0082] A characterization of an update to be made to the computer program is received, for example, by user input to a user interface (154), such that when the computer program is modified pursuant to the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system, such as a cloud-based system.

[0083] The computer program is modified to implement the updates, thereby generating a modified computer program (156). The modification may include modifying one or more of the lineage, architecture, or behavior of the computer program. The modification may include merging a characterization of the updates with characterizations of the lineage, architecture, and behavior of the computer program.

[0084] Where the computer program is a data flow graph, modifying the computer program may include modifying values ​​or representations of parameters of data processing components or links of the computer program, adding new data processing components or links, deleting data processing components or links, or other suitable modifications.

[0085] At least a portion of the modified computer program is tested (158). Testing includes providing input test data records to at least a portion of the modified computer program and obtaining a first processed data record from at least a portion of the modified computer program. Testing also includes testing at least a portion of the computer program corresponding to the tested portion of the modified computer program, including providing input test data records to at least a portion of the computer program and obtaining a second processed data record from at least a portion of the computer program. The first processed data record and the second processed data record are compared. Testing may include providing test data to the modified computer program that triggers each or substantially all functions of the computer program to generate the processed data record as an output.

[0086] 12, an exemplary process for updating a dataflow graph includes accessing 250 the dataflow graph. The dataflow graph specification defines nodes, at least one of which represents a data processing component that defines operations to be performed to process data in one or more fields of data records having a record format, the data records being provided to the data processing component, and one or more links connecting the nodes, each representing a flow of the data records.

[0087] A first set of records is generated, the first set of records representing the dataflow graph (252), including generating data records corresponding to each data processing component of the dataflow graph, each data record including an identifier for the data processing component and attribute values ​​for attributes of the data processing component.

[0088] A characterization of updates to the dataflow graph is received 254, the characterization of the updates indicating a target value for a particular attribute. The first set of data records is filtered based on the target value indicated by the characterization of the updates to obtain a second set of data records 256, including filtering to remove data records that do not include an attribute value for the particular attribute that matches the target value indicated by the characterization of the updates.

[0089] For each data record in the second set of data records, the corresponding component is replaced 258 with a new component as indicated by the characterization of the update.

[0090] 13 illustrates an example of a data processing system 850 for developing and executing data flow graphs in which the techniques described herein may be used. The system 850 includes data sources 852, which may include one or more data sources, such as connections to storage devices or online data streams, each of which may store or provide data in any of a variety of formats (e.g., database tables, spreadsheet files, flat text files, or native formats used by mainframe computers). The data may be logistics data, analytical data, or industrial machine data. The execution or runtime environment 854 includes a pre-processing module 856 and an execution module 862. The execution environment 854 may be hosted on one or more general-purpose computers under the control of a suitable operating system, such as, for example, a version of the UNIX operating system. For example, the execution environment 854 may include a multi-node parallel computing environment, including the configuration of computer systems using multiple processing units (such as central processing units (CPUs)) or processor cores, either local (e.g., a multiprocessor system such as a symmetric multi-processing (SMP) computer), or locally distributed (e.g., multiple processors coupled as a cluster or massively parallel processing (MPP) system), or remote, or remotely distributed (e.g., multiple processors coupled via a local area network (LAN) and / or wide-area network (WAN)), or any combination thereof.

[0091] The storage device providing the data source 852 may be local to the execution environment 854, e.g., stored on a storage medium (e.g., hard drive 858) connected to the computer hosting the execution environment 854, or may be remote to the execution environment 854, e.g., hosted on a remote system (e.g., mainframe computer 860) that communicates with the computer hosting the execution environment 854 via a remote connection (e.g., provided by a cloud computing infrastructure).

[0092] The pre-processing module 856 reads data from the data sources 852 and prepares a data processing application (e.g., an executable data flow graph) for execution. For example, the pre-processing module 856 may compile the data processing application, store and / or load the compiled data processing application to and / or from a data storage system 866 accessible to the execution environment 854, and perform other tasks to prepare the data processing application for execution.

[0093] The execution module 862 executes the data processing application prepared by the pre-processing module 856 to process the data set and generate output data 864 resulting from the processing. The output data 864 may be stored back to the data source 852, or in a data storage system 866 accessible to the execution environment 854, or may be otherwise used. The data storage system 866 is also accessible to an optional development environment 868 in which a developer 870 can design and edit the data processing application to be executed by the execution module 862. The development environment 868, in some implementations, is a system for developing applications as data flow graphs that include vertices (representing data processing components or data sets) connected by directed links (representing work elements, i.e., flows of data) between vertices (representing data processing components or data sets). For example, such an environment is described in more detail in U.S. Patent Application Publication No. 2007 / 0011668, entitled "Managing Parameters for Graph-Based Applications," the contents of which are incorporated herein by reference in their entirety. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," the contents of which are incorporated herein by reference in their entirety. Dataflow graphs created in accordance with this system provide a method for capturing information flowing into and out of individual processes represented by graph components in order to move information between the processes and define the order of execution of the processes. The system includes an algorithm for selecting a method of communication between the processes from any available method (e.g., communication paths through links in the graph could use TCP / IP or UNIX domain sockets, or shared memory could be used to pass data between processes).

[0094] The pre-processing module 856 can receive data from various types of systems that may embody the data source 852, including different forms of database systems. The data may be organized as records with values ​​for each field (also called "attributes" or "columns"), possibly including null values. When initially reading data from a data source, the pre-processing module 856 typically starts with some initial format information about the records in that data source. In some situations, the record structure of a data source may not be known initially, but instead may be determined after analysis of the data source or data. The initial information about the records may include, for example, the number of bits that represent different values, the order of the fields in the record, and the type of values ​​represented by the bits (e.g., string, signed / unsigned integer).

[0095] In other words, as generally applicable to executable dataflow graphs described herein, an executable dataflow graph implements graph-based computations performed on data flowing from one or more input datasets of a data source 852 through data processing components, the dataflow graph being specified by a data structure in data storage 864, the dataflow graph having nodes representing data processing components specified by the data structure and connected by one or more links, the links being specified by the data structure and representing data flow between the data processing components. An execution environment or runtime environment 854 is coupled to the data storage 864 and hosted on one or more computers, the runtime environment 854 including a pre-processing module 856 configured to read the stored data structure specifying the dataflow graph and to allocate and configure system resources (e.g., processes, memory, CPU, etc.) to execute computations of the data processing components assigned to the dataflow graph by the pre-processing module 856, the runtime environment 854 including an execution module 862 for scheduling and controlling execution of the computations of the data processing components. In other words, a runtime or execution environment 854 hosted on one or more computers is configured to read data from data sources 852 and process the data using executable computer programs represented in the form of a dataflow graph.

[0096] The above techniques can be implemented using a computing system executing suitable software. For example, the software may include procedures in one or more computer programs running on one or more programmed or programmable computing systems (which may be of various architectures such as distributed, client / server, or grid), each of the computing systems including at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), at least one user interface (for receiving input using at least one input device or port and providing output using at least one output device or port). The software may include one or more modules of a larger program that provides services related to, for example, graph design, configuration, and execution. The modules of the program (e.g., elements of the graph) can be implemented as data structures or other organized data conforming to a data model stored in a data repository.

[0097] The software may be provided on a tangible, non-transitory medium such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or may be delivered (e.g., encoded in a propagating signal) via a communication medium of a network to a tangible, non-transitory medium of a computing system to be executed, where it is executed. Part or all of the processing may be executed on a special-purpose computer, or may be executed using special-purpose hardware such as a coprocessor or a field programmable gate array (FPGA) or a special-purpose application-specific integrated circuit (ASIC). The processing may be implemented in a distributed manner, with different parts of the computation specified by the software being performed by different computing elements. Each such computer program is preferably stored or downloaded into a computer-readable storage medium (e.g., solid-state memory or medium, or magnetic or optical medium) of a storage device accessible by a general-purpose or special-purpose programmable computer, such that when the storage device medium is read by the computer, it configures and operates the computer to perform the processing described herein. The systems of the present invention may also be considered to be implemented as a tangible, non-transitory medium configured with a computer program, the medium so configured causing a computer to operate in a particular, predetermined manner to perform one or more of the process steps described herein.

[0098] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present invention. For example, some of the steps described above may be order independent and therefore can be performed in an order different from that described.

[0099] Other implementations are within the scope of the following claims.

Claims

1. 1. A method for updating a computer program, the method being performed by a computing system, comprising: reading from a data storage device a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output data records including the processed data to a data target; analyzing, by one or more processors, the retrieved computer program and storing in a data storage device a characterization of the lineage, architecture, and behavior of the computer program; the lineage of the computer program includes static or runtime dependencies between elements of the computer program, static or runtime dependencies between the computer program and other computer programs, or both; the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process the data contained in the data records; the operations of the computer program include processes of the computer program being executed to process the data from the data records; and retrieving from a data storage device a characterization of an update to be made to the computer program, wherein when the computer program is modified according to the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system, and the characterization of the update includes a requirement for anonymization of person identifiable information (PII) in data stored or processed by the second computing system; storing the modified computer program in a data storage device, the modified computer program being generated by modifying the computer program to implement the update, the modification of the computer program including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the operation of the computer program; Migrating the modified portion of the computer program to the second computing system; executing the modified computer program, including executing portions of the modified computer program by the second computing system; A method comprising:

2. The method of claim 1 , wherein the second computing system is a cloud-based system.

3. The method of claim 1 , wherein modifying the computer program comprises merging the characterization of the update with the characterization of the lineage, the architecture, and the behavior of the computer program.

4. 2. The method of claim 1 , wherein the computer program includes data processing components configured to receive the data records from the data source, process values ​​in fields of the received data records, and output data records including results of the processing of the values, the data processing components being connected by links representing a flow of data records.

5. The method of claim 4 , wherein modifying the computer program comprises modifying values ​​or representations of parameters of data processing components or links of the computer program.

6. The method of claim 4 , wherein modifying the computer program comprises adding a new data processing component, deleting a data processing component, adding a new link, or deleting a link.

7. A method for updating a computer program, the method being performed by a computing system, comprising: reading from a data storage device a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output data records including the processed data to a data target; analyzing, by one or more processors, the retrieved computer program and storing in a data storage device a characterization of the lineage, architecture, and behavior of the computer program; the lineage of the computer program includes static or runtime dependencies between elements of the computer program, static or runtime dependencies between the computer program and other computer programs, or both; the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process the data contained in the data records; the operations of the computer program include processes of the computer program being executed to process the data from the data records; and reading from a data storage device a characterization of an update to be made to the computer program, wherein when the computer program is modified in accordance with the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system, the characterization of the update including characterizing a distributed processing scheme for at least a portion of the computer program, and modifying the computer program including modifying a layout of the computer program to implement distribution of processing operations in accordance with the distributed processing scheme; storing the modified computer program in a data storage device, the modified computer program being generated by modifying the computer program to implement the update, the modification of the computer program including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the operation of the computer program; Migrating the modified portion of the computer program to the second computing system; executing the modified computer program, including executing portions of the modified computer program by the second computing system; A method comprising:

8. Modifying the computer program comprises: generating a specification of a first new data processing component configured to perform a partitioning operation; generating a specification for a second new data processing component configured to perform the collection operation; inserting the first new data processing component into the computer program upstream of the at least part of the computer program; inserting the second new data processing component into the computer program downstream of the at least part of the computer program; The method of claim 7, comprising:

9. The method of claim 1, wherein the data source includes a first file system or database, and wherein the characterization of the updates made to the computer program includes identifying a second file system or database from which the modified computer program receives data records.

10. Modifying the computer program comprises: deleting a first data source component of the computer program corresponding to the first file system or database; inserting a second data source component corresponding to the second file system or database; 10. The method of claim 9, comprising:

11. The method of claim 10, wherein the data target comprises a first file system or a database; the characterization of the updates made to the computer program includes identifying a set of multiple second file systems or databases to which the modified computer program outputs data records; The method of claim 1 , wherein at least one of the second file systems or databases is in a different location than the location of at least one other of the second file systems or databases.

12. Modifying the computer program comprises: replicating a flow within a computer program connecting a data processing component of the computer program to a first data target component corresponding to the first file system or database; inserting a new component corresponding to one of the second file system or database, the replicated flow connecting the data processing component to the new component; The method of claim 11 , comprising:

13. At least one of the second file systems or databases is a cloud-based file system or database; The characterization of the update includes: Identifying a first characteristic of the data stored in the non-cloud-based storage location; Identifying a second characteristic of the data stored in the second cloud-based file system or database; or Both The method of claim 11 , comprising:

14. Analyzing the computer program includes performing a data lineage analysis; The method comprises: a first component configured to receive or output data records having said first characteristic; a second component configured to receive or output data records having said second characteristic; or Both The method of claim 13, comprising identifying:

15. The method of claim 13 , wherein the first characteristic comprises personally identifiable information (PII).

16. 14. The method of claim 13, wherein modifying the computer program comprises modifying a specification of a first data processing component that outputs data having the first characteristic, modifying a specification of a second data processing component that outputs data having the second characteristic, or both.

17. A method for updating a computer program, the method being performed by a computing system, comprising: reading from a data storage device a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output data records including the processed data to a data target; analyzing, by one or more processors, the retrieved computer program and storing in a data storage device a characterization of the lineage, architecture, and behavior of the computer program; the lineage of the computer program includes static or runtime dependencies between elements of the computer program, static or runtime dependencies between the computer program and other computer programs, or both; the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process the data contained in the data records; the operations of the computer program include processes of the computer program being executed to process the data from the data records; and reading from a data storage device a characterization of an update to be made to the computer program, wherein when the computer program is modified according to the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system; storing a modified computer program in a data storage device, the modified computer program being generated by modifying the computer program to implement the update, the modifying the computer program including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the operation of the computer program, the modifying the computer program including identifying a data processing component of the computer program that implements a first type of file transfer protocol and modifying a specification of the data processing component in accordance with a second type of file transfer protocol, and the characterization of the update including an indication of a change from the first type of file transfer protocol to the second type of file transfer protocol; Migrating the modified portion of the computer program to the second computing system; executing the modified computer program, including executing portions of the modified computer program by the second computing system; A method comprising:

18. The method of claim 17, wherein modifying the specifications of the data processing component includes changing a value or expression for each of one or more parameters of the data processing component.

19. Analyzing the computer program includes identifying a data processing component configured to receive a first data record having one or more fields containing PII; modifying the computer program includes adding a component configured to implement a tokenization service upstream of the identified data processing component; 2. The method of claim 1, wherein the tokenization service is configured to receive the first data record and generalize the PII contained in the fields of the received record.

20. The method of claim 19, wherein modifying the computer program includes modifying the specifications of the identified data processing component to change the definition of a record format of a data record processed by the identified data processing component.

21. The method of claim 20, wherein the second computing system is a cloud-based computing system, and modifying the computer program includes designating a non-cloud-based computing system for execution of the tokenization service.

22. Testing at least a portion of the modified computer program, said testing comprising: providing input test data records to said at least a portion of said modified computer program; obtaining a first processed data record from the at least part of the modified computer program; testing at least a portion of the computer program, the at least a portion of the computer program corresponding to the tested portion of the modified computer program, and testing the at least a portion of the computer program includes: providing said input test data records to said at least a portion of said computer program; obtaining a second processed data record from said at least a portion of said computer program; comparing the first processed data record with the second processed data record; The method of claim 1 , comprising:

23. Modifying the computer program comprises: identifying a data processing component of the computer program having an attribute value that matches a target attribute value indicated by the characterization of the update; replacing the identified data processing component with a new data processing component; The method of claim 1 , comprising:

24. Identifying a data processing component of the computer program having an attribute value that matches a target attribute value indicated by the characterization of the update, comprising: generating a first set of data records, the data record corresponding to each data processing component of the computer program, each data record including an identifier of the respective data processing component and an attribute value of an attribute of the respective data processing component; filtering the first set of data records based on the attribute values ​​included in the data records of the first set to obtain a second set of data records, wherein said filtering includes removing the data records of the first set that do not contain values ​​for the particular attribute that match the target attribute value indicated by the characterization of the update; 24. The method of claim 23, comprising:

25. A computer system for updating a computer program, the computer system including one or more processors coupled to a memory, the one or more processors and the memory: reading from a data storage device a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output data records including the processed data to a data target; analyzing, by one or more processors, the retrieved computer program and storing in a data storage device a characterization of the lineage, architecture, and behavior of the computer program; the lineage of the computer program includes static or runtime dependencies between elements of the computer program, static or runtime dependencies between the computer program and other computer programs, or both; the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process the data contained in the data records; the operations of the computer program include processes of the computer program being executed to process the data from the data records; and retrieving from a data storage device a characterization of an update to be made to the computer program, wherein when the computer program is modified according to the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system, and the characterization of the update includes a requirement for anonymization of person identifiable information (PII) in data stored or processed by the second computing system; storing the modified computer program in a data storage device, the modified computer program being generated by modifying the computer program to implement the update, the modification of the computer program including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the operation of the computer program; Migrating the modified portion of the computer program to the second computing system; executing the modified computer program, including executing portions of the modified computer program by the second computing system; 1. A computer system configured to perform operations including:

26. 1. A non-transitory computer-readable storage medium storing instructions for causing a computing system to perform operations for updating a computer program, the non-transitory computer-readable storage medium comprising: The operation is reading from a data storage device a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output data records including the processed data to a data target; analyzing, by one or more processors, the retrieved computer program and storing in a data storage device a characterization of the lineage, architecture, and behavior of the computer program; the lineage of the computer program includes static or runtime dependencies between elements of the computer program, static or runtime dependencies between the computer program and other computer programs, or both; the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process the data contained in the data records; the operations of the computer program include processes of the computer program being executed to process the data from the data records; and retrieving from a data storage device a characterization of an update to be made to the computer program, wherein when the computer program is modified according to the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system, and the characterization of the update includes a requirement for anonymization of person identifiable information (PII) in data stored or processed by the second computing system; storing the modified computer program in a data storage device, the modified computer program being generated by modifying the computer program to implement the update, the modification of the computer program including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the operation of the computer program; Migrating the modified portion of the computer program to the second computing system; executing the modified computer program, including executing portions of the modified computer program by the second computing system; 1. A non-transitory computer-readable storage medium comprising:

27. The method of claim 1, wherein the family of the computer program is defined by one or more parameters associated with the computer program.

28. The method of claim 27, wherein the static dependency is defined by one or more parameters of a stored parameter set associated with the computer program.

29. The method of claim 27, wherein the runtime dependencies are defined by parameters indicated by a runtime log generated during execution of the computer program.

30. A non-transitory computer-readable storage medium storing instructions for causing a computing system to perform operations for updating a computer program, comprising: The operation is reading from a data storage device a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output data records including the processed data to a data target; analyzing, by one or more processors, the retrieved computer program and storing in a data storage device a characterization of the lineage, architecture, and behavior of the computer program; the lineage of the computer program includes static or runtime dependencies between elements of the computer program, static or runtime dependencies between the computer program and other computer programs, or both; the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process the data contained in the data records; the operations of the computer program include processes of the computer program being executed to process the data from the data records; and reading from a data storage device a characterization of an update to be made to the computer program, wherein when the computer program is modified in accordance with the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system, the characterization of the update including characterizing a distributed processing scheme for at least a portion of the computer program, and modifying the computer program including modifying a layout of the computer program to implement distribution of processing operations in accordance with the distributed processing scheme; storing the modified computer program in a data storage device, the modified computer program being generated by modifying the computer program to implement the update, the modification of the computer program including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the operation of the computer program; Migrating the modified portion of the computer program to the second computing system; executing the modified computer program, including executing portions of the modified computer program by the second computing system; 1. A computer-readable storage medium comprising:

31. A non-transitory computer-readable storage medium storing instructions for causing a computing system to perform operations for updating a computer program, comprising: The operation is reading from a data storage device a computer program hosted on and configured to be executed by a first computing system, the computer program being configured, when executed, to receive data records from a data source, process data contained in fields of the data records, and output data records including the processed data to a data target; analyzing, by one or more processors, the retrieved computer program and storing in a data storage device a characterization of the lineage, architecture, and behavior of the computer program; the lineage of the computer program includes static or runtime dependencies between elements of the computer program, static or runtime dependencies between the computer program and other computer programs, or both; the architecture of the computer program includes characteristics of the data source, characteristics of the data target, and characteristics of one or more processors configured to process the data contained in the data records; the operations of the computer program include processes of the computer program being executed to process the data from the data records; and reading from a data storage device a characterization of an update to be made to the computer program, wherein when the computer program is modified according to the update, at least a portion of the modified computer program is configured to be hosted on and executed by a second computing system; storing a modified computer program in a data storage device, the modified computer program being generated by modifying the computer program to implement the update, the modifying the computer program including modifying one or more of the lineage of the computer program, the architecture of the computer program, or the operation of the computer program, the modifying the computer program including identifying a data processing component of the computer program that implements a first type of file transfer protocol and modifying a specification of the data processing component in accordance with a second type of file transfer protocol, and the characterization of the update including an indication of a change from the first type of file transfer protocol to the second type of file transfer protocol; Migrating the modified portion of the computer program to the second computing system; executing the modified computer program, including executing portions of the modified computer program by the second computing system; 1. A computer-readable storage medium comprising:

Images