Method, device, electronic equipment and storage medium for constructing and maintaining data pipeline

Abstract

Provided are a method, device, electronic device, storage medium, and program product for constructing a data pipeline for a target data platform by a client. The method includes creating a data collection component, providing raw data collected by the data collection component to the client for a user to determine whether the raw data meets collection requirements, in response to receiving an indication from the client that the raw data meets the collection requirements, creating a data processing component for processing the raw data to obtain processed data, providing the processed data to the client for the user to determine whether the processed data meets processing requirements, and in response to receiving an indication from the client that the processed data meets the processing requirements, creating a data loading component for loading the processed data to the target data platform. Also provided are a method, device, electronic device, storage medium, and program product for maintaining a data pipeline for a target data platform by a client.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a method and device for constructing and maintaining a data pipeline for a target data platform by a client, electronic equipment, non-transitory computer readable storage medium and computer program product. BACKGROUND

[0002] A data pipeline (may also be referred to as an ETL pipeline) is a set of data processing procedures, which includes extracting data from a data source, transforming and cleaning the data, and loading the transformed and cleaned data to a target data platform. The use of data pipeline is very common for big data applications. In the construction phase of the data pipeline, the operations to be completed include defining a data model, writing data processing logic, interfacing with upstream and downstream systems (data source and target data platform), etc. The interfacing with the upstream and downstream systems and the data processing logic need to be repeatedly modified and verified. In the maintenance phase, when the data format changes, the data processing logic also needs to be adjusted accordingly.

[0003] The methods described in this section can not be the methods that have been previously conceived or adopted. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, nothing in this section should be assumed to be known or to be part of the prior art merely because of its inclusion in this section. SUMMARY

[0004] The present disclosure provides a method and device for constructing and maintaining a data pipeline for a target data platform by a client, electronic equipment, non-transitory computer readable storage medium and computer program product.

[0005] According to a first aspect of the present disclosure, a method for constructing a data pipeline for a target data platform by a client is provided, the data pipeline comprising a data collection component, a data processing component and a data loading component, the method comprising: creating the data collection component, the data collection component being configured to collect raw data; providing the raw data collected by the data collection component to the client for a user to determine whether the raw data meets a collection requirement; in response to receiving an indication from the client that the raw data meets the collection requirement, creating the data processing component, the data processing component being configured to process the raw data to obtain processed data; providing the processed data to the client for the user to determine whether the processed data meets a processing requirement; and in response to receiving an indication from the client that the processed data meets the processing requirement, creating the data loading component, the data loading component being configured to load the processed data to the target data platform.

[0006] According to a second aspect of the present disclosure, a method for maintaining a data pipeline for a target data platform by a client is provided, comprising: in response to receiving an indication from the client to modify an original data pipeline of the target data platform, constructing a temporary data pipeline, the temporary data pipeline comprising a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline, and a temporary data loading component, wherein the data collection component copy is configured to collect raw data, the data processing component copy is configured to process the raw data to obtain processed data, and the temporary data loading component is configured to discard the raw data and the processed data so that the raw data and the processed data do not enter the target data platform; in response to receiving third information from the client for modifying a configuration of the data collection component copy, modifying the configuration of the data collection component copy; providing the raw data collected by the modified data collection component copy to the client for a user to determine whether the raw data meets a collection requirement; in response to receiving an indication from the client that the raw data meets the collection requirement, and in response to receiving fourth information from the client for modifying a configuration of the data processing component copy, modifying the configuration of the data processing component copy; providing the processed data processed by the modified data processing component copy to the client for the user to determine whether the processed data meets a processing requirement; and in response to receiving an indication from the client that the processed data meets the processing requirement, creating a data loading component to replace the temporary data loading component and disabling the original data pipeline, the data loading component being configured to load the processed data to the target data platform.

[0007] According to a third aspect of the present disclosure, there is provided an apparatus for constructing, by a client, a data pipeline for a target data platform, the data pipeline comprising a data collection component, a data processing component, and a data loading component, the apparatus comprising: a first creation module configured to create the data collection component, the data collection component being configured to collect raw data; a first provision module configured to provide, to the client, the raw data collected by the data collection component for a user to determine whether the raw data meets a collection requirement; a second creation module configured to, in response to receiving an indication from the client that the raw data meets the collection requirement, create the data processing component, the data processing component being configured to process the raw data to obtain processed data; a second provision module configured to provide, to the client, the processed data for the user to determine whether the processed data meets a processing requirement; and a third creation module configured to, in response to receiving an indication from the client that the processed data meets the processing requirement, create the data loading component, the data loading component being configured to load the processed data to the target data platform.

[0008] According to a fourth aspect of the present disclosure, there is provided an apparatus for maintaining a data pipeline for a target data platform by a client, the apparatus comprising: a building module configured to, in response to receiving an indication from the client to modify an original data pipeline of the target data platform, build a temporary data pipeline, the temporary data pipeline comprising a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline, and a temporary data loading component, wherein the data collection component copy is configured to collect raw data, the data processing component copy is configured to process the raw data to obtain processed data, and the temporary data loading component is configured to discard the raw data and the processed data so that the raw data and the processed data do not enter the target data platform; a first modifying module configured to, in response to receiving third information from the client for modifying a configuration of the data collection component copy, modify the configuration of the data collection component copy; a third providing module configured to provide the raw data collected by the modified data collection component copy to the client for a user to determine whether the raw data meets a collection requirement; a second modifying module configured to, in response to receiving an indication from the client that the raw data meets the collection requirement, and in response to receiving fourth information from the client for modifying a configuration of the data processing component copy, modify the configuration of the data processing component copy; a fourth providing module configured to provide the processed data processed by the modified data collection processing component copy to the client for the user to determine whether the processed data meets a processing requirement; and a replacing module configured to, in response to receiving an indication from the client that the processed data meets the processing requirement, create a data loading component to replace the temporary data loading component and deactivate the original data pipeline, the data loading component being configured to load the processed data to the target data platform.

[0009] According to a fifth aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected with the at least one processor, the memory storing a computer program, the computer program causing the at least one processor to execute a method according to the present disclosure when executed by the at least one processor.

[0010] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing a computer program, the computer program causing a processor to execute a method according to the present disclosure when executed by the processor.

[0011] According to a seventh aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, causes the processor to perform a method according to the present disclosure.

[0012] According to one or more embodiments of the present disclosure, in the construction phase of the data pipeline, by providing the raw data collected by the data collection component to the client and providing the processed data processed by the data processing component to the client, it is possible to capture and view the processing results of the data collection component and the data processing component in real time during the construction of the data pipeline, and only after determining that the processing results of the data collection component and the data processing component meet the expectations, the data loading component is created to connect to the target data platform, thus improving the efficiency of constructing the data pipeline.

[0013] According to one or more embodiments of the present disclosure, in the maintenance phase of the data pipeline, by constructing a temporary data pipeline comprising a copy of the data collection component of the original data pipeline, a copy of the data processing component of the original data pipeline and a temporary data loading component, the processing results of the modified copy of the data collection component and the modified data processing component are captured and viewed in real time, and the temporary data pipeline after debugging is replaced by the original data pipeline to connect to the target data platform, thus, without affecting the operation of the original data pipeline, the efficiency of maintaining the data pipeline is improved.

[0014] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the claims. In all the drawings, like reference numerals refer to like parts throughout the several views.

[0016] Figure 1 A schematic diagram of an example system in which the various methods described herein can be implemented according to embodiments of the present disclosure is shown;

[0017] Figure 2 An exemplary flowchart of a method of constructing a data pipeline for a target data platform by a client according to embodiments of the present disclosure is shown;

[0018] Figures 3 to 5 A schematic diagram of a method of constructing a data pipeline for a target data platform by a client according to embodiments of the present disclosure is shown;

[0019] Figure 6An exemplary flowchart of a method of maintaining a data pipeline for a target data platform by a client is shown according to an embodiment of the present disclosure;

[0020] Figures 7 to 10 An exemplary flowchart of a method of maintaining a data pipeline for a target data platform by a client is shown according to an embodiment of the present disclosure;

[0021] Figure 11 An exemplary block diagram of an apparatus of constructing a data pipeline for a target data platform by a client is shown according to an embodiment of the present disclosure;

[0022] Figure 12 An exemplary block diagram of an apparatus of maintaining a data pipeline for a target data platform by a client is shown according to an embodiment of the present disclosure; and

[0023] Figure 13 An exemplary block diagram of an apparatus of maintaining a data pipeline for a target data platform by a client is shown according to an embodiment of the present disclosure; and DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are provided for the purpose of illustration only. The details of various embodiments of the present disclosure are described herein for the purpose of illustrating the present disclosure, and it should be understood that the various embodiments described herein are intended to be exemplary only. Thus, various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Also, the description set forth herein with respect to the various embodiments describes the features of the present disclosure in the context of particular embodiments. However, these descriptions serve merely as examples, and none of these descriptions should be construed as limiting the scope of the present disclosure. Furthermore, since numerous modifications and changes can be made to the order of operations, and processes described herein without departing from the scope of the present disclosure, it should understood that there are many variations that will be apparent to those having ordinary skill in the art.

[0025] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only for the purpose of distinguishing one element from another, and are not intended to limit the positions, sequence, or importance of the elements. In some examples, a first element and a second element can refer to the same instance of the element, and in some cases, they can refer to different instances of the element based on the context of the description.

[0026] The terminology used in the description of the various described examples herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Unless specifically defined herein, any terms used herein that are not specifically defined are intended to be given their ordinary meaning to a person of ordinary skill in the art. As used herein, the term "plurality" is intended to mean two or more, and the term "based on" is intended to be interpreted as "based, at least in part, on." Furthermore, the terms "and / or" and "at least one of" are intended to encompass any and all possible combinations of the listed items, and the term "coupled" is intended to mean directly or indirectly connected.

[0027] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user information comply with relevant laws and regulations and do not violate public order and good customs.

[0028] Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings.

[0029] Figure 1 FIG. 1 is a schematic diagram illustrating an example system 100 in which various methods described herein can be implemented, according to example embodiments.

[0030] With reference to Figure 1 The system 100 includes a client device 110, a server 120, and a network 130 communicatively coupling the client device 110 and the server 120.

[0031] The client device 110 includes a display 114 and a client application (APP) 112 that can be displayed via the display 114. The client application 112 can be an application program that needs to be downloaded and installed before running or a lite app as a lightweight application program. In the case that the client application 112 is an application program that needs to be downloaded and installed before running, the client application 112 can be pre-installed on the client device 110 and activated. In the case that the client application 112 is a lite app, the user 102 can directly run the client application 112 on the client device 110 without installing the client application 112 by searching for the client application 112 (e.g., by the name of the client application 112, etc.) in a host application or scanning a graphic code (e.g., a bar code, a QR code, etc.) of the client application 112, etc. In some embodiments, the client device 110 can be any type of mobile computer device, including a mobile computer, a mobile phone, a wearable computer device (e.g., a smart watch, a smart glass, etc. a head-mounted device), or other types of mobile devices. In some embodiments, the client device 110 can alternatively be a stationary computer device, such as a desktop computer, a server computer, or other types of stationary computer devices.

[0032] The server 120 is typically a server deployed by an Internet Service Provider (ISP) or an Internet Content Provider (ICP). The server 120 can represent a single server, a cluster of multiple servers, a distributed system, or a cloud server providing basic cloud services such as cloud database, cloud computing, cloud storage, cloud communication. It will be understood that, although Figure 1 Although the server 120 is shown in FIG. 1 to be in communication with only one client device 110, the server 120 can provide background services for multiple client devices at the same time.

[0033] Examples of network 130 include local area networks (LANs), wide area networks (WANs), personal area networks (PAN s), and / or combinations of these and other communications networks, such as the Internet. Network 130 can be a wired or wireless network. In some embodiments, data exchanged over network 130 is handled using technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. Furthermore, encryption technologies such as the secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc. can be used to encrypt all or some of the links between nodes in the network. In some embodiments, custom and / or proprietary data communications technologies and / or protocols can be used in place of or in combination with the above technologies.

[0034] For the purpose of embodiments of the present disclosure, in Figure 1 In an example, client application 112 can be an application that builds and maintains a data pipeline, which can provide various functions based on building and maintaining a data pipeline. Accordingly, server 120 can be a server used with the application that builds and maintains a data pipeline. The server 120 can provide services such as building and maintaining a data pipeline to client application 112 running in client device 110. Alternatively, services such as building and maintaining a data pipeline can also be provided by client application 112 running in client device 110.

[0035] Figure 1 System 100 of FIG. 1 can be configured and operated in various ways to enable application of various methods and apparatuses described in accordance with the present disclosure.

[0036] Figure 2 An example flowchart of a method 200 of building a data pipeline for a target data platform by a client according to embodiments of the present disclosure is shown. Method 200 can be performed at a client device (e.g., client device 110 shown in FIG. 1), i.e., the execution subject of each step of method 200 can be client device 110 shown in FIG. 1. Figure 1 In some embodiments, method 200 can be performed at a server (e.g., server 120 shown in FIG. 1). In some embodiments, method 200 can be performed by a combination of a client device (e.g., client device 110) and a server (e.g., server 120). In the following, each step of method 200 is described in detail with the execution subject being client device 110 as an example. Figure 1 Figure 1

[0037] ​​According to some embodiments of the present disclosure, in the construction phase of the data pipeline, by providing the raw data collected by the data collection component to the client and providing the processed data processed by the data processing component to the client, it is possible to capture and view the processing results of the data collection component and the data processing component in real time during the construction of the data pipeline, and only after determining that the processing results of the data collection component and the data processing component meet the expectations, the data loading component is created to connect to the target data platform, so that the processing results of the data collection component and the data processing component do not need to be viewed after the data pipeline is constructed, thereby improving the efficiency of constructing the data pipeline.

[0038] As shown in Figure 2 According to an embodiment of the present disclosure, a method 200 for constructing a data pipeline for a target data platform by a client is provided. The data pipeline includes a data collection component, a data processing component, and a data loading component. The method 200 includes the following steps 210 to 250.

[0039] Step 210, creating a data collection component, which is used to collect raw data.

[0040] In some embodiments of the present disclosure, the data collection component can be a source component used to construct the data pipeline, which is responsible for interfacing with a data source (e.g., a target data file or a target database).

[0041] Step 220, providing the raw data collected by the data collection component to the client for the user to determine whether the raw data meets the collection requirements (e.g., whether the collected raw data is the data that the user expects to obtain).

[0042] Step 230, in response to receiving an indication from the client that the raw data meets the collection requirements, creating a data processing component, which is used to process the raw data (e.g., adding or modifying some fields in the raw data) to obtain processed data.

[0043] In some embodiments of the present disclosure, the data processing component can be a pipeline component used to construct the data pipeline, which contains data processing logic and is responsible for processing the raw data collected by the data collection component.

[0044] In some embodiments of the present disclosure, the indication that the raw data meets the collection requirements can be explicit, for example, the user can select an option that the raw data meets the collection requirements in the judgment options or the user can input a confirmation instruction that the raw data meets the collection requirements.

[0045] In some embodiments of the present disclosure, the indication that the raw data meets the collection requirement can be implicit, for example, the user can directly create the data processing component without making a judgment or giving a confirmation instruction on whether the raw data meets the collection requirement. Since the user directly creates the data processing component, it can be inferred that the raw data meets the collection requirement.

[0046] In some embodiments of the present disclosure, the method 200 can further include receiving first information for modifying the configuration of the data collection component from the client, and modifying the configuration of the data collection component based on the first information, so that the raw data collected by the modified data collection component meets the collection requirement. The first information can include an instruction for modifying the configuration of the data collection component. In some embodiments of the present disclosure, the first information can be passive, for example, when the raw data is determined not to meet the collection requirement, the user initiates the first information to modify the data collection component. In some embodiments of the present disclosure, the first information can be active, for example, when the raw data is determined to meet the collection requirement, the user initiates the first information to modify the data collection component, thereby optimizing the data collection component.

[0047] According to some embodiments of the present disclosure, in the construction phase of the data pipeline, the data collection result of the data collection component in the data pipeline is quickly verified, and only after confirming that the behavior of the data collection component meets the expectation, the subsequent operation of the data pipeline construction is performed, thereby improving the efficiency of constructing the data pipeline.

[0048] Step 240, providing the processing data to the client for the user to determine whether the processing data meets the processing requirement (for example, whether the data processing logic is appropriate, whether the raw data is processed as expected by the user, etc.).

[0049] Step 250, in response to receiving an indication from the client that the processing data meets the processing requirement, creating a data loading component, the data loading component being used to load the processing data to the target data platform.

[0050] In some embodiments of the present disclosure, the data loading component can be a sink component used to construct the data pipeline, which is responsible for loading the processing data to the target data platform. After the data loading component is created, the operation of constructing the data pipeline for the target data platform is completed, and the data pipeline is able to interface the data source and the target data platform.

[0051] In some embodiments of the present disclosure, the indication that the processing data meets the processing requirement can be explicit, for example, the user can select an option that the processing data meets the processing requirement in the judgment options or the user can input a confirmation instruction that the processing data meets the processing requirement.

[0052] In some embodiments of the present disclosure, the indication that the processing data satisfies the processing requirement can be implicit, for example, the user can directly create the data loading component without making a judgment or giving a confirmation instruction on whether the processing data satisfies the processing requirement. Since the user directly creates the data loading component, it can be inferred that the processing data satisfies the processing requirement.

[0053] In some embodiments of the present disclosure, the method 200 can further include receiving, from the client, second information for modifying the configuration of the data processing component, and modifying the configuration of the data processing component based on the second information, so that the processing data processed via the modified data processing component satisfies the processing requirement. The second information can include an instruction to modify the configuration of the data processing component. In some embodiments of the present disclosure, the second information can be passive, for example, when the processing data is determined not to satisfy the processing requirement, the user initiates the second information to modify the data processing component. In some embodiments of the present disclosure, the second information can be active, for example, when the processing data is determined to satisfy the processing requirement, the user initiates the second information to modify the data processing component, thereby optimizing the data processing component.

[0054] According to some embodiments of the present disclosure, in the construction phase of the data pipeline, the data processing result of the data processing component in the data pipeline is quickly verified, and only after confirming that the behavior of the data processing component meets the expectation, the subsequent operation of the data pipeline construction is performed, thereby improving the efficiency of constructing the data pipeline.

[0055] In some embodiments of the present disclosure, the method 200 can further include, before creating the data loading component, invoking a temporary data loading component for discarding the raw data and the processing data so as to prevent the raw data and the processing data from entering the target data platform. The temporary data loading component may, for example, be a built-in / dev / null sink component that automatically discards all data received thereby, preventing dirty data from entering the target data platform, thereby improving the efficiency of constructing the data pipeline.

[0056] In some embodiments of the present disclosure, the data pipeline can be constructed via a Web UI. By using the Web UI, it is possible to quickly obtain data on the webpage, clean and process the data on the webpage, and then load it to the target data platform.

[0057] Figures 3 to 5 A schematic diagram of a method for constructing a data pipeline for a target data platform by a client according to an embodiment of the present disclosure is shown. The following refers to Figures 3 to 5 The method for constructing a data pipeline of the present disclosure is further described.

[0058] Reference is made to Figure 3As shown in the dashed box, first, a data collection component is created. The data collection component can obtain raw data from a target file or a target database. After the data collection component collects the raw data, the raw data is provided to the client for a user to determine whether the raw data meets the collection requirement. If the raw data does not meet the collection requirement, the user can provide instructions to the client to modify the data collection component until the raw data collected by the modified data collection component meets the collection requirement. If the raw data meets the collection requirement, the user can still provide instructions to the client to modify the data collection component, so as to optimize the data collection component. Alternatively, the user can provide an indication (explicitly or implicitly) to the client that the raw data meets the collection requirement, and in response to receiving the indication, the configuration of the data collection component is saved and a data processing component is created, which includes data processing logic and is used to process the raw data to obtain processed data. Since the internal temporary data loading component is invoked in this process, the raw data collected by the data collection component will be discarded to prevent dirty data from entering the target data platform.

[0059] Next, reference is made to Figure 4 As shown in the dashed box, after the data processing component processes the raw data to obtain processed data, the processed data is provided to the client for a user to determine whether the processed data meets the processing requirement. If the processed data does not meet the processing requirement, the user can provide instructions to the client to modify the data processing component until the processed data processed by the modified data processing component meets the processing requirement. If the processed data meets the processing requirement, the user can still provide instructions to the client to modify the data processing component, so as to optimize the data processing component. Alternatively, the user can provide an indication (explicitly or implicitly) to the client that the processed data meets the processing requirement, and in response to receiving the indication, the configuration of the data processing component is saved and a data loading component (as shown in Figure 5 Since the internal temporary data loading component is invoked before the data loading component is created, the raw data processed by the data processing component will be discarded to prevent dirty data from entering the target data platform.

[0060] Reference is made to Figures 3 to 5 The method of building a data pipeline described above enables quick verification of the data processing results of each link in the data pipeline, and automatically isolates the data pipeline under debugging from the target data platform, thereby avoiding the risk of dirty data being loaded into the target data platform and improving the efficiency of building the data pipeline.

[0061] Figure 6An example flowchart of a method 600 of maintaining a data pipeline for a target data platform by a client according to embodiments of the present disclosure is shown. The method 600 can be performed at a client device (e.g., the client device 110 shown in Figure 1 FIG. 1), i.e., the subject performing each step of the method 600 can be the client device 110 shown in Figure 1 FIG. 1. In some embodiments, the method 600 can be performed at a server (e.g., the server 120 shown in Figure 1 FIG. 1). In some embodiments, the method 600 can be performed by a combination of a client device (e.g., the client device 110) and a server (e.g., the server 120). In the following, each step of the method 600 is described in detail with the subject performing the client device 110 as an example.

[0062] According to some embodiments of the present disclosure, in the maintenance phase of the data pipeline, by constructing a temporary data pipeline including a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline, and a temporary data loading component, the modified data collection component copy and the processing result of the modified data processing component are captured and viewed in real time, and the temporary data pipeline after debugging is replaced by the original data pipeline to connect to the target data platform, so that the efficiency of maintaining the data pipeline can be improved without affecting the operation of the original data pipeline.

[0063] As shown in Figure 6 According to embodiments of the present disclosure, a method 600 of maintaining a data pipeline for a target data platform by a client is provided, including the following steps 610 to 660.

[0064] Step 610, in response to receiving an indication from a client to modify an original data pipeline of a target data platform, a temporary data pipeline is constructed, the temporary data pipeline including a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline, and a temporary data loading component. The data collection component copy is used to collect raw data, the data processing component copy is used to process the raw data to obtain processed data, and the temporary data loading component is used to discard the raw data and the processed data so as to prevent the raw data and the processed data from entering the target data platform.

[0065] In some embodiments of the present disclosure, the data collection component copy can be a source component copy of the original data pipeline, the data processing component copy can be a pipeline component copy of the original data pipeline, and the temporary data loading component can be a built-in / dev / null sink component.

[0066] Step 620, in response to receiving third information for modifying the configuration of the data collection component copy from the client, the configuration of the data collection component copy is modified.

[0067] In some embodiments of the present disclosure, the third information can include instructions for modifying the configuration of the data collection component copy.

[0068] At step 630, the original data collected by the modified data collection component copy is provided to the client for the user to determine whether the original data meets the collection requirement (e.g., whether the original data collected by the modified data collection component copy is the data that the user intended to obtain).

[0069] At step 640, in response to receiving an indication from the client that the original data meets the collection requirement, and in response to receiving fourth information from the client for modifying the configuration of the data processing component copy, the configuration of the data processing component copy is modified. The fourth information can include instructions for modifying the configuration of the data processing component copy.

[0070] In some embodiments of the present disclosure, the indication that the original data meets the collection requirement can be explicit, for example, the user can select an option in a judgment option that the original data meets the collection requirement or the user can input a confirmation instruction that the original data meets the collection requirement.

[0071] In some embodiments of the present disclosure, the indication that the original data meets the collection requirement can be implicit, for example, the user can directly modify the data processing component copy without making a judgment or giving a confirmation instruction on whether the original data meets the collection requirement. Since the user directly modifies the data processing component copy, it can be inferred that the original data meets the collection requirement.

[0072] In some embodiments of the present disclosure, the method 600 can further include receiving fifth information from the client for re-modifying the configuration of the data collection component copy, and based on the fifth information, re-modifying the configuration of the data collection component copy so that the original data collected by the re-modified data collection component copy meets the collection requirement. The fifth information can include instructions for modifying the configuration of the data collection component copy. It should be understood that the data collection component copy can be repeatedly modified based on the received fifth information, so that the behavior of the modified data collection component copy is more in line with expectations. In some embodiments of the present disclosure, the fifth information can be passive, for example, when the original data is determined not to meet the collection requirement, the user initiates the fifth information to modify the data collection component copy. In some embodiments of the present disclosure, the fifth information can be active, for example, when the original data is determined to meet the collection requirement, the user initiates the fifth information to modify the data collection component copy, thereby optimizing the data collection component copy.

[0073] According to some embodiments of the present disclosure, during the maintenance phase of the data pipeline, the modified data collection component copy is used to view and verify the data collection result of the modified data collection component copy in real time, and only after confirming that the data collection result meets the expectation, the subsequent operation of the data pipeline maintenance is performed, thereby improving the efficiency of maintaining the data pipeline.

[0074] At step 650, the processed data processed via the modified data processing component copy is provided to the client for the user to determine whether the processed data meets the processing requirement (e.g., whether the data processing logic is appropriate, whether the original data is processed as expected by the user, etc.).

[0075] At step 660, in response to receiving an indication from the client that the processed data meets the processing requirement, a data loading component is created to replace the temporary data loading component and the original data pipeline is disabled, the data loading component being used to load the processed data to the target data platform.

[0076] In some embodiments of the present disclosure, the indication that the processed data meets the processing requirement can be explicit, for example, the user can select an option in the judgment option that the processed data meets the processing requirement or the user can input a confirmation instruction that the processed data meets the processing requirement.

[0077] In some embodiments of the present disclosure, the indication that the processed data meets the processing requirement can be implicit, for example, the user can directly create a data loading component to replace the temporary data loading component without making a judgment or giving a confirmation instruction on whether the processed data meets the processing requirement. Since the user directly creates the data loading component to replace the temporary data loading component, it can be inferred that the processed data meets the processing requirement.

[0078] In some embodiments of the present disclosure, the method 600 can further include receiving sixth information for re-modifying the configuration of the data processing component copy from the client, and re-modifying the configuration of the data processing component copy based on the sixth information, so that the processed data processed via the re-modified data processing component copy meets the processing requirement. The sixth information can include an instruction for modifying the configuration of the data processing component copy. It should be understood that the data processing component copy can be repeatedly modified based on the received sixth information, so that the behavior of the modified data processing component copy is more in line with the expectation. In some embodiments of the present disclosure, the sixth information can be passive, for example, when the processed data is determined not to meet the processing requirement, the user initiates the sixth information to modify the data processing component copy. In some embodiments of the present disclosure, the sixth information can be active, for example, when the processed data is determined to meet the processing requirement, the user initiates the sixth information to modify the data processing component copy, thereby optimizing the data processing component copy.

[0079] According to some embodiments of the present disclosure, during the construction phase of the data pipeline, the modified data processing component copy is used to view and verify the data processing result of the modified data processing component copy in real time, and only after confirming that the data processing result meets the expectation, the subsequent operation of maintaining the data pipeline is performed, thereby improving the efficiency of maintaining the data pipeline.

[0080] In some embodiments of the present disclosure, the data pipeline can be maintained via a Web UI. By using the Web UI, it is possible to quickly obtain the data on the webpage, clean and process the data on the webpage, and then load it to the target data platform.

[0081] Figures 7 to 10 A schematic diagram of a method of maintaining a data pipeline for a target data platform by a client according to an embodiment of the present disclosure is shown. For example, when the data format changes, the original data pipeline needs to be adjusted and maintained to adapt to the new data format.

[0082] Reference is made to Figure 7 When the data pipeline needs to be maintained, a temporary data pipeline can first be constructed according to the maintenance instruction provided by the user, which includes a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline, and a temporary data loading component.

[0083] Reference is made to Figure 8 As shown in the dashed box, the data collection component copy of the original data pipeline is modified. After the modified data collection component copy collects the original data, the original data is provided to the client for the user to determine whether the original data meets the collection requirement. If the original data does not meet the collection requirement, the user can provide an instruction to the client to continue modifying the data collection component copy until the original data collected by the modified data collection component copy meets the collection requirement. If the original data meets the collection requirement, the user can still provide an instruction to the client to modify the data collection component copy, thereby optimizing the data collection component copy. Or the user can provide an indication (explicit or implicit) that the original data meets the collection requirement to the client, and in response to receiving the indication, the configuration of the modified data collection component copy is saved. Since the internal temporary data loading component is called in the process, the original data collected by the modified data collection component copy will be discarded to prevent dirty data from entering the target data platform.

[0084] Reference is made to Figure 9As shown by the dashed box, the data processing component copy of the original data pipeline is modified. After the modified data processing component copy processes the original data to obtain processing data, the processing data is provided to the client for the user to determine whether the processing data meets the processing requirement. If the processing data does not meet the processing requirement, the user can provide an instruction to the client to continue modifying the data processing component copy until the processing data processed by the modified data processing component copy meets the processing requirement. If the processing data meets the processing requirement, the user can still provide an instruction to the client to modify the data processing component copy to optimize the data processing component copy. Alternatively, the user can provide an indication (explicit or implicit) that the processing data meets the processing requirement to the client, in response to receiving the indication, the configuration of the modified data processing component copy is saved, the data loading component is created to replace the temporary data loading component and the original data pipeline is disabled (as shown in Figure 10 Since the internal temporary data loading component is still invoked before the data loading component is created, the processing data processed by the modified data processing component copy will be discarded to prevent dirty data from entering the target data platform.

[0085] Reference is made to Figures 7 to 10 The method of constructing a data pipeline described above establishes a temporary data pipeline based on the original data pipeline, so that the maintenance process does not affect the operation of the original data pipeline. In addition, when debugging the temporary data pipeline, the data processing results of each link in the temporary data pipeline can be quickly verified, and the temporary data pipeline under debugging is automatically isolated from the target data platform, thereby avoiding the risk of dirty data being loaded into the target data platform and improving the efficiency of constructing the data pipeline.

[0086] Figure 11 A structural block diagram of an apparatus for constructing a data pipeline for a target data platform by a client according to an embodiment of the present disclosure is shown.

[0087] As Figure 11As shown, the embodiment according to the present disclosure provides an apparatus 1100 for constructing a data pipeline for a target data platform by a client. The data pipeline comprises a data collection component, a data processing component and a data loading component. The apparatus 1100 comprises: a first creating module 1110 configured to create the data collection component, the data collection component being used for collecting raw data; a first providing module 1120 configured to provide the raw data collected by the data collection component to the client for a user to determine whether the raw data meets a collection requirement; a second creating module 1130 configured to, in response to receiving an indication from the client that the raw data meets the collection requirement, create the data processing component, the data processing component being used for processing the raw data to obtain processed data; a second providing module 1140 configured to provide the processed data to the client for the user to determine whether the processed data meets a processing requirement; and a third creating module 1150 configured to, in response to receiving an indication from the client that the processed data meets the processing requirement, create the data loading component, the data loading component being used for loading the processed data to the target data platform.

[0088] Figure 12 A structural block diagram of an apparatus for maintaining a data pipeline for a target data platform by a client according to an embodiment of the present disclosure is shown.

[0089] As Figure 12As shown, the apparatus 1200 for maintaining a data pipeline for a target data platform by a client according to an embodiment of the present disclosure includes: a construction module 1210 configured to, in response to receiving an indication from the client to modify an original data pipeline of the target data platform, construct a temporary data pipeline, the temporary data pipeline including a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline, and a temporary data loading component. The data collection component copy is used to collect raw data, the data processing component copy is used to process the raw data to obtain processed data, and the temporary data loading component is used to discard the raw data and the processed data so as to prevent the raw data and the processed data from entering the target data platform; a first modification module 1220 configured to, in response to receiving third information from the client for modifying a configuration of the data collection component copy, modify the configuration of the data collection component copy; a third providing module 1230 configured to provide the raw data collected by the modified data collection component copy to the client for the user to determine whether the raw data meets a collection requirement; a second modification module 1240 configured to, in response to receiving an indication from the client that the raw data meets the collection requirement, and in response to receiving fourth information from the client for modifying a configuration of the data processing component copy, modify the configuration of the data processing component copy; a fourth providing module 1250 configured to provide the processed data processed by the modified data processing component copy to the client for the user to determine whether the processed data meets a processing requirement; and a replacement module 1260 configured to, in response to receiving an indication from the client that the processed data meets the processing requirement, create a data loading component to replace the temporary data loading component, the data loading component being used to load the processed data to the target data platform, and disable the original data pipeline.

[0090] While specific functionality is discussed above with reference to particular modules, it should be noted that the functionality of the various modules discussed herein can be split into multiple modules, and / or at least some functionality of multiple modules can be combined into a single module. A particular module discussed herein performing an action includes that particular module itself performing the action, or alternatively, that particular module invoking or otherwise accessing another component or module that performs the action (or that performs the action in conjunction with the particular module). Thus, a particular module performing an action can include the particular module itself performing the action, and / or another module that the particular module invokes or otherwise accesses to perform the action.

[0091] It should also be understood that various techniques described herein can be implemented in software, hardware, or a combination thereof. Figure 11 and Figure 12The various modules described can be implemented in hardware or in hardware combined with software and / or firmware. For example, the modules can be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, the modules can be implemented as hardware logic / circuitry. For example, in some embodiments, one or more of the modules 1110-1150 and 1210-1260 can be implemented together in a System on Chip (SoC). The SoC can include an integrated circuit chip (which includes one or more of a processor (e.g., a Central Processing Unit (CPU), a microcontroller, a microprocessor, a Digital Signal Processor (DSP), etc.), a memory, one or more communication interfaces, and / or other circuitry), and can optionally execute received program code and / or include embedded firmware to perform functions.

[0092] According to another aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory; wherein the memory stores instructions which, when executed by the processor, cause the processor to perform the method according to the present disclosure.

[0093] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing instructions is also provided, wherein the instructions, when executed by a processor, cause the processor to perform the method according to the present disclosure.

[0094] According to another aspect of the present disclosure, a computer program product is also provided, comprising instructions which, when executed by a processor, cause the processor to perform the method according to the present disclosure.

[0095] Referring now to Figure 13 A structural block diagram of an electronic device 1300 that can be employed as the electronic device 1300 of the present disclosure will now be described, which is an example of a hardware device that can be applied to aspects of the present disclosure. The electronic device can be different types of computer devices, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections, and relationships, and their functions, are meant only as examples, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0096] Figure 13 A block diagram of an electronic device according to embodiments of the present disclosure is shown. As Figure 13As shown, the electronic device 1300 can include at least one processor 1301, working memory 1302, I / O devices 1304, display device 1305, storage 1306, and communication interface 1307 that are capable of communicating with each other using a system bus 1303.

[0097] The processor 1301 can be a single processing unit or a plurality of processing units, all of which can include single or multiple computing cores or processing units. The processor 1301 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. The processor 1301 can be configured to retrieve and execute program code stored in the working memory 1302, the storage 1306, or other computer-readable media, such as program code of an operating system 1302a, program code of application programs 1302b, and the like.

[0098] The working memory 1302 and the storage 1306 are examples of computer-readable storage media for storing instructions that are executed by the processor 1301 to implement the various functionalities described previously. The working memory 1302 can include both volatile memory and nonvolatile memory (e.g., RAM, ROM, etc.). In addition, the storage 1306 can include hard drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD, DVD), storage arrays, network-attached storage, storage area networks, and the like. Both the working memory 1302 and the storage 1306 can be collectively referred to herein as memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by the processor 1301 as a specific machine configured to implement the operations and functions described in the examples herein.

[0099] The I / O devices 1304 can include input devices and / or output devices, which can be any type of devices that enable input and / or output of information by the electronic device 1300, and can include, but are not limited to, a mouse, a keyboard, a touchscreen, a trackpad, a trackball, a joystick, a microphone, and / or a remote control. The output devices can be any type of devices that present information, and can include, but are not limited to, including video / audio output terminals, a vibrator, and / or a printer.

[0100] The communication interface 1307 allows the electronic device 1300 to exchange information / data with other devices over computer networks, such as the Internet, and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a BluetoothTM Devices, 802.11 devices, Wi-Fi devices, WiMAX devices, cellular communication devices, and / or the like.

[0101] The application 1302b in the working register 1302 can be loaded to perform the various methods and processes described above. In some embodiments, portions or all of the computer program can be loaded onto the electronic device 1300 via the storage 1306 and / or the communications interface 1307. When the computer program is loaded and executed by the processor 1301, one or more steps of the data processing methods described above can be performed.

[0102] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0103] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable task scheduling apparatus to cause a function / operation specified in the flow diagrams and / or block diagrams to be implemented when the program code is executed by the processor or controller. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or a server.

[0104] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0106] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0107] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0108] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps recited in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0109] While embodiments or examples of this disclosure have been described with reference to the drawings, it is to be understood that the foregoing methods, systems, and devices are merely examples of embodiments or examples, and that the scope of the present disclosure is not to be limited by these embodiments or examples, but only by the claims as issued. Various elements from the embodiments or examples can be omitted or replaced by equivalents thereof. Furthermore, the steps can be performed in an order different from that described in the present disclosure. Further, the various elements from the embodiments or examples can be combined in various ways. It is important that many of the elements described herein can be replaced by equivalents that appear after the present disclosure.

Claims

1. A method of building a data pipeline for a target data platform by a client, the data pipeline comprising a data collection component, a data processing component and a data loading component, the method comprising: creating the data collection component, the data collection component being configured to collect raw data; providing the raw data collected by the data collection component to the client for a user to determine whether the raw data meets a collection requirement; in response to receiving an indication from the client that the raw data meets the collection requirement, creating the data processing component, the data processing component being configured to process the raw data to obtain processed data; providing the processed data to the client for the user to determine whether the processed data meets a processing requirement; and in response to receiving an indication from the client that the processed data meets the processing requirement, creating the data loading component, the data loading component being configured to load the processed data to the target data platform; wherein before creating the data loading component, a temporary data loading component is invoked, the temporary data loading component being configured to discard the raw data and the processed data so that the raw data and the processed data do not enter the target data platform. 2.The method of claim 1, further comprising: receiving first information from the client for modifying a configuration of the data collection component; and based on the first information, modifying the configuration of the data collection component such that raw data collected by the modified data collection component meets the collection requirement. 3.The method of claim 1, further comprising: receiving second information from the client for modifying a configuration of the data processing component; and based on the second information, modifying the configuration of the data processing component such that processed data processed by the modified data processing component meets the processing requirement. the data pipeline is built via a web UI. 5.A method of maintaining a data pipeline for a target data platform by a client, comprising: in response to receiving an indication from the client to modify an original data pipeline of the target data platform, building a temporary data pipeline, the temporary data pipeline comprising a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline and a temporary data loading component, wherein the data collection component copy is configured to collect raw data, the data processing component copy is configured to process the raw data to obtain processed data, and the temporary data loading component is configured to discard the raw data and the processed data so that the raw data and the processed data do not enter the target data platform; 4. The method of any one of claims 1 to 3, wherein, in response to receiving third information from the client for modifying a configuration of the data collection component copy, modifying the configuration of the data collection component copy; providing raw data collected by the modified data collection component copy to the client for a user to determine whether the raw data meets a collection requirement. ​ ​ ​ in response to receiving an indication from the client that the raw data satisfies the collection requirement, and in response to receiving fourth information from the client for modifying the configuration of the data processing component copy, modifying the configuration of the data processing component copy; providing processed data processed via the modified data processing component copy to the client for the user to determine whether the processed data satisfies a processing requirement; and in response to receiving an indication from the client that the processed data satisfies the processing requirement, creating a data loading component to replace the temporary data loading component and deactivate the original data pipeline, the data loading component being configured to load the processed data to the target data platform.

6. The method of claim 5, further comprising: receiving fifth information from the client for re-modifying the configuration of the data collection component copy; and based on the fifth information, re-modifying the configuration of the data collection component copy such that raw data collected by the re-modified data collection component copy satisfies the collection requirement.

7. The method of claim 5, further comprising: receiving sixth information from the client for re-modifying the configuration of the data processing component copy; and based on the sixth information, re-modifying the configuration of the data processing component copy such that processed data processed via the re-modified data processing component copy satisfies the processing requirement.

8. The method of any one of claims 5-7, wherein, the data pipeline is maintained via a Web UI.

9. An apparatus for constructing a data pipeline for a target data platform by a client, the data pipeline comprising a data collection component, a data processing component, and a data loading component, the apparatus comprising: a first creation module configured to create the data collection component, the data collection component being configured to collect raw data; a first providing module configured to provide raw data collected by the data collection component to the client for a user to determine whether the raw data satisfies a collection requirement; a second creation module configured to, in response to receiving an indication from the client that the raw data satisfies the collection requirement, create the data processing component, the data processing component being configured to process the raw data to obtain processed data; a second providing module configured to provide the processed data to the client for the user to determine whether the processed data satisfies a processing requirement; and a third creation module configured to, in response to receiving an indication from the client that the processed data satisfies the processing requirement, create the data loading component, the data loading component being configured to load the processed data to the target data platform, wherein, prior to creating the data loading component, a temporary data loading component is invoked, the temporary data loading component being configured to discard the raw data and the processed data from entering the target data platform.

10. An apparatus for maintaining a data pipeline for a target data platform by a client, the apparatus comprising: a building module configured to build a temporary data pipeline in response to receiving an indication from the client to modify an original data pipeline of the target data platform, the temporary data pipeline comprising a data collection component copy of the original data pipeline, a data processing component copy of the original data pipeline, and a temporary data loading component, wherein the data collection component copy is configured to collect raw data, the data processing component copy is configured to process the raw data to obtain processed data, and the temporary data loading component is configured to discard the raw data and the processed data from entering the target data platform; a first modifying module configured to modify a configuration of the data collection component copy in response to receiving third information from the client for modifying the configuration of the data collection component copy; a third providing module configured to provide the raw data collected by the modified data collection component copy to the client for a user to determine whether the raw data meets a collection requirement; a second modifying module configured to modify a configuration of the data processing component copy in response to receiving an indication from the client that the raw data meets the collection requirement and in response to receiving fourth information from the client for modifying the configuration of the data processing component copy; a fourth providing module configured to provide the processed data processed by the modified data collection processing copy to the client for the user to determine whether the processed data meets a processing requirement; and a replacing module configured to create a data loading component to replace the temporary data loading component and deactivate the original data pipeline in response to receiving an indication from the client that the processed data meets the processing requirement, the data loading component configured to load the processed data to the target data platform.

11. An electronic device, the electronic device comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program which, when executed by the at least one processor, implements the method according to any one of claims 1-8.

12. A non-transitory computer readable storage medium storing a computer program, wherein, The computer program, when executed by the processor, implements the method according to any one of claims 1-8.

13. A computer program product comprising a computer program, wherein, The computer program, when executed by the processor, implements the method according to any one of claims 1-8.

Images