Data synchronization method, device, equipment and storage medium based on NiFi

Abstract

The embodiment of the present application provides a data synchronization method, device, equipment and storage medium based on NiFi, which relates to the field of data synchronization technology and aims to reduce the increased complexity brought by the high degree of customization of NiFi, and reduce the repeated design and configuration of the same data synchronization scenario. The data synchronization method based on NiFi includes: according to the current data synchronization scenario, obtaining a process configuration parameter file; assigning a value to the process configuration parameter file to obtain an assigned process configuration parameter file; replacing the parameter value in the assigned process configuration parameter file with the process configuration file to obtain an assigned process configuration file, which represents the overall structure and configuration of the NiFi data synchronization process; and performing data synchronization based on the assigned process configuration file.

Description

Technical Field

[0001] The present application relates to the field of data synchronization technology, and in particular to a data synchronization method, apparatus, device and storage medium based on NiFi. Background Art

[0002] NiFi is a system for processing and distributing data. It provides a visual graphical process design interface, through which data synchronization processes can be designed and configured. NiFi supports a variety of processors for processing data, and different process arrangements can achieve the same data synchronization scenario.

[0003] Although the high degree of customizability is NiFi's advantage, too many customization options may increase the complexity of configuration, which may seem redundant to users. It takes a lot of time for users to master all the concepts, components, and workflows, and as the complexity of the process increases, the graphical interface in NiFi may become complicated and difficult to understand. Some large-scale and complex processes will become difficult to maintain and manage; and the same data synchronization scenario, for example, the data synchronization scenario between the same source and target database types, requires repeated design and configuration in NiFi.

[0004] Therefore, how to solve the increased complexity brought about by NiFi's high degree of customizability and the problem of repeated design and configuration for the same data synchronization scenarios is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, embodiments of the present application provide a data synchronization method, apparatus, device and storage medium based on NiFi to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a data synchronization method based on NiFi is disclosed, the method comprising: According to the current data synchronization scenario, obtain the process configuration parameter file; Assigning values ​​to the process configuration parameter file to obtain an assigned process configuration parameter file; Replace the parameter values ​​in the assignment process configuration parameter file with the process configuration file to obtain the assignment process configuration file, wherein the process configuration file represents the overall structure and configuration of the NiFi data synchronization process; Data synchronization is performed based on the assignment process configuration file.

[0007] Optionally, according to the current data synchronization scenario, obtain the process configuration parameter file, including: Determine whether there is a data synchronization scenario that is the same as the current data synchronization scenario; In the absence of a data synchronization scenario identical to the current data synchronization scenario, generating a process configuration parameter file based on the configured NiFi data synchronization process template process group; In the case that there is a data synchronization scenario that is the same as the current data synchronization scenario, the process configuration parameter file corresponding to the data synchronization scenario is directly called.

[0008] Optionally, in the absence of a data synchronization scenario identical to the current data synchronization scenario, a process configuration parameter file is generated based on the configured NiFi data synchronization process template process group, including: According to the current data synchronization scenario, configure the NiFi data synchronization process template process group; Export the process configuration file of the NiFi data synchronization process template process group; The data source parameters of the process configuration file are extracted, and a process configuration parameter file is generated according to the data source parameters, wherein the data source parameters include: a path of a source data source, a path of a target data source, and a data source parameter name.

[0009] Optionally, assigning a value to the process configuration parameter file to obtain an assigned process configuration parameter file includes: Determine the data source information according to the source data source and target data source for data synchronization; According to the data source information, each data source parameter in the process configuration parameter file is assigned a value to obtain an assigned process configuration parameter file.

[0010] Optionally, the parameter values ​​in the assignment process configuration parameter file are replaced into the process configuration file to obtain the assignment process configuration file, including: Parsing the process configuration file to generate a first object; Parsing the assignment process configuration parameter file to generate a second object; The first object is replaced by the second object to obtain an assignment process configuration file.

[0011] Optionally, performing data synchronization based on the assignment process configuration file includes: Using the assignment process configuration file as a request parameter, calling the NiFi flow definition upload application program interface to create a NiFi process group; Data synchronization between data sources is performed by running the NiFi process group.

[0012] In a second aspect of an embodiment of the present application, a data synchronization device based on NiFi is disclosed, the device comprising: The acquisition module is used to obtain the process configuration parameter file according to the current data synchronization scenario; An assignment module, used for assigning values ​​to the process configuration parameter file to obtain an assignment process configuration parameter file; A replacement module is used to replace the parameter values ​​in the assignment process configuration parameter file with the process configuration file to obtain the assignment process configuration file, wherein the process configuration file represents the overall structure and configuration of the NiFi data synchronization process; A synchronization module is used to synchronize data based on the assignment process configuration file.

[0013] A third aspect of an embodiment of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the NiFi-based data synchronization method described in the first aspect of the embodiment of the present application are implemented.

[0014] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the NiFi-based data synchronization method described in the first aspect of the embodiment of the present application are implemented.

[0015] In a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the NiFi-based data synchronization method described in the first aspect of the embodiment of the present application.

[0016] The embodiments of the present application include the following advantages: In an embodiment of the present application, according to the current data synchronization scenario, a process configuration parameter file is obtained; the process configuration parameter file is assigned to obtain an assigned process configuration parameter file; the parameter values ​​in the assigned process configuration parameter file are replaced with the process configuration file to obtain an assigned process configuration file, and the process configuration file represents the overall structure and configuration of the NiFi data synchronization process; data synchronization is performed based on the assigned process configuration file.

[0017] Since the process configuration parameter file is obtained according to the data synchronization scenario, and the assignment process configuration file is obtained by assignment, the assignment process configuration file contains the overall structure and configuration of the NiFi data synchronization process, so data synchronization can be quickly achieved based on the assignment process configuration file. In this way, for different data synchronization scenarios, data synchronization can be achieved without complex configuration; for the same data synchronization scenario, repeated configuration is not required, and data synchronization can be achieved by simply assigning the process configuration parameter file. This method solves the problems of increased complexity brought about by NiFi's high degree of customizability and repeated design and configuration for the same data synchronization scenario. It is applicable to various data synchronization scenarios based on NiFi and has universality.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 It is a flowchart of a data synchronization method based on NiFi provided in an embodiment of the present application; Figure 2 It is a flow chart of extracting data source parameters of a process configuration file to generate a process configuration parameter file provided by an embodiment of the present application; Figure 3 It is a flow chart provided by an embodiment of the present application for replacing parameter values ​​in an assignment process configuration parameter file with a process configuration file; Figure 4 is a data synchronization flow chart provided by an embodiment of the present application in the case where there is no data synchronization scenario that is the same as the current data synchronization scenario; Figure 5 is a data synchronization flow chart provided by an embodiment of the present application in the case where there is a data synchronization scenario that is the same as the current data synchronization scenario; Figure 6 It is a structural diagram of a data synchronization device based on NiFi provided in an embodiment of the present application; Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] In order to better understand the technical solution of this application, the technical concepts involved in this application are explained: NiFi: is a system for processing and distributing data. It provides a visual graphical process design interface, through which data synchronization processes can be designed and configured. NiFi provides highly customizable data stream processing capabilities and supports a variety of data processing processors. Different process orchestrations can achieve the same data synchronization scenario.

[0023] FlowDefinition (process configuration): The overall structure and configuration of the NiFi data flow, including the processors, connection relationships, input and output ports used in the data flow, that is, Flow Definition defines how the data flow flows in NiFi and the processing steps that the data goes through.

[0024] API: Application Programming Interface, application program interface.

[0025] NiFi provides processors that support multiple data processing, and different process orchestrations can achieve the same data synchronization scenario. Although the high degree of customizability is NiFi's advantage, too many customization options may increase the complexity of the configuration, which may seem redundant to users; it takes a lot of time for users to master all the concepts, components, and workflows, and as the complexity of the process increases, the graphical interface in NiFi may become complex and difficult to understand. Some large-scale and complex processes will become difficult to maintain and manage; and the same data synchronization scenario requires repeated design and configuration in NiFi.

[0026] Therefore, in order to solve the increased complexity brought by NiFi's high degree of customization and the problem of repeated design and configuration for the same data synchronization scenario, the technical concept of the embodiment of the present application is: obtain the process configuration parameter file according to the data synchronization scenario, assign values ​​to the process configuration parameter file, and obtain the process configuration file based on the assigned process configuration parameter file. The process configuration file represents the overall structure and configuration of the NiFi data synchronization process, so data synchronization can be quickly achieved based on the assigned process configuration file. In this way, for different data synchronization scenarios, data synchronization can be achieved without complex configuration; for the same data synchronization scenario, there is no need for repeated configuration, and data synchronization can be achieved by simply assigning values ​​to the process configuration parameter file.

[0027] The following is a detailed description of the NiFi-based data synchronization method in accordance with an embodiment of the present application in conjunction with the accompanying drawings.

[0028] Reference Figure 1 As shown, Figure 1This is a flowchart of a data synchronization method based on NiFi provided in an embodiment of the present application. Figure 1 As shown, a data synchronization method based on NiFi provided in an embodiment of the present application may include steps S110 to S140: Step S110: Obtain a process configuration parameter file according to the current data synchronization scenario.

[0029] In the embodiment of the present application, the data synchronization scenario refers to the data flow and synchronization between different data sources (systems / databases), and the data synchronization scenario includes real-time data stream synchronization, data synchronization scenarios between heterogeneous data sources, etc. For example, the data synchronization scenario can be offline synchronization of MySQL data to HDFS. Through NiFi, the data in MySQL can be imported into HDFS and compressed and stored. This data synchronization scenario is usually used for big data processing and analysis. For example, the data synchronization scenario can be real-time monitoring of MySQL binlog for data synchronization to Hive. Through NiFi, MySQL binlog can be monitored in real time, and the changed data can be synchronized to Hive in real time. This data synchronization scenario is used for real-time data analysis. For another example, the data synchronization scenario can be data synchronization for configuring Kafka. Through NiFi, data can be synchronized to Kafka to realize data synchronization and management in the message queue. Among them, MySQL is a relational database management system, HDFS is a distributed file system, MySQL binlog refers to the log file used in MySQL for all operations that modify the database, Hive is a data warehouse tool based on Hadoop, and Kafka is a high-throughput distributed message queue system that can handle a large amount of real-time data.

[0030] The process configuration parameter file refers to the FlowDefinition parameter file. The process configuration (FlowDefinition) refers to the overall structure and configuration of the NiFi data flow, including the processors, connection relationships, input and output ports used in the data synchronization process. That is, the process configuration defines how the data flow flows in NiFi and the processing steps that the data passes through. The process configuration parameter file contains the data source parameters corresponding to the overall structure and configuration of the NiFi data synchronization process. Specifically, the process configuration parameter file contains: the path of the source data source, the path of the target data source, and the data source parameter name and other data source parameters. In some embodiments, the process configuration parameter file is a file in JSON format.

[0031] The synchronization processes and data sources corresponding to different data synchronization scenarios may be different, that is, the process configuration parameter files corresponding to different data synchronization scenarios are different. Therefore, when performing data synchronization, obtain the process configuration parameter file according to the current data synchronization scenario.

[0032] Step S120: assigning values ​​to the process configuration parameter file to obtain an assigned process configuration parameter file.

[0033] In the embodiment of the present application, the process configuration parameter file contains the overall structure of the NiFi data synchronization process and the corresponding data source parameters of the configuration. By editing the process configuration parameter file, each data source parameter is assigned according to the data source information to obtain the assigned process configuration parameter file; wherein the data source information includes the data source port, IP address (Internet Protocol Address), password, account number, synchronized database, data table and other information.

[0034] For example, the data synchronization scenario can be offline synchronization of MySQL data to HDFS, and assigning values ​​to the process configuration parameter file means assigning the data source information of MySQL and HDFS (i.e., data source port, IP address, password, account, synchronized database, data table, etc.) to the process configuration parameter file. For another example, the data synchronization scenario can be real-time monitoring of MySQL binlog and data synchronization to Hive, and assigning values ​​to the process configuration parameter file means assigning the data source information of MySQL binlog and Hive to the process configuration parameter file.

[0035] Step S130: Replace the parameter values ​​in the assignment process configuration parameter file with the process configuration file to obtain the assignment process configuration file, which represents the overall structure and configuration of the NiFi data synchronization process.

[0036] In the embodiment of the present application, the process configuration file refers to a FlowDefinition file, which is a file in JSON format; the process configuration file represents the overall structure and configuration of the NiFi data synchronization process, including the processors, connection relationships, and input and output ports used in the data synchronization process, that is, the process configuration file defines how the data flow flows in NiFi and the processing steps that the data goes through.

[0037] The assignment process configuration parameter file contains all the data source information of the current data synchronization scenario. The data source information (parameter value) in the assignment process configuration parameter file is replaced with the process configuration file through the FlowDefinition parser to obtain the assignment process configuration file.

[0038] Step S140: performing data synchronization based on the assignment process configuration file.

[0039] In the embodiment of the present application, the assignment process configuration file is a JSON format file containing data source information, as well as the overall structure and configuration of the NiFi data synchronization process. After obtaining the assignment process configuration file, the assignment process configuration file can be used as a request parameter to synchronize data through NiFi.

[0040] In the above process, the process configuration parameter file is obtained according to the data synchronization scenario, and the assignment process configuration file is obtained by assignment. The assignment process configuration file contains the overall structure and configuration of the NiFi data synchronization process, so data synchronization can be quickly achieved based on the assignment process configuration file. In this way, for different data synchronization scenarios, data synchronization can be achieved without complex configuration; for the same data synchronization scenario, repeated configuration is not required, and data synchronization can be achieved by assigning the process configuration parameter file. This method solves the problems of increased complexity brought about by the high degree of customizability of NiFi and repeated design and configuration for the same data synchronization scenario. It is applicable to various data synchronization scenarios based on NiFi and has universality.

[0041] In combination with the above embodiments, in one implementation, the present application embodiment further provides a data synchronization method based on NiFi. In this method, the "acquiring a process configuration parameter file according to the current data synchronization scenario" in the above step S110 may specifically include steps S110-1 to S110-3: Step S110 - 1 : Determine whether there is a data synchronization scenario that is the same as the current data synchronization scenario.

[0042] Step S110 - 2 : In the absence of a data synchronization scenario identical to the current data synchronization scenario, a process configuration parameter file is generated based on the configured NiFi data synchronization process template process group.

[0043] Step S110 - 3 : when there is a data synchronization scenario that is the same as the current data synchronization scenario, directly call the process configuration parameter file corresponding to the data synchronization scenario.

[0044] In the embodiment of the present application, different data synchronization scenarios correspond to different process configuration parameter files. The process configuration parameter file contains the overall structure of the NiFi data synchronization process and the corresponding data source parameters. After the process configuration parameter file corresponding to a data synchronization scenario is created, NiFi can directly call the corresponding process configuration parameter file for data synchronization when performing data synchronization for the same data synchronization scenario in the subsequent process.

[0045] When performing data synchronization through NiFi, for the current data synchronization scenario, first execute step S110-1 to determine whether there is a data synchronization scenario that is the same as the current data synchronization scenario. If there is no data synchronization scenario that is the same as the current data synchronization scenario (wherein, different data synchronization scenarios refer to the same database type of the data source and / or different data synchronization operations), execute step S110-2 to generate a corresponding process configuration parameter file based on the configured NiFi data synchronization process template process group; wherein, the NiFi data synchronization process template process group includes all configuration information of the data synchronization process, such as the source data source, the target data source, and the operations performed by data synchronization.

[0046] When there is a data synchronization scenario that is identical to the current data synchronization scenario (wherein the same data synchronization scenario means that the database type of the data source is the same and the data synchronization operations are the same), it means that a process configuration parameter file corresponding to the data synchronization scenario has been created in advance. At this time, step S110-3 is executed to directly call the process configuration parameter file corresponding to the data synchronization scenario.

[0047] It can be understood that in order to be able to directly call the existing process configuration parameter file in the same data synchronization scenario, after generating the corresponding process configuration parameter file based on the configured NiFi data synchronization process template process group in step S110-2, the corresponding process configuration parameter file can be cached, and then when the data is synchronized for the same data synchronization scenario in the future, it can be directly called from the cache.

[0048] In this way, the process configuration parameter file corresponding to the current data synchronization scenario can be quickly obtained. For new data synchronization scenarios, the process configuration parameter file is obtained based on the NiFi data synchronization process template process group. There is no need to configure too many configuration items, thus reducing the complexity brought by NiFi's high customizability. For the same data synchronization scenario, the existing process configuration parameter file is directly called, thus solving the problem of repeated design and configuration of the same data synchronization scenario.

[0049] In a specific implementation, in the absence of a data synchronization scenario identical to the current data synchronization scenario, a process configuration parameter file is generated based on the configured NiFi data synchronization process template process group, including steps S110-2-1 to S110-2-3: Step S110-2-1: According to the current data synchronization scenario, configure the NiFi data synchronization process template process group.

[0050] Specifically, the NiFi data synchronization process template process group is configured according to the source data source, target data source, data synchronization operation and other information in the current data synchronization scenario. For example, if MySQL is the source data source and Oracle (a relational database management system) is the target data source, the NiFi data synchronization process template process group is configured based on MySQL and Oracle.

[0051] Step S110-2-2: Export the process configuration file of the NiFi data synchronization process template process group.

[0052] After obtaining the NiFi data synchronization process template, export the process configuration file (i.e., FlowDefinition file) of the NiFi data synchronization process template process group. The process configuration file is a file in JSON format, and the process configuration file represents the overall structure and configuration of the NiFi data synchronization process.

[0053] Step S110-2-3: extract the data source parameters of the process configuration file, and generate a process configuration parameter file according to the data source parameters, wherein the data source parameters include: the path of the source data source, the path of the target data source, and the data source parameter name.

[0054] The path of the source data source refers to the JSON PATH of the source data source, the path of the target data source refers to the JSON PATH of the target data source, and the data source parameter name refers to the FlowDefinition data source parameter name. For example, if the source data source and the target data source are MySQL and Oracle respectively, the JSON PATH of the MySQL source data source and Oracle target data source parameters in the process configuration file and the FlowDefinition data source parameter name are extracted.

[0055] like Figure 2 As shown, Figure 2 It is a flowchart of extracting data source parameters of a process configuration file to generate a process configuration parameter file provided by an embodiment of the present application. The process of extracting data source parameters of a process configuration file to generate a process configuration parameter file is as follows: first, extract the JSONPATH of the source data source and the target data source of the process configuration file (FlowDefinition file), then extract the data source parameter name of the process configuration file, and finally generate a process configuration parameter file in JSON format according to the JSONPATH of the source data source and the target data source, and the data source parameter name, that is, generate a process configuration parameter file (FlowDefinition parameter file).

[0056] Through the above process, the NiFi data synchronization process template process group is configured according to the current data synchronization scenario, and then the process configuration parameter file is generated; for users, there is no need to perform complex configurations, and there is no need to master all the concepts, components and workflows of NiFi. Only a simple NiFi data synchronization process template process group configuration is required according to the current data synchronization scenario to generate the process configuration parameter file. Therefore, there is no need to configure too many configuration items, which reduces the complexity brought by NiFi's high degree of customizability.

[0057] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data synchronization method based on NiFi. In this method, the "assigning the process configuration parameter file to obtain the assigned process configuration parameter file" in the above step S120 may specifically include steps S120-1 to S120-2: Step S120 - 1 : Determine data source information according to the source data source and the target data source for data synchronization.

[0058] Step S120 - 2 : assigning values ​​to the various data source parameters in the process configuration parameter file according to the data source information to obtain an assigned process configuration parameter file.

[0059] In the embodiment of the present application, the source data source refers to the database (system) of outflow data, and the target data source refers to the database (system) of inflow data. NiFi synchronizes the data of the source data source to the target data source in a certain way. The data source information includes the data source port, IP address, password, account number, synchronized database, data table and other information. For the source data source and target data source that need to synchronize data, their respective data source information is determined respectively, and each data source parameter in the process configuration parameter file is assigned based on the data source information, and finally the assignment process configuration parameter file is obtained.

[0060] In this way, by assigning values ​​to the process configuration parameter file according to the data source information corresponding to the source data source and the target data source, an assignment process configuration file can be obtained. The assignment process configuration file contains the overall structure and configuration of the NiFi data synchronization process. Therefore, data synchronization can be quickly achieved based on the assignment process configuration file, which simplifies the operational process of data synchronization and reduces the complexity brought by NiFi's high degree of customizability. This data synchronization method has better versatility.

[0061] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a data synchronization method based on NiFi. In this method, the above step S130 of "replacing the parameter values ​​in the assignment process configuration parameter file with the process configuration file to obtain the assignment process configuration file" may specifically include steps S130-1 to S130-3: Step S130 - 1 : Parse the process configuration file to generate a first object.

[0062] Step S130 - 2 : Parse the assignment process configuration parameter file to generate a second object.

[0063] Step S130 - 3 : Use the second object to replace the first object to obtain an assignment process configuration file.

[0064] In an embodiment of the present application, a FlowDefinition parser is used to parse a flow configuration file (FlowDefinition file) to obtain a first object, which is a JSON object; an assignment flow configuration parameter file (assignment FlowDefinition parameter file) is parsed to obtain a second object, which is also a JSON object; and then, the attribute value in the first object is replaced according to the attribute value in the second object (i.e., JSON PATH) to obtain the assignment flow configuration file.

[0065] like Figure 3 As shown, Figure 3 It is a flowchart provided by an embodiment of the present application for replacing parameter values ​​in an assignment process configuration parameter file with a process configuration file. Specifically, the process of replacing parameter values ​​in an assignment process configuration parameter file with a process configuration file is as follows: parsing the process configuration file (FlowDefinition file) to generate a corresponding JSON object A (first object); secondly, parsing the assignment process configuration parameter file (FlowDefinition parameter file) to generate a corresponding JSON object B (second object); then, replacing the attribute value of the first object according to the attribute value of the second object in the assignment process configuration parameter file, that is, replacing the JSON PATH of the first object with the JSON PATH of the second object to obtain the assignment process configuration file; finally, passing the assignment process configuration file as a request parameter into the NiFi API (NiFi Application Programming Interface) to achieve data synchronization.

[0066] In combination with the above embodiments, in one implementation, the present application embodiment further provides a data synchronization method based on NiFi. In this method, the "data synchronization based on the assignment process configuration file" in the above step S140 may specifically include steps S140-1 to S140-2: Step S140 - 1 : Using the assignment process configuration file as a request parameter, calling the NiFi flow definition upload application program interface to create a NiFi process group.

[0067] Step S140 - 2 : Synchronize data between data sources by running the NiFi process group.

[0068] In an embodiment of the present application, the assignment process configuration file is used as a request parameter, and the NiFi flow definition upload API is called to create a NiFi process group. At this time, the NiFi process group is the data synchronization process configured in the NiFi data synchronization process template process group, and the configuration of the source data source and the target data source has been completed. By running the NiFi process group, data synchronization between the source data source and the target data source is achieved.

[0069] In the above process, the corresponding NiFi process group can be created through the NiFi API, and data synchronization between the source data source and the target data source can be achieved without complex configuration, solving the problem of increased complexity caused by NiFi's high degree of customizability.

[0070] The following is an explanation of the data synchronization method based on NiFi in the embodiment of the present application in combination with two specific embodiments.

[0071] Embodiment 1: Embodiment 1 describes the data synchronization process when there is no data synchronization scenario that is the same as the current data synchronization scenario. Figure 4 As shown, Figure 4 It is a data synchronization flow chart provided by an embodiment of the present application in the case where there is no data synchronization scenario that is the same as the current data synchronization scenario. Specifically, the data synchronization process in the case where there is no data synchronization scenario that is the same as the current data synchronization scenario includes steps S410 to S470: Step S410: According to the current data synchronization scenario, configure the NiFi data synchronization process template process group.

[0072] Specifically, the NiFi data synchronization process template process group is configured according to the source data source, target data source, data synchronization operation and other information in the current data synchronization scenario.

[0073] Step S420: Export the process configuration file of the NiFi data synchronization process template process group.

[0074] Among them, the process configuration file is a file in JSON format, which represents the overall structure and configuration of the NiFi data synchronization process.

[0075] Step S430: extracting the data source parameters of the process configuration file, and generating a process configuration parameter file according to the data source parameters, wherein the data source parameters include: the path of the source data source, the path of the target data source, and the data source parameter name.

[0076] The path of the source data source refers to the JSON PATH of the source data source, the path of the target data source refers to the JSON PATH of the target data source, and the data source parameter name refers to the FlowDefinition data source parameter name.

[0077] Step S440: assigning values ​​to the process configuration parameter file to obtain an assigned process configuration parameter file.

[0078] Specifically, the data source information is determined according to the source data source and the target data source for data synchronization; and the various data source parameters in the process configuration parameter file are assigned values ​​according to the data source information to obtain the assigned process configuration parameter file.

[0079] Among them, the data source information includes the data source port, IP address, password, account number, synchronized database, data table and other information. For the source data source and target data source that need to be synchronized, their respective data source information is determined respectively, and based on the data source information, each data source parameter in the process configuration parameter file is assigned a value, and finally the assigned process configuration parameter file is obtained.

[0080] Step S450: Replace the parameter values ​​in the assignment process configuration parameter file into the process configuration file to obtain the assignment process configuration file.

[0081] Specifically, the process configuration file is parsed to generate a first object; the assignment process configuration parameter file is parsed to generate a second object; and the first object is replaced by the second object to obtain the assignment process configuration file.

[0082] The first object and the second object are both JSON objects, and replacing the first object with the second object means replacing the JSON PATH of the first object with the JSON PATH of the second object.

[0083] Step S460: Using the assignment process configuration file as a request parameter, calling the NiFi flow definition upload application program interface to create a NiFi process group.

[0084] Step S470: Synchronize data between data sources by running the NiFi process group.

[0085] Among them, the NiFi process group is the data synchronization process configured in the NiFi data synchronization process template process group, and the configuration of the source data source and the target data source has been completed. By running the NiFi process group, data synchronization between the source data source and the target data source can be achieved.

[0086] In the embodiment of the present application, the NiFi data synchronization process template process group is configured according to the current data synchronization scenario, and then the process configuration parameter file is generated, and the assignment process configuration file is obtained by assignment. The assignment process configuration file contains the overall structure and configuration of the NiFi data synchronization process, so data synchronization can be quickly achieved based on the assignment process configuration file. In this way, for different data synchronization scenarios, data synchronization can be achieved without complex configuration, which solves the problem of increased complexity brought by NiFi's high customizability and repeated design and configuration for the same data synchronization scenario. This method is applicable to various data synchronization scenarios based on NiFi and has universality.

[0087] Embodiment 2: Embodiment 2 describes the data synchronization process in the case where there is a data synchronization scenario that is the same as the current data synchronization scenario. Figure 5 As shown, Figure 5 It is a data synchronization flow chart provided by an embodiment of the present application in the case where there is a data synchronization scenario that is the same as the current data synchronization scenario. Specifically, the data synchronization process in the case where there is a data synchronization scenario that is the same as the current data synchronization scenario includes steps S510 to S550: Step S510: According to the current data synchronization scenario, directly call the process configuration parameter file corresponding to the data synchronization scenario.

[0088] Step S520: assigning values ​​to the process configuration parameter file to obtain an assigned process configuration parameter file.

[0089] Specifically, the data source information is determined according to the source data source and the target data source for data synchronization; and the various data source parameters in the process configuration parameter file are assigned values ​​according to the data source information to obtain the assigned process configuration parameter file.

[0090] Among them, the data source information includes the data source port, IP address, password, account number, synchronized database, data table and other information. For the source data source and target data source that need to be synchronized, their respective data source information is determined respectively, and based on the data source information, each data source parameter in the process configuration parameter file is assigned a value, and finally the assigned process configuration parameter file is obtained.

[0091] Step S530: Replace the parameter values ​​in the assignment process configuration parameter file into the process configuration file to obtain the assignment process configuration file.

[0092] Specifically, the process configuration file is parsed to generate a first object; the assignment process configuration parameter file is parsed to generate a second object; and the first object is replaced by the second object to obtain the assignment process configuration file.

[0093] The first object and the second object are both JSON objects, and replacing the first object with the second object means replacing the JSON PATH of the first object with the JSON PATH of the second object.

[0094] Step S540: Using the assignment process configuration file as a request parameter, calling the NiFi flow definition upload application program interface to create a NiFi process group.

[0095] Step S550: Synchronize data between data sources by running the NiFi process group.

[0096] Among them, the NiFi process group is the data synchronization process configured in the NiFi data synchronization process template process group, and the configuration of the source data source and the target data source has been completed. By running the NiFi process group, data synchronization between the source data source and the target data source can be achieved.

[0097] In the embodiment of the present application, for the same data synchronization scenario, the existing process configuration parameter file is directly called, and the assignment process configuration file is obtained by assignment. The assignment process configuration file contains the overall structure and configuration of the NiFi data synchronization process, so data synchronization can be quickly achieved based on the assignment process configuration file. In this way, for the same data synchronization scenario, there is no need to perform repeated configuration, and data synchronization can be achieved by only assigning the process configuration parameter file. Therefore, for the problem of repeated design and configuration of the same data synchronization scenario, this method is applicable to various data synchronization scenarios based on NiFi and has universality.

[0098] The present application also provides a data synchronization device based on NiFi. Figure 6 As shown, Figure 6 : is a structural diagram of a NiFi-based data synchronization device provided in an embodiment of the present application, the device comprising: The acquisition module 610 is used to acquire the process configuration parameter file according to the current data synchronization scenario; The assignment module 620 is used to assign values ​​to the process configuration parameter file to obtain an assignment process configuration parameter file; A replacement module 630 is used to replace the parameter values ​​in the assignment process configuration parameter file with the process configuration file to obtain the assignment process configuration file, wherein the process configuration file represents the overall structure and configuration of the NiFi data synchronization process; The synchronization module 640 is used to perform data synchronization based on the assignment process configuration file.

[0099] The above-mentioned data synchronization device based on NiFi obtains the process configuration parameter file according to the data synchronization scenario, and obtains the assignment process configuration file by assignment. The assignment process configuration file contains the overall structure and configuration of the NiFi data synchronization process, so data synchronization can be quickly achieved based on the assignment process configuration file. In this way, for different data synchronization scenarios, data synchronization can be achieved without complex configuration; for the same data synchronization scenario, repeated configuration is not required, and data synchronization can be achieved by simply assigning the process configuration parameter file. This device solves the problems of increased complexity brought about by the high degree of customizability of NiFi, as well as repeated design and configuration for the same data synchronization scenario. It is suitable for various data synchronization scenarios based on NiFi and has universality.

[0100] In an optional embodiment, the acquisition module includes: A determination module, used to determine whether there is a data synchronization scenario that is the same as the current data synchronization scenario; A generation module, for generating a process configuration parameter file based on a configured NiFi data synchronization process template process group when there is no data synchronization scenario identical to the current data synchronization scenario; The calling module is used to directly call the process configuration parameter file corresponding to the data synchronization scenario when there is a data synchronization scenario that is the same as the current data synchronization scenario.

[0101] In an optional embodiment, the generating module includes: A configuration module, used to configure the NiFi data synchronization process template process group according to the current data synchronization scenario; An export module is used to export the process configuration file of the NiFi data synchronization process template process group; The extraction module is used to extract the data source parameters of the process configuration file and generate a process configuration parameter file according to the data source parameters. The data source parameters include: the path of the source data source, the path of the target data source, and the data source parameter name.

[0102] In an optional embodiment, the assignment module includes: A determination module, used to determine data source information according to a source data source and a target data source for data synchronization; The assignment submodule is used to assign values ​​to various data source parameters in the process configuration parameter file according to the data source information to obtain an assignment process configuration parameter file.

[0103] In an optional embodiment, the replacement module includes: A first parsing module, used to parse the process configuration file to generate a first object; A second parsing module, used to parse the assignment process configuration parameter file to generate a second object; The object replacement module is used to replace the first object with the second object to obtain an assignment process configuration file.

[0104] In an optional embodiment, the synchronization module includes: A calling module, used to take the assignment process configuration file as a request parameter, and call the NiFi flow definition upload application program interface to create a NiFi process group; The running module is used to synchronize data between data sources by running the NiFi process group.

[0105] The NiFi-based data synchronization device provided in the embodiment of the present application can implement the various processes implemented in the NiFi-based data synchronization method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0106] The present application also provides an electronic device, referring to Figure 7 , Figure 7 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes: a memory 710 and a processor 720. The memory 710 and the processor 720 are connected via a bus communication. A computer program is stored in the memory 710. The computer program can be run on the processor 720, thereby implementing the steps of the NiFi-based data synchronization method described in the embodiment of the present application.

[0107] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the NiFi-based data synchronization method described in the embodiment of the present application are implemented.

[0108] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the NiFi-based data synchronization method described in the embodiment of the present application.

[0109] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0110] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0111] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0112] The above is a detailed introduction to a NiFi-based data synchronization method, device, equipment and storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data synchronization method based on NiFi, characterized in that: The method comprises: According to the current data synchronization scenario, obtain the process configuration parameter file; Assigning values ​​to the process configuration parameter file to obtain an assigned process configuration parameter file; Replace the parameter values ​​in the assignment process configuration parameter file with the process configuration file to obtain the assignment process configuration file, wherein the process configuration file represents the overall structure and configuration of the NiFi data synchronization process; Data synchronization is performed based on the assignment process configuration file.

2. The method according to claim 1, characterized in that According to the current data synchronization scenario, obtain the process configuration parameter file, including: Determine whether there is a data synchronization scenario that is the same as the current data synchronization scenario; In the absence of a data synchronization scenario identical to the current data synchronization scenario, generating a process configuration parameter file based on the configured NiFi data synchronization process template process group; In the case that there is a data synchronization scenario that is the same as the current data synchronization scenario, the process configuration parameter file corresponding to the data synchronization scenario is directly called.

3. The method according to claim 2, characterized in that In the absence of a data synchronization scenario that is the same as the current data synchronization scenario, a process configuration parameter file is generated based on the configured NiFi data synchronization process template process group, including: According to the current data synchronization scenario, configure the NiFi data synchronization process template process group; Export the process configuration file of the NiFi data synchronization process template process group; The data source parameters of the process configuration file are extracted, and a process configuration parameter file is generated according to the data source parameters, wherein the data source parameters include: a path of a source data source, a path of a target data source, and a data source parameter name.

4. The method according to claim 1, characterized in that: Assigning the process configuration parameter file to obtain an assigned process configuration parameter file, including: Determine the data source information according to the source data source and target data source for data synchronization; According to the data source information, each data source parameter in the process configuration parameter file is assigned a value to obtain an assigned process configuration parameter file.

5. The method according to claim 1, characterized in that The parameter values ​​in the assignment process configuration parameter file are replaced with the process configuration file to obtain the assignment process configuration file, including: Parsing the process configuration file to generate a first object; Parsing the assignment process configuration parameter file to generate a second object; The first object is replaced by the second object to obtain an assignment process configuration file.

6. The method according to any one of claims 1 to 5, characterized in that: Data synchronization is performed based on the assignment process configuration file, including: Using the assignment process configuration file as a request parameter, calling the NiFi flow definition upload application program interface to create a NiFi process group; Data synchronization between data sources is performed by running the NiFi process group.

7. A data synchronization device based on NiFi, characterized in that: The device comprises: The acquisition module is used to obtain the process configuration parameter file according to the current data synchronization scenario; An assignment module, used for assigning values ​​to the process configuration parameter file to obtain an assignment process configuration parameter file; A replacement module is used to replace the parameter values ​​in the assignment process configuration parameter file with the process configuration file to obtain the assignment process configuration file, wherein the process configuration file represents the overall structure and configuration of the NiFi data synchronization process; A synchronization module is used to synchronize data based on the assignment process configuration file.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the NiFi-based data synchronization method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the NiFi-based data synchronization method described in any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the NiFi-based data synchronization method as described in any one of claims 1 to 6 are implemented.

Images