Data processing method and device based on data flow mechanism, medium and equipment

Abstract

The invention provides a data processing method and device based on a data flow mechanism, a medium and equipment. The method comprises the steps that a first configuration page is provided through a front end, so that a user configures a database and a NiFi plug-in on the first configuration page; establishing a connection between the NiFi plug-in and the database; the method comprises the following steps: providing a flow configuration page through a front end, so that a user constructs a plurality of nodes on the flow configuration page in a mode of dragging a graphic identifier, and arranging the plurality of nodes into an expected data processing flow according to a data processing requirement; data query operation is carried out from the database through the NiFi plug-in, so that the database processes the queried data according to the processing steps in the data processing flow, and an obtained processing result is returned to the NiFi plug-in; and the NiFi plug-in performs format conversion processing on the processing result, and displays the data after the format conversion processing on the front end. The method is visual, simple and convenient.

Description

Technical Field

[0001] The present invention relates to the technical field of data warehouses, and in particular to a data processing method and device, medium and equipment based on a data flow mechanism. Background Art

[0002] With the gradual rise of the concept of big data, more and more companies are paying close attention to how to store and process their own data. Initially, big data processing usually relied on the MapReduce programming model. Due to the high entry threshold of this model, the big data warehouse Hive came into being. Business personnel only need to master simple SQL statements to get started.

[0003] In the data processing process of an enterprise, it is impossible to orchestrate tasks by relying solely on Hive. Later, you can use Shell scripts to connect multiple HQL files in series to orchestrate complex processing tasks. HQL is the abbreviation of Hive Query Language, which is a query language used to query data stored in the Hadoop distributed file system. HQL is very similar to SQL in design, which makes it easy for users who are familiar with SQL to learn and use HQL for big data processing.

[0004] However, the method of scheduling HQL files through Shell scripts is too rough and cannot guarantee the visualization of data flows. When errors occur during data processing, users cannot directly view the specific error links. Secondly, the scheduling capabilities of Shell scripts are limited. For example, it is impossible to automatically configure the number of failed reruns. If scheduling relationships such as upstream and downstream dependencies and parallel operations between multiple processes are involved, even if Shell scripts can be implemented, professional programmers are required to implement the code. In summary, there are two disadvantages to the implementation method through Shell scripts: one is that it is not intuitive, and the other is that it is cumbersome. Summary of the invention

[0005] In order to solve at least one of the above technical problems, an embodiment of the present invention provides a data processing method, device, medium and equipment based on a data flow mechanism.

[0006] According to the first aspect, the data processing method based on the data stream mechanism provided by the embodiment of the present invention includes:

[0007] Providing a first configuration page through the front end so that the user can configure the database and the NiFi plug-in on the first configuration page;

[0008] Establishing a connection between the NiFi plug-in and the database;

[0009] A process configuration page is provided through the front end, so that the user can build multiple nodes by dragging and dropping graphic icons on the process configuration page, and arrange the multiple nodes into an expected data processing process according to data processing requirements; wherein each node corresponds to a data processing step;

[0010] Based on the connection between the NiFi plug-in and the database, a data query operation is performed from the database through the NiFi plug-in, so that the database processes the queried data according to the processing steps in the data processing flow, and returns the obtained processing results to the NiFi plug-in; wherein, after receiving the processing results, the NiFi plug-in performs format conversion processing on the processing results, and displays the format-converted data on the front end.

[0011] In one embodiment, the method also includes: after configuring the NiFi plug-in on the first configuration page, notifying the backend to configure the processing logic scheme corresponding to the NiFi plug-in; wherein the processing logic scheme is used to perform data query operations from the database according to the query conditions entered by the user at the query node in the data processing flow, and after receiving the processing results returned by the database, format conversion processing is performed on the processing results, and the data after format conversion processing is displayed on the front end.

[0012] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in on the back end includes: creating a first preset class and a first method in the first preset class, the first method being used to perform data query operations from the database according to the query conditions entered by the user on the front end, and receiving the processing results returned by the database; wherein the query conditions include the access address and query string of the database.

[0013] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in on the back end further includes: creating a second method, wherein the second method is used to obtain the query condition input by the user on the front end.

[0014] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in by the backend also includes: configuring the processor; creating a second preset class for describing the configured processor; registering the second preset class with the NiFi plug-in, so that the NiFi plug-in can use the processor to perform format conversion processing on the processing result to obtain a stream file in a preset format.

[0015] In one embodiment, the process of configuring the processor by the backend includes: configuring the properties of the processor, wherein the properties include the preset format.

[0016] In one embodiment, establishing a connection between the NiFi plug-in and the database includes: establishing a connection between the NiFi plug-in and the database through a JDBC driver; wherein the JDBC driver is a plug-in in Java for database connection.

[0017] According to a second aspect, a data processing device based on a data stream mechanism provided by an embodiment of the present invention includes:

[0018] A first configuration module, used to provide a first configuration page through the front end, so that the user can configure the database and the NiFi plug-in on the first configuration page;

[0019] A connection establishment module, used to establish a connection between the NiFi plug-in and the database;

[0020] The second configuration module is used to provide a process configuration page through the front end, so that the user can build multiple nodes by dragging and dropping graphic icons on the process configuration page, and arrange the multiple nodes into an expected data processing process according to data processing requirements; wherein each node corresponds to a data processing step;

[0021] A data processing module is used to perform data query operations from the database through the NiFi plug-in based on the connection between the NiFi plug-in and the database, so that the database processes the queried data according to the processing steps in the data processing flow, and returns the obtained processing results to the NiFi plug-in; wherein, after receiving the processing results, the NiFi plug-in performs format conversion processing on the processing results, and displays the format-converted data on the front end.

[0022] In one embodiment, the device may further include:

[0023] The third configuration module is used to notify the backend to configure the processing logic scheme corresponding to the NiFi plug-in after configuring the NiFi plug-in on the first configuration page; wherein the processing logic scheme is used to perform data query operations from the database according to the query conditions entered by the user at the query node in the data processing flow, and after receiving the processing results returned by the database, perform format conversion processing on the processing results, and display the data after format conversion processing on the front end.

[0024] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in on the back end includes: creating a first preset class and a first method in the first preset class, the first method being used to perform data query operations from the database according to the query conditions entered by the user on the front end, and receiving the processing results returned by the database; wherein the query conditions include the access address and query string of the database.

[0025] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in on the back end further includes: creating a second method, wherein the second method is used to obtain the query condition input by the user on the front end.

[0026] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in by the backend also includes: configuring the processor; creating a second preset class for describing the configured processor; registering the second preset class with the NiFi plug-in, so that the NiFi plug-in can use the processor to perform format conversion processing on the processing result to obtain a stream file in a preset format.

[0027] In one embodiment, the process of configuring the processor by the backend includes: configuring the properties of the processor, wherein the properties include the preset format.

[0028] In one embodiment, the connection establishment module is specifically used to: establish a connection between the NiFi plug-in and the database through a JDBC driver; wherein the JDBC driver is a plug-in in Java for database connection.

[0029] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method provided in the first aspect.

[0030] According to a fourth aspect, a computing device provided by an embodiment of the present invention includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided by the first aspect is implemented.

[0031] The data processing method, device, medium, and equipment based on the data flow mechanism provided by the embodiment of the present invention, the user configures the database and the NiFi plug-in on the first configuration page, and then can establish a connection between the NiFi plug-in and the database, and then the user can build multiple nodes by dragging and dropping graphic identifiers on the process configuration page, and arrange the multiple nodes into the expected data processing process according to the data processing requirements. Based on the connection between the NiFi plug-in and the database, the NiFi plug-in performs a data query operation from the database, so that the database processes the queried data according to the processing steps in the data processing process, and returns the obtained processing results to the NiFi plug-in, so that the NiFi plug-in performs format conversion processing on the processing results, and displays the format converted data on the front end. It can be seen that the embodiment of the present invention provides an intuitive drag-and-drop user interface, which makes it simple and intuitive to build a data processing process. Developers and operation and maintenance personnel can easily design data processing processes without having to implement them through Shell scripts, which is not only intuitive, but also simple and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of a data processing method based on a data flow mechanism in one embodiment of the present invention;

[0033] Figure 2 It is a structural block diagram of a data processing device based on a data flow mechanism in one embodiment of the present invention. DETAILED DESCRIPTION

[0034] In a first aspect, an embodiment of the present invention provides a data processing method based on a data flow mechanism, see Figure 1 The method comprises the following steps S110 to S140:

[0035] S110, providing a first configuration page through the front end, so that the user can configure the database and the NiFi plug-in on the first configuration page;

[0036] That is to say, users can configure the database and NiFi plug-in on the first configuration page of the front end, where the database is the Hive database. Configuring the NiFi plug-in helps to establish a connection with the Hive database later. In actual scenarios, it is necessary to add necessary JAR files to the NiFi plug-in to support the client libraries of Hive and Hadoop. Hive is a database built on the database Hadoop.

[0037] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without understanding the underlying details of the distribution. It fully utilizes the power of clusters for high-speed computing and storage. Hadoop implements a distributed file system, namely Distributed File System.

[0038] Among them, NiFi is an easy-to-use, powerful and reliable plug-in for data pulling, data processing and distribution, which is used to automatically manage data flows between systems.

[0039] Specifically, the database configuration includes inputting the Hive table name, the HDFS path of Hive, and the partition when Hive is a partitioned table through visual configuration. It also supports the configuration of advanced properties to achieve personalized configuration.

[0040] S120, establishing a connection between the NiFi plug-in and the database;

[0041] That is, after the configuration in S110 is completed, a connection can be established between the NiFi plug-in and the database. Only when the connection is established can data be queried and results returned by the database be received in subsequent steps.

[0042] In one embodiment, establishing a connection between the NiFi plug-in and the database may include: establishing a connection between the NiFi plug-in and the database through a JDBC driver; wherein the JDBC driver is a plug-in in Java for database connection.

[0043] JDBC, or Java Database Connectivity, is a plug-in for database connection in Java. It allows Java developers to interact with the database using standard SQL language.

[0044] S130, providing a process configuration page through the front end, so that the user can construct multiple nodes by dragging and dropping graphic icons on the process configuration page, and arrange the multiple nodes into an expected data processing process according to data processing requirements; wherein each node corresponds to a data processing step;

[0045] That is to say, users can drag and drop related graphic icons on the front-end process configuration page to form multiple nodes on the page, and then arrange these multiple nodes according to data processing requirements to obtain a data processing process. A node represents a step in the data processing process and is reflected on the page as an icon. Specifically, the user selects a graphic icon in the graphic icon selection area and drags it to the editing area, so that a node is formed in the editing area.

[0046] It can be seen that in S130, each node is connected together by a line by dragging and dropping, and there is no need to manually write a script to execute the steps corresponding to the node like the previous shell script. In fact, the steps of the node correspond to the task, and the relevant information of the scheduling execution of the task can be configured on the page, for example, the task execution mode is configured. The embodiment of the present invention supports multiple task execution modes, such as polling, timing, daily, weekly, monthly and custom. At the same time, the retry time interval when the task fails, the number of parallel tasks, etc. can be configured.

[0047] Among them, users can add, delete, modify and other operations on the front end. For example, right-click and select the Delete button to delete the node. Right-click and select the Modify button to adjust the node. You can hold down the left mouse button to drag and drop the node and connect the nodes.

[0048] S140. Based on the connection between the NiFi plug-in and the database, a data query operation is performed from the database through the NiFi plug-in, so that the database processes the queried data according to the processing steps in the data processing flow, and returns the obtained processing result to the NiFi plug-in; wherein, after receiving the processing result, the NiFi plug-in performs format conversion processing on the processing result, and displays the format-converted data on the front end.

[0049] That is to say, when the user has a query demand, the query request is passed to the database through the NiFi plug-in, so that the database will perform a query operation based on the request parameters in the query request, obtain the queried data, and then perform processing operations on the queried data according to the above data processing flow, for example, cleaning processing, desensitization processing, aggregation processing, statistical analysis processing, etc., to obtain the processing result. Then the database returns the processing result to the NiFi plug-in. After receiving the processing result, the NiFi plug-in can convert the format of the processing result and obtain a standardized unified format displayed on the front end, so that the user can see the processing result after the format conversion.

[0050] Among them, the front-end page can be developed using the current mainstream front-end technology VUE. Based on the user-centered design principle, the core is to enhance the user's experience and interaction with the system, put the core needs of data processing first, and enable data developers to quickly and easily build data flows in daily data development work. The lines of the input box on the front-end page are clear and the color is lighter. Different modules are designed and laid out according to functional classification, which is clear for users to use. Users can arrange various complex processes by dragging and dropping.

[0051] In one embodiment, the method may further include:

[0052] After configuring the NiFi plug-in on the first configuration page, notify the backend to configure the processing logic scheme corresponding to the NiFi plug-in; wherein the processing logic scheme is used to perform data query operations from the database according to the query conditions entered by the user at the query node in the data processing flow, and after receiving the processing results returned by the database, perform format conversion processing on the processing results, and display the data after format conversion processing on the front end.

[0053] That is to say, after the user configures the NiFi plug-in on the first configuration page, the front end can notify the back end to configure the processing logic scheme corresponding to the NiFi plug-in. The function of the processing logic scheme is to implement the processing logic of the NiFi plug-in on the back end. The processing logic is specifically: sending a query request to the database and receiving the processing result returned by the database; after receiving the processing result, performing format conversion processing on the processing result, and displaying the format converted data on the front end.

[0054] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in by the backend may include:

[0055] 1. Create a first preset class and a first method in the first preset class, wherein the first method is used to perform a data query operation from the database according to the query conditions input by the user at the front end, and receive the processing results returned by the database; wherein the query conditions include the access address of the database and the query string.

[0056] It is understandable that the standard version of NiFi does not directly provide methods, so the embodiment of the present invention configures a first preset class and a first method in the class. The first preset class is the AbstractProcessor class, and the first method is the onTrigger method. The function of the onTrigger method is to make a query request to the database based on the query conditions and receive the processing result returned by the database.

[0057] The query string is the filter condition in the query condition.

[0058] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in by the backend may also include:

[0059] 2. Create a second method, which is used to obtain the query conditions entered by the user on the front end.

[0060] Among them, the second method is the ProcessContext method, and the second method can obtain the query conditions entered by the user on the front end.

[0061] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in at the back end may also include the following steps 3 to 5:

[0062] 3. Configure the processor;

[0063] In order to make the processor usable, the processor needs to be configured. Specifically, the properties of the processor can be configured, and the properties include the preset format. That is, the preset format required for the processor to perform data conversion, that is, the processed file format. For example, the configuration of the processor can be completed through the PropertyDescriptor interface and the getSupportedPropertyDescriptors method.

[0064] 4. Create a second preset class for describing the configured processor;

[0065] The second preset class, namely the ProcessorDescriptor class, is used to describe a configured processor.

[0066] 5. Register the second preset class in the NiFi plug-in, so that the NiFi plug-in can use the processor to perform format conversion processing on the processing result to obtain a stream file in a preset format.

[0067] That is, registering the second preset class in the NiFi plug-in is equivalent to binding the processor to the NiFi plug-in. When the processor is required to process, the processing result can be formatted according to the description information in the second preset class to obtain a stream file in a preset format.

[0068] It can be seen that the embodiment of the present invention further expands the connection, data processing and processing of Hive on the basis of the NiFi framework. It also realizes the front-end and back-end interaction in the form of a visual interface and provides a convenient method of use from the user's perspective. The main beneficial effects are manifested in the following aspects:

[0069] (1) Visual interface: It provides an intuitive drag-and-drop user interface that makes building data processing flows simple and intuitive. Developers and operation and maintenance personnel can easily design, test, and deploy data processing flows.

[0070] (2) Flexibility: Supports multiple data formats, such as JSON, XML, CSV, Parquet, etc., and multiple protocols, such as HTTP, FTP, SFTP, Kafka, etc.

[0071] (3) Reliability: Ensure data persistence and consistency, and ensure that data will not be lost even when a failure occurs. Support transactional data processing to ensure the integrity of data flow.

[0072] (4) Scalability: The architectural design of NiFi supports horizontal expansion, and more nodes can be added as needed to further expand the data processing process.

[0073] Among them, in the embodiment of the present invention, the front end uses the vue framework and the back end uses the springboot framework. The logic code corresponding to the method of the embodiment of the present invention can be developed using the java language.

[0074] It can be seen that the embodiment of the present invention provides a method for connecting to the Hive data source and performing visual configuration and orchestration based on the NiFi framework, which solves the pain points that the conventional data processing process is not convenient to program using shell scripts and has a high entry threshold. In the embodiment of the present invention, the bottom layer is secondary encapsulated based on NiFi, and Hive is used as a data source to access, process and process data. In addition, several core classes of NiFi are implemented using the JAVA language, and the trigger method is rewritten. All operations are configured based on the front-end page, and there is no need to write shell scripts to implement them. The visualization of the data processing process is achieved, allowing users to orchestrate the data processing process in a concise and clear manner.

[0075] In a second aspect, an embodiment of the present invention provides a data processing device based on a data flow mechanism, see Figure 2 , the device 100 comprises:

[0076] A first configuration module 110, configured to provide a first configuration page through a front end, so that a user can configure a database and a NiFi plug-in on the first configuration page;

[0077] A connection establishment module 120, used to establish a connection between the NiFi plug-in and the database;

[0078] The second configuration module 130 is used to provide a process configuration page through the front end, so that the user can build multiple nodes by dragging and dropping graphic icons on the process configuration page, and arrange the multiple nodes into an expected data processing process according to data processing requirements; wherein each node corresponds to a data processing step;

[0079] The data processing module 140 is used to perform data query operations from the database through the NiFi plug-in based on the connection between the NiFi plug-in and the database, so that the database processes the queried data according to the processing steps in the data processing flow, and returns the obtained processing results to the NiFi plug-in; wherein, the NiFi plug-in performs format conversion processing on the processing results after receiving the processing results, and displays the format-converted data on the front end.

[0080] In one embodiment, the device may further include:

[0081] The third configuration module is used to notify the backend to configure the processing logic scheme corresponding to the NiFi plug-in after configuring the NiFi plug-in on the first configuration page; wherein the processing logic scheme is used to perform data query operations from the database according to the query conditions entered by the user at the query node in the data processing flow, and after receiving the processing results returned by the database, perform format conversion processing on the processing results, and display the data after format conversion processing on the front end.

[0082] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in on the back end includes: creating a first preset class and a first method in the first preset class, the first method being used to perform data query operations from the database according to the query conditions entered by the user on the front end, and receiving the processing results returned by the database; wherein the query conditions include the access address and query string of the database.

[0083] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in on the back end further includes: creating a second method, wherein the second method is used to obtain the query condition input by the user on the front end.

[0084] In one embodiment, the process of configuring the processing logic scheme corresponding to the NiFi plug-in by the backend also includes: configuring the processor; creating a second preset class for describing the configured processor; registering the second preset class with the NiFi plug-in, so that the NiFi plug-in can use the processor to perform format conversion processing on the processing result to obtain a stream file in a preset format.

[0085] In one embodiment, the process of configuring the processor by the backend includes: configuring the properties of the processor, wherein the properties include the preset format.

[0086] In one embodiment, the connection establishment module is specifically used to: establish a connection between the NiFi plug-in and the database through a JDBC driver; wherein the JDBC driver is a plug-in in Java for database connection.

[0087] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0088] In a third aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0089] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0090] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0091] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0092] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0093] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0094] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0095] In a fourth aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any embodiment of the specification.

[0096] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0097] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0098] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0099] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method based on a data flow mechanism, characterized in that: include: Providing a first configuration page through the front end so that the user can configure the database and the NiFi plug-in on the first configuration page; Establishing a connection between the NiFi plug-in and the database; A process configuration page is provided through the front end, so that the user can build multiple nodes by dragging and dropping graphic icons on the process configuration page, and arrange the multiple nodes into an expected data processing process according to data processing requirements; wherein each node corresponds to a data processing step; Based on the connection between the NiFi plug-in and the database, a data query operation is performed from the database through the NiFi plug-in, so that the database processes the queried data according to the processing steps in the data processing flow, and returns the obtained processing results to the NiFi plug-in; wherein, after receiving the processing results, the NiFi plug-in performs format conversion processing on the processing results, and displays the format-converted data on the front end.

2. The method according to claim 1, characterized in that Also includes: After configuring the NiFi plug-in on the first configuration page, notify the backend to configure the processing logic scheme corresponding to the NiFi plug-in; wherein the processing logic scheme is used to perform data query operations from the database according to the query conditions entered by the user at the query node in the data processing flow, and after receiving the processing results returned by the database, perform format conversion processing on the processing results, and display the data after format conversion processing on the front end.

3. The method according to claim 2, characterized in that The process of configuring the processing logic scheme corresponding to the NiFi plug-in at the back end includes: Create a first preset class and a first method in the first preset class, wherein the first method is used to perform data query operations from the database according to query conditions input by a user at the front end, and receive processing results returned by the database; wherein the query conditions include an access address of the database and a query string.

4. The method according to claim 3, characterized in that The process of configuring the processing logic scheme corresponding to the NiFi plug-in at the back end also includes: A second method is created, where the second method is used to obtain the query condition entered by the user on the front end.

5. The method according to claim 3, characterized in that: The process of configuring the processing logic scheme corresponding to the NiFi plug-in at the back end also includes: Configure the processor; creating a second preset class for describing the configured processor; The second preset class is registered in the NiFi plug-in, so that the NiFi plug-in can use the processor to perform format conversion processing on the processing result to obtain a stream file in a preset format.

6. The method according to claim 5, characterized in that The process of configuring the processor by the back end includes: Attributes of the processor are configured, wherein the attributes include the preset format.

7. The method according to claim 1, characterized in that The establishing of a connection between the NiFi plug-in and the database includes: A connection between the NiFi plug-in and the database is established through a JDBC driver; wherein the JDBC driver is a plug-in in Java for database connection.

8. A data processing device based on a data flow mechanism, characterized in that: include: A first configuration module, used to provide a first configuration page through the front end, so that the user can configure the database and the NiFi plug-in on the first configuration page; A connection establishment module, used to establish a connection between the NiFi plug-in and the database; The second configuration module is used to provide a process configuration page through the front end, so that the user can build multiple nodes by dragging and dropping graphic icons on the process configuration page, and arrange the multiple nodes into an expected data processing process according to data processing requirements; wherein each node corresponds to a data processing step; A data processing module is used to perform data query operations from the database through the NiFi plug-in based on the connection between the NiFi plug-in and the database, so that the database processes the queried data according to the processing steps in the data processing flow, and returns the obtained processing results to the NiFi plug-in; wherein, after receiving the processing results, the NiFi plug-in performs format conversion processing on the processing results, and displays the format-converted data on the front end.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Images