NiFi-based multi-model data processing method and device, equipment and medium

Abstract

The invention discloses a NiFi-based multi-model data processing method and device, equipment and a medium, and relates to the technical field of data processing, and the method comprises the steps: configuring connection configuration information of a native multi-model database through a preset control service of NiFi, and building a connection relation between the NiFi and the native multi-model database based on the connection configuration information; reading target data from the native multi-model database based on the connection relationship, a set name of a data set configured in the NiFi in advance and a query language of the native multi-model database, and performing incremental extraction on the target data by using incremental information configured in the NiFi in advance; and writing the obtained extracted data into a corresponding preset database based on a preset data writing mode, judging whether the writing operation is successful or not based on the started transaction mechanism in the writing process, and if the writing operation is not successful, triggering a transaction rollback mechanism until the writing operation is successful. The complexity and the cost of data management are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-model data processing method, device, equipment and medium based on NiFi. Background Art

[0002] With the acceleration of digital transformation, data volumes are exploding, while business needs are also becoming increasingly diverse. These changes have led to the complexity of data types and structures, such as complex data relationship analysis, social network data, and IoT data. Traditional relational databases may need to use different database systems or tools to handle this complex and unstructured data, increasing the complexity and cost of data management and facing performance bottlenecks such as slow query speeds and poor scalability.

[0003] As can be seen from the above, how to improve data query speed when processing large amounts of data to reduce the complexity and cost of data management is an urgent problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a multi-model data processing method, device, equipment and medium based on NiFi, which can improve data query speed when processing large amounts of data, thereby reducing the complexity and cost of data management. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a multi-model data processing method based on NiFi, comprising:

[0006] Configure the connection configuration information of the native multi-model database through the preset control service in NiFi, and establish a connection relationship between NiFi and the native multi-model database based on the connection configuration information;

[0007] Based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database, read the corresponding target data from the native multi-model database, and use the incremental information pre-configured in the NiFi to perform an incremental extraction operation on the target data to obtain the extracted data;

[0008] The extracted data is written into the corresponding preset database based on the preset data writing method, and the transaction mechanism is started during the writing process to determine whether the write operation is successful based on the transaction mechanism. If unsuccessful, the transaction rollback mechanism is triggered until the write operation is successful.

[0009] Optionally, configuring the connection configuration information of the native multi-model database through a preset control service in NiFi, and establishing a connection relationship between NiFi and the native multi-model database based on the connection configuration information, includes:

[0010] Configuring a driver and a software development kit corresponding to the native multi-model database in a preset development environment, and configuring connection configuration information including a host address and a port number of the native multi-model database in a preset configuration file based on the driver and the software development kit;

[0011] A preset control service for a native multi-model database is defined, and the connection configuration information is obtained based on the preset control service, and a connection relationship between NiFi and the native multi-model database is established based on the connection configuration information.

[0012] Optionally, reading corresponding target data from the native multi-model database based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database includes:

[0013] Configure the collection name of the corresponding data set in NiFi based on user needs; the data set includes a document set and an edge set;

[0014] determining a target query language of the native multi-model database corresponding to the data set based on the set name, and reading corresponding target data from the native multi-model database based on the target query language and the connection relationship;

[0015] The target query language includes a first query language corresponding to the document set, a second query language corresponding to the edge set, and a third query language corresponding to the document set and the edge set.

[0016] Optionally, performing an incremental extraction operation on the target data using incremental information pre-configured in the NiFi to obtain extracted data includes:

[0017] Determine whether there is incremental information configured in the NiFi;

[0018] If there is configured incremental information in the NiFi, triggering the step of performing an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi;

[0019] If the configured incremental information does not exist in the NiFi, a full extraction operation is performed on the target data to obtain the extracted data.

[0020] Optionally, performing an incremental extraction operation on the target data using incremental information pre-configured in the NiFi to obtain extracted data includes:

[0021] Configure the corresponding incremental fields and incremental initial values in NiFi based on user needs;

[0022] An incremental extraction operation is performed on the target data using the incremental initial value to obtain extracted data, and the incremental initial value is updated based on the extracted data, and then the obtained updated incremental value is determined as the incremental initial value of the next incremental extraction operation.

[0023] Optionally, writing the extracted data into a corresponding preset database based on a preset data writing method includes:

[0024] Encapsulating the extracted data into a stream file, and parsing the stream file to obtain the data to be written;

[0025] A storage set is constructed based on the native multi-model database, and the data to be written is written in parallel into the storage set or the relational database.

[0026] Optionally, the transaction mechanism is enabled during the writing process to determine whether the writing operation is successful based on the transaction mechanism, and if unsuccessful, a transaction rollback mechanism is triggered until the writing operation is successful, including:

[0027] During the writing process, a transaction mechanism is enabled to monitor the writing process based on the transaction mechanism, and a monitoring result is used to determine whether there is a transaction anomaly or an anomaly in the preset database during the writing process;

[0028] If there is a transaction exception or a preset database exception during the writing process, the writing operation is determined to be unsuccessful, and then the transaction rollback mechanism is triggered, and the process jumps to the step of monitoring the writing process based on the transaction mechanism until the writing operation is successful.

[0029] In a second aspect, the present application provides a multi-model data processing device based on NiFi, comprising:

[0030] A connection relationship establishment module is used to configure the connection configuration information of the native multi-model database through the preset control service in NiFi, and establish a connection relationship between NiFi and the native multi-model database based on the connection configuration information;

[0031] A target data extraction module is configured to read the corresponding target data from the native multi-model database based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database, and perform an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi to obtain extracted data;

[0032] The data writing module is used to write the extracted data into the corresponding preset database based on the preset data writing method, and start the transaction mechanism during the writing process to determine whether the writing operation is successful based on the transaction mechanism. If it is unsuccessful, the transaction rollback mechanism is triggered until the writing operation is successful.

[0033] In a third aspect, the present application provides an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the aforementioned NiFi-based multi-model data processing method.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned NiFi-based multi-model data processing method when executed by a processor.

[0037] This application configures the connection configuration information of the native multi-model database through the preset control service in NiFi, and establishes a connection relationship between the NiFi and the native multi-model database based on the connection configuration information; based on the connection relationship, the collection name of the data collection pre-configured in the NiFi and the query language of the native multi-model database, the corresponding target data is read from the native multi-model database, and the incremental information pre-configured in the NiFi is used to perform an incremental extraction operation on the target data to obtain the extracted data; based on the preset data writing method, the extracted data is written into the corresponding preset database, and the transaction mechanism is started during the writing process to determine whether the write operation is successful based on the transaction mechanism. If unsuccessful, the transaction rollback mechanism is triggered until the write operation is successful.

[0038] As can be seen from the above, this application configures the connection configuration information of the native multi-model database through the preset control service in NiFi to ensure that NiFi can successfully connect to the native multi-model database, and then by configuring the collection name and the query language of the native multi-model database, it can accurately locate and read the required target data, and then perform incremental extraction operations on the target data, reducing the amount of data processing and improving processing efficiency. The extracted data is written to the corresponding preset database, and the transaction mechanism is enabled during the writing process, which improves the accuracy and consistency of data writing. In this way, by using NiFi as a data stream processing framework and combining it with the native multi-model database for data query and writing, the efficiency, consistency and reliability of data processing can be improved, and the processing requirements of complex data can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A flowchart of a multi-model data processing method based on NiFi disclosed in this application;

[0041] Figure 2 This is a flowchart of NIFI reading native multi-model database disclosed in this application;

[0042] Figure 3 This is a schematic diagram of the structure of a multi-model data processing device based on NiFi disclosed in this application;

[0043] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] Currently, the amount of data is exploding. Traditional relational databases may need to use different database systems or tools when processing complex and unstructured data, which increases the complexity and cost of data management and faces performance bottlenecks such as slow query speed and poor scalability. To this end, this application provides a multi-model data processing method based on NiFi. By using NiFi as a data stream processing framework and combining it with a native multi-model database for data query and writing, it can improve the efficiency, consistency and reliability of data processing and meet the processing requirements of complex data.

[0046] See also Figure 1 As shown, the embodiment of the present invention discloses a multi-model data processing method based on NiFi, including:

[0047] Step S11: Configure the connection configuration information of the native multi-model database through the preset control service in NiFi, and establish a connection relationship between NiFi and the native multi-model database based on the connection configuration information.

[0048] The preset development environment in this embodiment is Java code development, using the Java driver arangodb-java-driver of ArangoDB (i.e., the native multi-model database), and configuring the SDK (i.e., software development kit) of the native multi-model database. Based on the driver and the software development kit, connection configuration information is configured in a preset configuration file; wherein, the connection configuration information includes but is not limited to the host address, port number, user name and password of the native multi-model database. Then, an ArangoDBControllerService (i.e., preset control service) corresponding to the native multi-model database is defined in NiFi, and the connection configuration information is configured in the preset control service as a global service for other components to call. The connection configuration information is obtained by calling the preset control service, and a connection relationship between NiFi and the native multi-model database is established based on the connection configuration information.

[0049] Specifically, the connection configuration information of the native multi-model database is configured through the preset control service in NiFi, and the connection relationship between the NiFi and the native multi-model database is established based on the connection configuration information, including: configuring a driver and a software development kit corresponding to the native multi-model database in a preset development environment, and configuring the connection configuration information including the host address and port number of the native multi-model database in a preset configuration file based on the driver and the software development kit; defining a preset control service for the native multi-model database, and obtaining the connection configuration information based on the preset control service, and establishing a connection relationship between the NiFi and the native multi-model database based on the connection configuration information.

[0050] Step S12: Based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database, read the corresponding target data from the native multi-model database, and use the incremental information pre-configured in the NiFi to perform an incremental extraction operation on the target data to obtain the extracted data.

[0051] In this embodiment, after establishing a connection between NiFi and the native multi-model database, the collection name and related parameters for reading or writing data sets from or into the native multi-model database are configured based on user needs. The collection name includes the names of the document and edge. The target query language of the native multi-model database corresponding to the data set is determined based on the collection name. The target query language includes a first query language corresponding to the document collection (i.e., Document AQL), a second query language corresponding to the edge collection (i.e., Edge AQL), and a third query language combining the document collection and the edge collection. The corresponding target data is read from the native multi-model database based on the target query language and the connection relationship. It is worth noting that the AQL (i.e., the query language of the native multi-model database) query in this embodiment supports filter condition queries.

[0052] Specifically, the reading of corresponding target data from the native multi-model database based on the connection relationship, the collection name of the data set pre-configured in the NiFi, and the query language of the native multi-model database includes: configuring the collection name of the corresponding data set in the NiFi based on user needs; the data set includes a document set and an edge set; determining the target query language of the native multi-model database corresponding to the data set based on the collection name, so as to read the corresponding target data from the native multi-model database based on the target query language and the connection relationship; wherein, the target query language includes a first query language corresponding to the document set, a second query language corresponding to the edge set, and a third query language corresponding to the document set and the edge set.

[0053] It is understood that after obtaining the target data, incremental information can be selectively configured in NiFi based on the user's needs. The incremental information includes an incremental field and an incremental initial value. The target data is then incrementally extracted using the incremental initial value to obtain the extracted data, and the incremental initial value is updated based on the extracted data. The updated incremental value is then determined as the incremental initial value for the next incremental extraction operation, so that the corresponding incremental extraction operation is determined based on the incremental initial value. In a specific embodiment, if only the incremental field is set, the first extraction is a full extraction of data. After the full extraction is completed, the incremental field value of the last extracted record is stored in the NiFi StateMap (a data structure for storing and accessing state information) as the starting value for the next incremental extraction of data, and the incremental value is updated sequentially. If both the incremental field and the incremental initial value are set, the initial value of the first extraction starts with the set incremental initial value, and the next extraction is performed dynamically according to the dynamically increasing incremental data. Specifically, the incremental extraction operation is performed on the target data using the incremental information pre-configured in the NiFi to obtain the extracted data, including: configuring the corresponding incremental field and incremental initial value in the NiFi based on user needs; performing an incremental extraction operation on the target data using the incremental initial value to obtain the extracted data, and updating the incremental initial value based on the extracted data, and then determining the obtained updated incremental value as the incremental initial value of the next incremental extraction operation.

[0054] In this embodiment, after the incremental information is configured, a corresponding incremental extraction operation is performed based on the incremental information. It is determined whether the configured incremental information exists in the NiFi; if the configured incremental information exists in the NiFi, the step of performing an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi is triggered; if the configured incremental information does not exist in the NiFi, a full extraction operation is performed on the target data to obtain the extracted data. Specifically, the incremental extraction operation is performed on the target data using the incremental information pre-configured in the NiFi to obtain the extracted data, including: determining whether the configured incremental information exists in the NiFi; if the configured incremental information exists in the NiFi, the step of performing an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi is triggered; if the configured incremental information does not exist in the NiFi, a full extraction operation is performed on the target data to obtain the extracted data.

[0055] Step S13: write the extracted data into the corresponding preset database based on the preset data writing method, and start the transaction mechanism during the writing process to determine whether the write operation is successful based on the transaction mechanism. If unsuccessful, trigger the transaction rollback mechanism until the write operation is successful.

[0056] In this embodiment, after obtaining the extracted data, the extracted data is encapsulated into a stream file, and then the stream file is parsed to obtain the data to be written, a storage set of the native multi-model database is constructed, the stream file is written to a relational database, and the data to be written is written to the storage set; multiple pieces of data can be written in batches at one time, and the number of concurrent threads of the reading and writing components is appropriately adjusted according to the amount of data written to the storage set. The writing speed or reading speed is increased by setting the number of parallel tasks Concurrent Tasks in the NIFI component. Specifically, the writing of the extracted data into the corresponding preset database based on the preset data writing method includes: encapsulating the extracted data into a stream file, and parsing the stream file to obtain the data to be written; constructing a storage set based on the native multi-model database, and writing the data to be written into the storage set or the relational database in parallel.

[0057] It is understood that during the data writing process, a transaction mechanism is enabled and the writing process is monitored based on the transaction mechanism. If an abnormality is found during the monitoring process, the abnormality includes a transaction abnormality, violation of data constraints, and the preset database; it indicates that the write operation is unsuccessful and a transaction rollback is automatically triggered. In a specific embodiment, when writing data, if the uniqueness constraint of a field is violated during the writing process, the database will detect this error and automatically roll back the entire transaction to ensure that the data is not in an inconsistent state. Specifically, the transaction mechanism is enabled during the writing process to determine whether the write operation is successful based on the transaction mechanism. If it is unsuccessful, the transaction rollback mechanism is triggered until the write operation is successful. The method includes: enabling the transaction mechanism during the writing process to monitor the writing process based on the transaction mechanism, and using the monitoring results to determine whether there is a transaction abnormality or a preset database abnormality during the writing process; if there is a transaction abnormality or a preset database abnormality during the writing process, determining that the write operation is unsuccessful, then triggering the transaction rollback mechanism, and jumping to the step of monitoring the write process based on the transaction mechanism until the write operation is successful.

[0058] It should be noted that the NiFi-based multi-model data processing solution in this embodiment is applicable to a variety of complex data processing scenarios, especially those with diverse data sources, dynamically changing processing requirements, and high real-time requirements. For example, it can be applied to synchronize market transaction data in the financial field, where the target data can be user transaction records or real-time market data.

[0059] As can be seen from the above, this application configures the connection configuration information of the native multi-model database through the preset control service in NiFi to ensure that NiFi can successfully connect to the native multi-model database, and then by configuring the collection name and the query language of the native multi-model database, it can accurately locate and read the required target data, and then perform incremental extraction operations on the target data, reducing the amount of data processing and improving processing efficiency. The extracted data is written to the corresponding preset database, and the transaction mechanism is enabled during the writing process, which improves the accuracy and consistency of data writing. In this way, by using NiFi as a data stream processing framework and combining it with the native multi-model database for data query and writing, the efficiency, consistency and reliability of data processing can be improved, and the processing requirements of complex data can be guaranteed.

[0060] As can be seen from the above examples, this application performs incremental data extraction and transaction processing based on the connection relationship between NiFi and a native multi-model database, thereby improving data query speed when processing large amounts of data and reducing the complexity and cost of data management. Therefore, the process of performing incremental data extraction and transaction processing based on the connection relationship between NiFi and a native multi-model database is described.

[0061] See also Figure 2 As shown, an embodiment of the present invention discloses a flowchart of NIFI reading a native multi-model database, including:

[0062] In this embodiment, a connection relationship is first established between NiFi and a native multi-model database (i.e., ArangoDB). Specifically, a driver and a software development kit corresponding to the native multi-model database are configured, and connection configuration information is configured in a preset configuration file based on the driver and the software development kit. Then, a preset control service corresponding to the native multi-model database is defined in NiFi, the connection configuration information is obtained by calling the preset control service, and a connection relationship is established between NiFi and the native multi-model database based on the connection configuration information. After the connection relationship is established, the data of the entire data set is read from the native multi-model database based on the connection relationship, that is, full reading. This method is suitable for situations where all data needs to be obtained, such as data initialization or full backup; the data of the entire data set is read from the native multi-model database based on the connection relationship, and then the newly added or updated data is continuously read, that is, full and incremental reading are combined. This method is suitable for scenarios where data synchronization needs to be maintained to ensure that the data in NiFi is consistent with the data in the native multi-model database; incremental data in the native multi-model database is read based on the connection relationship, that is, incremental reading. This method is suitable for situations where the data volume is large and only newly added or changed data needs to be processed to improve the efficiency of data extraction.

[0063] It is understood that the native multi-model database's query language (AQL) can also be used to read data to obtain the target data. After obtaining the target data, it can be written to a pre-created storage set of the native multi-model database. The target data can also be written to relational databases, which is suitable for scenarios where data from the native multi-model database needs to be migrated or synchronized to relational databases. The target data can also be written to other types of databases, such as NoSQL databases and time-series databases. This improves the flexible migration and integration capabilities of data between different database systems.

[0064] As can be seen from the above, this application performs data interaction through the connection relationship between NiFi and the native multi-model database, including reading the corresponding target data from the native multi-model database according to different reading methods, and writing the target data into the storage collection or other types of databases in the native multi-model database, so that the data flow between different systems is more flexible and efficient, meeting the processing requirements of complex data.

[0065] Accordingly, see Figure 3 As shown, the present application also provides a multi-model data processing device based on NiFi, including:

[0066] A connection relationship establishment module 11 is used to configure the connection configuration information of the native multi-model database through a preset control service in NiFi, and establish a connection relationship between NiFi and the native multi-model database based on the connection configuration information;

[0067] A target data extraction module 12 is configured to read corresponding target data from the native multi-model database based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database, and perform an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi to obtain extracted data;

[0068] The data writing module 13 is used to write the extracted data into the corresponding preset database based on the preset data writing method, and start the transaction mechanism during the writing process to determine whether the writing operation is successful based on the transaction mechanism. If it is unsuccessful, the transaction rollback mechanism is triggered until the writing operation is successful.

[0069] As can be seen from the above, this application configures the connection configuration information of the native multi-model database through the preset control service in NiFi to ensure that NiFi can successfully connect to the native multi-model database, and then by configuring the collection name and the query language of the native multi-model database, it can accurately locate and read the required target data, and then perform incremental extraction operations on the target data, reducing the amount of data processing and improving processing efficiency. The extracted data is written to the corresponding preset database, and the transaction mechanism is enabled during the writing process, which improves the accuracy and consistency of data writing. In this way, by using NiFi as a data stream processing framework and combining it with the native multi-model database for data query and writing, the efficiency, consistency and reliability of data processing can be improved, and the processing requirements of complex data can be guaranteed.

[0070] In some specific implementations, the connection relationship establishing module 11 may specifically include:

[0071] a connection configuration information configuration unit, configured to configure a driver and a software development kit corresponding to the native multi-model database in a preset development environment, and configure connection configuration information including a host address and a port number of the native multi-model database in a preset configuration file based on the driver and the software development kit;

[0072] A connection relationship establishing unit is used to define a preset control service for a native multi-model database, obtain the connection configuration information based on the preset control service, and establish a connection relationship between NiFi and the native multi-model database based on the connection configuration information.

[0073] In some specific implementations, the target data extraction module 12 may specifically include:

[0074] A collection name configuration unit, used to configure the collection name of the corresponding data collection in NiFi based on user requirements;

[0075] A target data reading unit is configured to determine a target query language of the native multi-model database corresponding to the data set based on the set name, so as to read corresponding target data from the native multi-model database based on the target query language and the connection relationship.

[0076] In some specific implementations, the target data extraction module 12 may specifically include:

[0077] An incremental information determination unit, configured to determine whether there is configured incremental information in the NiFi;

[0078] A data re-extraction unit, configured to trigger the step of performing an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi if there is incremental information configured in the NiFi;

[0079] The data full extraction unit is used to perform a full extraction operation on the target data to obtain extracted data if the configured incremental information does not exist in the NiFi.

[0080] In some specific implementations, the target data extraction module 12 may specifically include:

[0081] An incremental field configuration unit, configured to configure corresponding incremental fields and incremental initial values in the NiFi based on user requirements;

[0082] The incremental initial value updating unit is used to perform an incremental extraction operation on the target data using the incremental initial value to obtain extracted data, and to update the incremental initial value based on the extracted data, and then determine the obtained updated incremental value as the incremental initial value of the next incremental extraction operation.

[0083] In some specific implementations, the data writing module 13 may specifically include:

[0084] An extracted data parsing unit, configured to encapsulate the extracted data into a stream file and parse the stream file to obtain data to be written;

[0085] A storage set construction unit is used to construct a storage set based on the native multi-model database, and write the data to be written into the storage set or the relational database in parallel.

[0086] In some specific implementations, the data writing module 14 may specifically include:

[0087] A transaction mechanism activation unit, configured to activate the transaction mechanism during the writing process, monitor the writing process based on the transaction mechanism, and determine whether there is a transaction anomaly or an anomaly in the preset database during the writing process using the monitoring result;

[0088] The transaction rollback unit is used to determine that the write operation is unsuccessful if there is a transaction exception or a preset database exception during the write process, then trigger the transaction rollback mechanism, and jump to the step of monitoring the write process based on the transaction mechanism until the write operation is successful.

[0089] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the figure cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multi-model data processing method based on NiFi disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0090] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0091] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0092] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to implement the NiFi-based multi-model data processing method executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer programs 222 can further include computer programs that can be used to complete other specific tasks.

[0093] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned NiFi-based multi-model data processing method. For the specific steps of this method, please refer to the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0095] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0097] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0098] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-model data processing method based on NiFi, characterized in that: include: Configure the connection configuration information of the native multi-model database through the preset control service in NiFi, and establish a connection relationship between NiFi and the native multi-model database based on the connection configuration information; Based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database, read the corresponding target data from the native multi-model database, and use the incremental information pre-configured in the NiFi to perform an incremental extraction operation on the target data to obtain the extracted data; The extracted data is written into the corresponding preset database based on the preset data writing method, and the transaction mechanism is started during the writing process to determine whether the write operation is successful based on the transaction mechanism. If unsuccessful, the transaction rollback mechanism is triggered until the write operation is successful.

2. The multi-model data processing method based on NiFi according to claim 1, characterized in that Configuring connection configuration information of the native multi-model database through a preset control service in NiFi, and establishing a connection relationship between NiFi and the native multi-model database based on the connection configuration information, includes: Configuring a driver and a software development kit corresponding to the native multi-model database in a preset development environment, and configuring connection configuration information including a host address and a port number of the native multi-model database in a preset configuration file based on the driver and the software development kit; A preset control service for a native multi-model database is defined, and the connection configuration information is obtained based on the preset control service, and a connection relationship between NiFi and the native multi-model database is established based on the connection configuration information.

3. The multi-model data processing method based on NiFi according to claim 1, characterized in that The reading corresponding target data from the native multi-model database based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database includes: Configure the collection name of the corresponding data set in NiFi based on user needs; the data set includes a document set and an edge set; determining a target query language of the native multi-model database corresponding to the data set based on the set name, and reading corresponding target data from the native multi-model database based on the target query language and the connection relationship; The target query language includes a first query language corresponding to the document set, a second query language corresponding to the edge set, and a third query language corresponding to the document set and the edge set.

4. The multi-model data processing method based on NiFi according to claim 1, characterized in that The incremental extraction operation is performed on the target data using the incremental information pre-configured in the NiFi to obtain the extracted data, including: Determine whether there is incremental information configured in the NiFi; If there is configured incremental information in the NiFi, triggering the step of performing an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi; If the configured incremental information does not exist in the NiFi, a full extraction operation is performed on the target data to obtain the extracted data.

5. The multi-model data processing method based on NiFi according to claim 1, characterized in that: The incremental extraction operation is performed on the target data using the incremental information pre-configured in the NiFi to obtain the extracted data, including: Configure the corresponding incremental fields and incremental initial values in NiFi based on user needs; An incremental extraction operation is performed on the target data using the incremental initial value to obtain extracted data, and the incremental initial value is updated based on the extracted data, and then the obtained updated incremental value is determined as the incremental initial value of the next incremental extraction operation.

6. The multi-model data processing method based on NiFi according to claim 1, characterized in that: Writing the extracted data into a corresponding preset database based on a preset data writing method includes: Encapsulating the extracted data into a stream file, and parsing the stream file to obtain the data to be written; A storage set is constructed based on the native multi-model database, and the data to be written is written in parallel into the storage set or the relational database.

7. The multi-model data processing method based on NiFi according to any one of claims 1 to 6, characterized in that: The transaction mechanism is started during the writing process to determine whether the writing operation is successful based on the transaction mechanism. If unsuccessful, the transaction rollback mechanism is triggered until the writing operation is successful, including: During the writing process, a transaction mechanism is enabled to monitor the writing process based on the transaction mechanism, and a monitoring result is used to determine whether there is a transaction anomaly or an anomaly in the preset database during the writing process; If there is a transaction exception or a preset database exception during the writing process, the writing operation is determined to be unsuccessful, and then the transaction rollback mechanism is triggered, and the process jumps to the step of monitoring the writing process based on the transaction mechanism until the writing operation is successful.

8. A multi-model data processing device based on NiFi, characterized in that: include: A connection relationship establishment module is used to configure the connection configuration information of the native multi-model database through the preset control service in NiFi, and establish a connection relationship between NiFi and the native multi-model database based on the connection configuration information; A target data extraction module is configured to read the corresponding target data from the native multi-model database based on the connection relationship, the collection name of the data collection pre-configured in the NiFi, and the query language of the native multi-model database, and perform an incremental extraction operation on the target data using the incremental information pre-configured in the NiFi to obtain extracted data; The data writing module is used to write the extracted data into the corresponding preset database based on the preset data writing method, and start the transaction mechanism during the writing process to determine whether the writing operation is successful based on the transaction mechanism. If it is unsuccessful, the transaction rollback mechanism is triggered until the writing operation is successful.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the NiFi-based multi-model data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the NiFi-based multi-model data processing method according to any one of claims 1 to 7 is implemented.

Images