Data processing method, system, electronic device and storage medium

Abstract

The embodiment of the present invention discloses a data processing method, system, electronic device and storage medium. The method includes: a first node collects at least one target data in a first database and obtains a data identifier of the target data; the first node sends the target data to a second node according to the data identifier; wherein the second node corresponds to the data identifier one by one; the second node processes the target data to obtain at least one storage data, and sends the storage data to a third node; wherein the number of storage data is consistent with the number of target data; the third node corresponds to the second node one by one; the third node batches the storage data according to the operation type of the storage data to store the storage data in the second database. The technical solution of the embodiment of the present invention can improve the processing efficiency of real-time data warehousing without relying on the flink cluster.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a data processing method, system, electronic device, and storage medium. Background Art

[0002] Data warehousing refers to the process of storing source data (that is, data in the data source) in a data warehouse after a series of processing.

[0003] With the continuous development of big data technology, the demand for real-time data warehousing is increasing. At present, it is usually necessary to rely on flink (an open source stream processing and batch processing framework) cluster to process the source data and store it in the data warehouse to improve the processing efficiency of real-time data warehousing.

[0004] However, when the Flink cluster fails or its performance degrades, the processing efficiency of data warehousing may be reduced. Summary of the invention

[0005] The embodiments of the present invention provide a data processing method, system, electronic device and storage medium, which can improve the processing efficiency of real-time data warehousing without relying on a flink cluster.

[0006] According to one aspect of the present invention, there is provided a data processing method, the method comprising:

[0007] The first node collects at least one target data in the first database and obtains a data identifier of the target data;

[0008] The first node sends the target data to the second node according to the data identifier; wherein the second node corresponds to the data identifier one by one;

[0009] The second node processes the target data to obtain at least one storage data, and sends the storage data to the third node; wherein the number of the storage data is consistent with the number of the target data; the third node corresponds to the second node one by one;

[0010] The third node batches the incoming data according to the operation type of the incoming data and stores the incoming data in the second database.

[0011] According to another aspect of the present invention, there is provided a data processing system, the system comprising a first node, a second node and a third node;

[0012] A first node is used to collect at least one target data in the first database and obtain a data identifier of the target data;

[0013] The first node is further used to send the target data to the second node according to the data identifier; wherein the second node corresponds to the data identifier one by one;

[0014] The second node is used to process the target data to obtain at least one storage data, and send the storage data to the third node; wherein the number of the storage data is consistent with the number of the target data; the third node corresponds to the second node one by one;

[0015] The third node is used to batch process the incoming data according to the operation type of the incoming data and store the incoming data in the second database.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention collects target data in the first database through the first node and obtains the data identifier of the target data. The first node sends the target data to the second node according to the data identifier, and then processes the target data through the second node to obtain input data, and sends the input data to the third node, so that the third node batches the input data according to the operation type of the input data, thereby storing the input data in the second database. This solves the problem of reduced processing efficiency of data warehousing when the flink cluster fails or performance degrades due to the dependence of data warehousing on the flink cluster, and can improve the processing efficiency of real-time data warehousing without relying on the flink cluster.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a data processing method provided according to Embodiment 1 of the present invention;

[0025] Figure 2a and Figure 2b is a flowchart of another data processing method provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a flowchart of another data processing method provided according to Embodiment 3 of the present invention;

[0027] Figure 4 is a specific example flow chart of a data processing method provided according to Embodiment 4 of the present invention;

[0028] Figure 5a-5c This is a flowchart of continuously storing batch data into a database according to Embodiment 4 of the present invention;

[0029] Figure 6 is a schematic diagram of a data processing system provided according to Embodiment 5 of the present invention

[0030] Figure 7 It is a structural schematic diagram of an electronic device for implementing the data processing method of an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] NiFi is an easy-to-use, powerful and reliable streaming data processing and distribution system. It provides a web-based graphical user interface (GUI) that allows users to build data flow processing flows by dragging and connecting components without writing a lot of code.

[0034] NiFi can automatically pull data from various data sources (such as databases, file systems, network protocols, etc.) and process, transform, and distribute the data based on user-defined processing flows.

[0035] Embodiment 1

[0036] Figure 1 This is a flow chart of a data processing method provided by the first embodiment of the present invention. This embodiment is applicable to the case of improving the processing efficiency of real-time data warehousing. The method can be executed by a data processing system, which can be implemented by software and / or hardware, and can generally be directly integrated into an electronic device that executes the method. The electronic device can be a terminal device or a server device. The embodiment of the present invention does not limit the type of electronic device that executes the data processing method. Specifically, Figure 1 As shown, the data processing method may specifically include the following steps:

[0037] S110. The first node collects at least one target data in the first database, and obtains a data identifier of the target data.

[0038] The first node may be a NiFi node for real-time data collection. The first database may be a database where the data in the data source is located.

[0039] The target data may be data that has changed in the data source (i.e., the first database). Exemplarily, the target data may be inserted data, deleted data, updated data, etc., and the present application does not limit this. It is understood that the number of target data may be one or more, that is, there may be multiple data that have changed.

[0040] The data identifier can be used to identify the target data. Exemplarily, the data identifier can be the value of the primary key (ie, key value) corresponding to the target data in the database where the target data is located.

[0041] In the embodiment of the present invention, the target data in the first database is collected by the first node, and the data identifier of the target data is obtained.

[0042] S120. The first node sends the target data to the second node according to the data identifier; wherein the second node corresponds to the data identifier one by one.

[0043] The second node may be a NiFi node for performing data processing on the target data.

[0044] Specifically, the second nodes correspond to the data identifiers one by one, that is, the number of the second nodes can be consistent with the number of the data identifiers. Exemplarily, assuming that multiple target data correspond to three data identifiers, there can be three second nodes, and each second node can process target data with different data identifiers.

[0045] In the embodiment of the present invention, after the first node obtains the data identifier of the target data, the first node may further send the target data to the second node according to the data identifier, so that the second node may perform data processing on the target data.

[0046] It is understandable that the first node may send the target data to the second node by means of streaming data processing, that is, the target data may be sent to the second node in sequence according to the order in which the target data is collected.

[0047] S130, the second node processes the target data to obtain at least one stored data, and sends the stored data to the third node; wherein the number of stored data is consistent with the number of target data; the third node corresponds to the second node one by one.

[0048] Among them, data processing can be any processing of data, such as data cleaning, data conversion, etc., and this application does not limit this.

[0049] The storage data may be data that can be stored in a database after data processing is performed on the target data. Specifically, after data processing is performed on each target data, a storage data may be obtained.

[0050] Among them, the third node can be a NiFi node for storing the incoming data in the data warehouse. Specifically, the number of the third nodes is consistent with the number of the second nodes, that is, each second node has a third node corresponding to it. After the second node obtains the incoming data, the incoming data can be directly sent to the third node corresponding to the second node.

[0051] In the embodiment of the present invention, after the first node sends the target data to the second node, the second node may further process the target data to obtain storage data, so as to send the storage data to the third node.

[0052] S140. The third node batch-processes the incoming data according to the operation type of the incoming data to store the incoming data in the second database.

[0053] The operation type may be a type corresponding to operations such as adding, deleting, and modifying data in a database. Specifically, the operation type may include an insert type, an update type, and a delete type.

[0054] Among them, batching is a data processing technology that is mainly used to improve data processing efficiency and reduce resource consumption. In data stream processing, batching refers to merging multiple data batches together for processing instead of processing them one by one. This can reduce the number of status updates and avoid sending intermediate redundant messages, thereby improving processing efficiency and reducing system pressure.

[0055] The second database may be a database corresponding to the data warehouse, that is, during the data warehousing process, the source data may be stored in the second database after a series of processing.

[0056] In an embodiment of the present invention, after the second node sends the incoming data to the third node, the third node may further perform batch processing on the incoming data according to the operation type of the incoming data to store the incoming data in the second database.

[0057] The technical solution of this embodiment collects target data in the first database through the first node and obtains the data identifier of the target data. The first node sends the target data to the second node according to the data identifier, and then processes the target data through the second node to obtain input data, and sends the input data to the third node, so that the third node batches the input data according to the operation type of the input data, thereby storing the input data in the second database. This solves the problem of reduced processing efficiency of data warehousing when the flink cluster fails or performance degrades due to the dependence of data warehousing on the flink cluster, and can improve the processing efficiency of real-time data warehousing without relying on the flink cluster.

[0058] Embodiment 2

[0059] Figure 2a and Figure 2b 1 is a flow chart of a data processing method provided by Embodiment 2 of the present invention. This embodiment is a further refinement of the above-mentioned technical solutions, and provides a variety of specific optional implementation methods. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above-mentioned embodiments. Figure 2a and Figure 2b As shown, the method may include the following steps:

[0060] S210: The first node monitors data changes in the first database.

[0061] S220. When there is a data change in the first database, the first node collects the changed data and determines the changed data as the target data; wherein the number of the first database is at least one; and the number of the first node is at least one.

[0062] The data change may be a change in the data in the first database, such as inserting data, updating data, deleting data, etc., and the present application does not impose any restrictions on this.

[0063] In an embodiment of the present invention, the first node monitors data changes in the first database, and when there is a data change in the first database, the changed data is collected through the first node to determine the changed data as the target data.

[0064] Specifically, the first node may collect the changed data through the cdc component, and determine the changed data as the target data.

[0065] It is understandable that in a multi-concurrency scenario, usually at the same time point, many users access the same API interface or Url address at the same time. Therefore, at the same time point, there may be many data changes in the first database, that is, the first node can collect multiple target data.

[0066] In the present application, the number of the first databases may be one or more, and the number of the first nodes may be one or more.

[0067] Specifically, when the number of the first node is one, target data in one or more first databases may be collected through the first node.

[0068] Specifically, when the number of the first database is one and the number of the first nodes is multiple, target data in different tables of the first database may be collected through multiple first nodes.

[0069] For example, assuming that the first database includes Table 1, Table 2, and Table 3, and the number of first nodes is 3 (including Node 1, Node 2, and Node 3), then the target data in Table 1 can be collected through Node 1, the target data in Table 2 can be collected through Node 2, and the target data in Table 3 can be collected through Node 3.

[0070] Specifically, when there are multiple first databases and multiple first nodes, target data in different tables of each first database may be collected through multiple first nodes respectively.

[0071] Exemplarily, assuming that the first database includes database 1 and database 2, database 1 includes table 11 and table 12, database 2 includes table 21 and table 22, and the first node includes node 1, node 2, node 3, and node 4, then node 1 can collect the target data in table 11, node 2 can collect the target data in table 12, node 3 can collect the target data in table 21, and node 4 can collect the target data in table 22.

[0072] It should be noted that when there are multiple first nodes, no matter whether there are one or more first databases, each first node may correspond to one or more tables, and the present application does not impose any restrictions on this.

[0073] Exemplarily, assuming that the first database includes Table 1, Table 2, and Table 3, and the first node includes Node 1 and Node 2, then the target data in Table 1 and Table 3 can be collected by Node 1, and the target data in Table 2 can be collected by Node 2. Alternatively, the target data in Table 1 can be collected by Node 1, and the target data in Table 2 and Table 3 can be collected by Node 2, etc., and this application does not limit this.

[0074] In this way, by setting up multiple first nodes to collect target data in the first database, the parallel processing efficiency of data processing can be improved.

[0075] S230: The first node obtains a data identifier of the target data.

[0076] S240: The first node determines the second node identifier of each second node.

[0077] S250: When the data identifier of the target data and the second node identifier satisfy a corresponding relationship, the first node sends the target data to the second node corresponding to the second node identifier.

[0078] The second node identifier may be used to identify the second node. The corresponding relationship may be a one-to-one correspondence between the data identifier and the second node identifier.

[0079] Exemplarily, assuming that the data identifiers of the target data are identifier 1, identifier 2, and identifier 3, and the second node identifier may include node A, node B, and node C, then node A may correspond to identifier 1, node B may correspond to identifier 2, and node C may correspond to identifier 3.

[0080] In an embodiment of the present invention, after obtaining the data identifier of the target data, the first node can further determine the second node identifier of each second node, and when the data identifier of the target data and the second node identifier satisfy the corresponding relationship, the target data is sent to the second node corresponding to the second node identifier.

[0081] Exemplarily, assuming that node A corresponds to identifier 1, the target data with the data identifier being identifier 1 may be sent to a second node with the second node identifier being node A.

[0082] Optionally, the node identifier of each second node may be set in advance according to the data identifier, that is, the partition attribute of the second node may be specified, so that different second nodes correspond to different data identifiers.

[0083] S260: The first child node in the second node performs data type conversion on the target data.

[0084] The first child node may be a NiFi node in the second node, and may be used to perform data type conversion on target data.

[0085] Specifically, the first sub-node may perform data type conversion according to the data type of the target data and the data type in the second database.

[0086] Exemplarily, assuming that the data type of the target data is JSON data, and the data type in the second database is Avro data, the data type conversion may be to convert the JSON data into the Avro data type.

[0087] S270. The second child node in the second node performs field type mapping on the target data after data type conversion to obtain storage data.

[0088] The second child node may be another NiFi node in the second node, and may be used to perform field type mapping on the target data after data type conversion.

[0089] Specifically, the second sub-node may perform field type mapping according to the field type of the target data (ie, the field type of the source field) and the field type in the second database (ie, the field type of the target field).

[0090] S280. The third node batch-processes the incoming data according to the operation type of the incoming data to store the incoming data in the second database.

[0091] It can be understood that the number of the third nodes is consistent with the number of the second nodes, but the number of the second nodes and the number of the third nodes are independent of the number of the first nodes.

[0092] The technical solution of this embodiment is to monitor the data changes in the first database through the first node, and collect the changed data when there is data change in the first database, determine the changed data as the target data, obtain the data identifier of the target data, and determine the second node identifier of each second node, so that when the data identifier of the target data satisfies the corresponding relationship with the second node identifier, the target data is sent to the second node corresponding to the second node identifier; the target data is converted into a data type through the first child node in the second node, and then the target data after the data type conversion is mapped into a field type through the second child node in the second node to obtain the storage data; the storage data is batched and processed according to the operation type of the storage data through the third node, so that the storage data is stored in the second database. In this way, by using the load balancing method of NiFi multi-nodes to partition according to the data identifier (that is, the key value), the target data with the same data identifier is sent to the same node for processing, which ensures the sequentiality of the target data with the same key value, and at the same time, through the multi-node distributed capability of NiFi, the processing parallelism of data storage is improved.

[0093] Embodiment 3

[0094] Figure 3 is a flow chart of another data processing method provided by Embodiment 3 of the present invention. This embodiment is a further refinement of the above-mentioned technical solutions, and provides a variety of specific optional implementation methods. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above-mentioned embodiments. Figure 3 As shown, the method may include the following steps:

[0095] S310: The first node collects at least one target data in the first database, and obtains a data identifier of the target data.

[0096] S320: The first node sends the target data to the second node according to the data identifier.

[0097] S330: The second node processes the target data to obtain at least one storage data, and sends the storage data to the third node.

[0098] S340. The third node determines each stored data as current data in turn.

[0099] The current data refers to the data currently being put into storage.

[0100] It can be understood that the storage data and the target data correspond one to one, that is, the number of storage data can be multiple.

[0101] Specifically, the third node may adopt a streaming data processing method to sequentially determine the incoming data as current data according to the order of the incoming data in the data stream.

[0102] S350, the third node determines whether the current data is the first data; if so, execute S360; if not, execute S370.

[0103] The first piece of data is the first piece of data stored in the second database.

[0104] Specifically, after sending the incoming data to the third node, the second node may further determine each incoming data as current data in sequence through the third node, and determine whether the current data is the first piece of data.

[0105] S360. The third node puts the current data into the first queue; wherein the first queue is a queue corresponding to the operation type of the current data.

[0106] Specifically, when the current data is the first piece of data, the third node may put the current data into a queue corresponding to the operation type of the current data.

[0107] For example, if the operation type of the current data is insert type and the first queue is the insert queue, the current data can be put into the insert queue. If the operation type of the current data is update type and the first queue is the update queue, the current data can be put into the update queue. If the operation type of the current data is delete type and the first queue is the delete queue, the current data can be put into the delete queue.

[0108] S370. The third node determines the previous piece of data of the current data; and when the operation type of the current data is inconsistent with the operation type of the previous piece of data, puts the current data into the first queue.

[0109] The previous data may be the previous stored data corresponding to the current data.

[0110] Specifically, when the current data is not the first data, the third node can determine the previous data of the current data, and when the operation type of the previous data is inconsistent with the operation type of the current data, put the current data into the queue corresponding to the operation type of the current data.

[0111] For example, assuming that the operation type of the current data is insert type and the operation type of the previous data is update type, the current data can be put into the insert queue.

[0112] Optionally, when the operation type of the current data is inconsistent with the operation type of the previous data, the data in the second queue may also be stored in the second database.

[0113] The second queue is a queue corresponding to the operation type of the previous data when the operation type of the current data is inconsistent with the operation type of the previous data.

[0114] Exemplarily, assuming that the operation type of the current data is insert, the operation type of the previous data is update, and the second queue is an update queue, the data in the update queue can be stored in the second database.

[0115] Specifically, when the current data is not the first piece of data, if the operation type of the current data is inconsistent with the operation type of the previous piece of data, the data in the queue corresponding to the operation type of the previous data (that is, the second queue) is stored in the second database, and the current data is placed in the queue corresponding to the operation type of the current data (that is, the first queue).

[0116] It can be understood that if the operation type of the current data is consistent with the operation type of the previous data, it means that the queue corresponding to the operation type of the previous data is the same queue as the queue corresponding to the operation type of the current data, that is, the queue corresponding to the operation type of the previous data is the first queue, then the current data can be directly placed in the first queue.

[0117] S380. The third node determines the next piece of data of the current data.

[0118] S390. When the operation type of the current data is consistent with the operation type of the next data, the third node puts the next data into the first queue.

[0119] The next piece of data may be the next piece of data to be stored corresponding to the current data.

[0120] Specifically, after putting the current data into the first queue, the third node may further determine the next piece of data of the current data, and put the next piece of data into the first queue when the operation type of the current data is consistent with the operation type of the next piece of data.

[0121] S3100. When the operation type of the current data is inconsistent with the operation type of the next data, the third node stores the data in the first queue into the second database and puts the next data into the third queue.

[0122] The third queue is a queue corresponding to the operation type of the next piece of data when the operation type of the current data is inconsistent with the operation type of the next piece of data.

[0123] Specifically, when the operation type of the current data is inconsistent with the operation type of the next data, the third node stores the data in the queue corresponding to the operation type of the current data (that is, the first queue) into the second database, and puts the next data into the queue corresponding to the operation type of the next data (that is, the third queue).

[0124] For example, assuming that the operation type of the current data is insert, the first queue is the insert queue, the operation type of the next data is update, and the third queue is the update queue, the data in the insert queue can be stored in the second database, and the next data can be placed in the update queue.

[0125] In the prior art, for the data corresponding to DML (Data Manipulation Language) operations (such as update, insert, and delete), there are strict execution order requirements between them, and they cannot be mixed for batch operations. They can only be written one by one at a time, which results in low data warehousing efficiency, thereby affecting the processing efficiency of data warehousing.

[0126] In the embodiment of the present application, the third node improves the processing capacity of data entry through batch writing while ensuring the order of data through continuous batch accumulation, thereby solving the problem of being unable to enter the data corresponding to the DML operation into the warehouse in batches.

[0127] The technical solution of this embodiment is to collect the target data in the first database through the first node, and obtain the data identification of the target data, so as to send the target data to the second node according to the data identification, the second node performs data processing on the target data to obtain the storage data, and sends the storage data to the third node; the third node determines each storage data as the current data in turn, and puts the current data into the first queue when the current data is the first data; when the current data is not the first data, the previous data of the current data is determined, and when the operation type of the previous data is inconsistent with the operation type of the current data, the current data is put into the first queue; after the third node puts the current data into the first queue, it determines the next data of the current data, and when the operation type of the current data is consistent with the operation type of the next data, the next data is put into the first queue, and when the operation type of the current data is inconsistent with the operation type of the next data, the third node stores the data in the first queue into the second database, and puts the next data into the third queue. In this way, the strategy of flexible continuous batch storage of storage data of multiple operation types is realized, while ensuring the orderliness of the data, the processing efficiency of data storage is improved through batch writing.

[0128] Embodiment 4

[0129] In order to enable those skilled in the art to better understand the data processing method of this embodiment, the embodiment of the present invention is specifically described by taking the real-time data integration scenario as an example. In the real-time data integration scenario, the NiFi cluster runs the data synchronization process task, and the specific process includes data collection, data conversion, data mapping, and data storage. The NiFi cluster is composed of multiple nodes, and each step of the data entry process task is evenly distributed to each node for single concurrent execution. The NiFi component is used to parse the database log collection data at the source, and the NiFi component for storage stores the converted data in a continuous batch manner.

[0130] Figure 4 is a specific example flow chart of a data processing method provided by Embodiment 4 of the present invention, such as Figure 4 As shown, the method may specifically include the following steps:

[0131] Step A: NiFi's cdc component captures the source data and puts the primary key value into the attributes of the data flow file.

[0132] Specifically, NiFi's cdc component performs the task of real-time data collection and is responsible for capturing data changes in the first database (that is, the source database) in real time. When there is a data change, the data will be collected and partitioned by key value and flowed to the next NiFi node.

[0133] Step B: Predetermine the partition attributes of each NiFi node in the NiFi queue, and distribute data to different NiFi nodes for processing according to the primary key attribute value.

[0134] Specifically, according to the key value of the data, the data corresponding to the same key value is distributed to the same NiFi node for single-threaded processing.

[0135] Step C: The downstream NiFi node receives the distributed data and processes it in a single thread.

[0136] Specifically, after receiving the data, the NiFi node converts the data type, such as converting json data into avro data type. The converted data flows to the next NiFi node by key value partition. The next NiFi node maps the fields, that is, maps the source fields to the target fields, and sends the mapping results to the next NiFi node by key value partition.

[0137] Step D: The NiFi warehousing component stores data in batches.

[0138] Specifically, Figure 5a-5c As shown, in order to ensure the sequentiality of data operations and support batch storage, the process of continuous batch accumulation is implemented: there are three situations; the first is the accumulation of insert batches. In order to ensure the order of data, if the same data is encountered, it will continue to be put into the insert queue for batch accumulation. When update or delete is encountered, the insert batch accumulation will be ended and the data of the insert queue will be written into the warehouse in batches; the second is the accumulation of update batches. In order to ensure the order of data, if the same data is encountered, it will continue to be put into the update queue for batch accumulation. When insert or delete is encountered, the update batch accumulation will be ended and the data of the update queue will be written into the warehouse in batches; the third is the accumulation of delete batches. In order to ensure the order of data, if the same data is encountered, it will continue to be put into the delete queue for batch accumulation. When update or insert is encountered, the delete batch accumulation will be ended and the data of the delete queue will be written into the warehouse in batches.

[0139] In the embodiment of the present invention, by using the load balancing capability of NiFi, data is partitioned, ensuring that data with the same primary key can be distributed to the same node, and data is stored in the warehouse in a continuous batch manner, so that data can be stored in the warehouse based on NiFi's load balancing without relying on the flink cluster, and continuous batch storage is realized, which not only ensures the sequential operation of data, but also significantly improves the storage speed.

[0140] Embodiment 5

[0141] Figure 6 is a schematic diagram of a data processing system provided by Embodiment 5 of the present invention, such as Figure 6 As shown, the system includes: a first node 610, a second node 620 and a third node 630, wherein:

[0142] The first node 610 is used to collect at least one target data in the first database and obtain a data identifier of the target data;

[0143] The first node 610 is further used to send the target data to the second node according to the data identifier; wherein the second node corresponds to the data identifier one by one;

[0144] The second node 620 is used to process the target data to obtain at least one storage data, and send the storage data to the third node; wherein the number of storage data is consistent with the number of target data; the third node corresponds to the second node one by one;

[0145] The third node 630 is used to batch process the incoming data according to the operation type of the incoming data to store the incoming data in the second database.

[0146] The technical solution of this embodiment collects target data in the first database through the first node and obtains the data identifier of the target data. The first node sends the target data to the second node according to the data identifier, and then processes the target data through the second node to obtain input data, and sends the input data to the third node, so that the third node batches the input data according to the operation type of the input data, thereby storing the input data in the second database. This solves the problem of reduced processing efficiency of data warehousing when the flink cluster fails or performance degrades due to the dependence of data warehousing on the flink cluster, and can improve the processing efficiency of real-time data warehousing without relying on the flink cluster.

[0147] Optionally, the first node 610 can be specifically used to: determine the second node identifier of each second node before the first node sends the target data to the second node based on the data identifier; and send the target data to the second node corresponding to the second node identifier when the data identifier of the target data and the second node identifier satisfy a corresponding relationship.

[0148] Optionally, the first node 610 can also be specifically used for: the first node monitors data changes in the first database; when there are data changes in the first database, the first node collects the changed data and determines the changed data as target data; wherein the number of first databases is at least one; the number of first nodes is at least one.

[0149] Optionally, the second node 620 may include a first sub-node 621 and a second sub-node 622; wherein the first sub-node 621 is used to perform data type conversion on the target data; and the second sub-node 622 is used to perform field type mapping on the target data after data type conversion to obtain storage data.

[0150] Optionally, the third node 630 can be specifically used to: determine each stored data as the current data in turn; when the current data is the first data, put the current data into the first queue; wherein the first queue is a queue corresponding to the operation type of the current data; when the current data is not the first data, determine the previous data of the current data; when the operation type of the current data is inconsistent with the operation type of the previous data, put the current data into the first queue.

[0151] Optionally, the third node 630 can also be specifically used to: when the operation type of the current data is inconsistent with the operation type of the previous data, store the data in the second queue to the second database; wherein the second queue is the queue corresponding to the operation type of the previous data when the operation type of the current data is inconsistent with the operation type of the previous data.

[0152] Optionally, the third node 630 can also be specifically used to: after placing the current data into the first queue, determine the next data of the current data; when the operation type of the current data is consistent with the operation type of the next data, place the next data into the first queue; when the operation type of the current data is inconsistent with the operation type of the next data, store the data in the first queue to the second database, and place the next data into the third queue; wherein the third queue is a queue corresponding to the operation type of the next data when the operation type of the current data is inconsistent with the operation type of the next data.

[0153] The data processing system provided by the embodiment of the present invention can execute the data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0154] Embodiment 6

[0155] Figure 7A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0156] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0157] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data processing method.

[0159] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0160] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0162] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0165] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0167] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: The first node collects at least one target data in the first database, and obtains a data identifier of the target data; The first node sends the target data to the second node according to the data identifier; wherein the second node corresponds to the data identifier one by one; The second node processes the target data to obtain at least one storage data, and sends the storage data to the third node; wherein the quantity of the storage data is consistent with the quantity of the target data; and the third node corresponds to the second node one by one; The third node batch-processes the input data according to the operation type of the input data, and stores the input data in the second database.

2. The method according to claim 1, characterized in that: Before the first node sends the target data to the second node according to the data identifier, the method further includes: Determining, by the first node, a second node identifier of each of the second nodes; The first node sends the target data to the second node according to the data identifier, including: When the data identifier of the target data and the second node identifier satisfy a corresponding relationship, the first node sends the target data to a second node corresponding to the second node identifier.

3. The method according to claim 1, characterized in that: The first node collects at least one target data in the first database, including: The first node monitors data changes in the first database; When there is a data change in the first database, the first node collects the changed data and determines the changed data as the target data; Among them, the number of the first databases is at least one; the number of the first nodes is at least one.

4. The method according to claim 1, characterized in that: The second node includes a first subnode and a second subnode; The second node processes the target data to obtain at least one storage data, including: The first sub-node performs data type conversion on the target data; The second sub-node performs field type mapping on the target data after data type conversion to obtain the storage data.

5. The method according to claim 1, characterized in that The batch processing of the input data according to the operation type of the input data includes: Determine each of the stored data as current data in sequence; In the case where the current data is the first piece of data, the current data is placed in a first queue; wherein the first queue is a queue corresponding to the operation type of the current data; If the current data is not the first piece of data, determining a piece of data previous to the current data; When the operation type of the current data is inconsistent with the operation type of the previous data, the current data is placed in the first queue.

6. The method according to claim 5, characterized in that The method further comprises: When the operation type of the current data is inconsistent with the operation type of the previous data, storing the data in the second queue into the second database; The second queue is a queue corresponding to the operation type of the previous data when the operation type of the current data is inconsistent with the operation type of the previous data.

7. The method according to claim 5, characterized in that After placing the current data into the first queue, the method further includes: Determine the next piece of data of the current data; When the operation type of the current data is consistent with the operation type of the next data, the next data is placed in the first queue; When the operation type of the current data is inconsistent with the operation type of the next data, the data in the first queue is stored in the second database, and the next data is placed in the third queue; The third queue is a queue corresponding to the operation type of the next data when the operation type of the current data is inconsistent with the operation type of the next data.

8. A data processing system, characterized in that: include: A first node, a second node, and a third node; The first node is used to collect at least one target data in the first database and obtain a data identifier of the target data; The first node is further configured to send the target data to a second node according to the data identifier; wherein the second node corresponds to the data identifier in a one-to-one manner; The second node is used to process the target data to obtain at least one storage data, and send the storage data to the third node; wherein the number of the storage data is consistent with the number of the target data; the third node corresponds to the second node one by one; The third node is used to batch process the incoming data according to the operation type of the incoming data, and store the incoming data in the second database.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 7 when executed.

Images