Data processing method and device, equipment and storage medium
By preconfiguring heterogeneous data processing links and multiple processing nodes, heterogeneous data is processed to conform to the target database format, solving the problem of low writing efficiency of heterogeneous data in the prior art and achieving efficient data Transfer and save.
Patent Information
- Application Number
- CN202411456130.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2025-01-14
AI Technical Summary
In the prior art, when writing heterogeneous data to the target database, different data storage format processing logic codes need to be written in a targeted manner, resulting in low efficiency.
By preconfiguring the heterogeneous data processing link, multiple processing nodes (such as data analysis nodes, data filter nodes, and field processing nodes) are used to process heterogeneous data to be transferred to make it conform to the data storage format of the target database, and the After processing, the data is stored in the target database.
It improves the efficiency of writing heterogeneous data into the target database, and significantly improves processing speed and efficiency compared to the way of writing processing logic code.
Smart Images
Figure CN119311752A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art
[0002] In actual applications, when writing data from other data terminals to the target database, the data storage format of other data terminals may not be compatible with the data storage format of the target database. Therefore, the data from other data terminals cannot be directly written to the target database. The data from other data terminals must be processed before being stored in the target database. Obviously, the data from other data terminals is heterogeneous data relative to the data stored in the target database, and the above implementation process can be called a heterogeneous data writing process.
[0003] At present, for heterogeneous data from different data ends, it is usually necessary to write processing logic codes for different data storage formats, so as to transfer the heterogeneous data from different data ends to the target database by executing the processing logic codes. Obviously, this implementation method of writing heterogeneous data into the target database by writing processing logic codes leads to low efficiency in writing heterogeneous data. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data processing method, apparatus, device and storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a data processing method, the method comprising:
[0006] Receiving heterogeneous data to be transferred from the first data terminal;
[0007] Based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed to obtain processed data corresponding to the heterogeneous data to be transferred; wherein the heterogeneous data processing link includes a plurality of processing nodes with an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are used to process the heterogeneous data to be transferred into data that conforms to the data storage format of the target database;
[0008] The processed data is stored in the target database.
[0009] In an optional implementation manner, the processing of the to-be-transferred heterogeneous data based on the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the to-be-transferred heterogeneous data includes:
[0010] Based on the execution order relationship between the processing nodes in the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed in turn based on each processing node to obtain processed data corresponding to the heterogeneous data to be transferred.
[0011] In an optional implementation, the multiple processing nodes include a data parsing node, the data parsing node is pre-configured with a data parsing module determined based on the data storage format of the first data end, and the heterogeneous data processing link corresponding to the first data end processes the heterogeneous data to be transferred to obtain processed data corresponding to the heterogeneous data to be transferred, including:
[0012] Calling a data parsing module corresponding to the data parsing node in the heterogeneous data processing link to parse the heterogeneous data to be transferred and obtain parsing result data;
[0013] Based on the heterogeneous data processing link corresponding to the first data end, the parsing result data is processed to obtain processed data corresponding to the heterogeneous data to be transferred.
[0014] In an optional implementation, the multiple processing nodes include a data filtering node, the data filtering node is pre-configured with a data filter, the data filter is used to filter data based on a data filtering rule corresponding to the target database, the heterogeneous data processing link corresponding to the first data end is used to process the heterogeneous data to be transferred to obtain processed data corresponding to the heterogeneous data to be transferred, including:
[0015] Calling a data filter corresponding to the data filtering node in the heterogeneous data processing link to perform data filtering on the heterogeneous data to be transferred;
[0016] If the heterogeneous data to be transferred successfully passes the data filtering, the heterogeneous data to be transferred is processed based on the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the heterogeneous data to be transferred.
[0017] In an optional implementation, the multiple processing nodes include a field processing node, the field processing node is pre-configured with a field processing module generated based on a field processing rule corresponding to the first data end, and the heterogeneous data processing link corresponding to the first data end is used to process the heterogeneous data to be transferred to obtain processed data corresponding to the heterogeneous data to be transferred, including:
[0018] Calling a field processing module corresponding to the field processing node in the heterogeneous data processing link, and processing the target field in the to-be-transferred heterogeneous data based on the field processing rule to obtain field-processed data;
[0019] Based on the heterogeneous data processing link corresponding to the first data end, the field processed data is processed to obtain processed data corresponding to the heterogeneous data to be transferred.
[0020] In an optional implementation, the plurality of processing nodes include at least one first-type processing node, and there is a corresponding relationship between the at least one first-type processing node and a data processing module generated based on a common code library.
[0021] In an optional implementation, the plurality of processing nodes include at least one second-type processing node, and the at least one second-type processing node is generated based on heterogeneous data features of the first data end.
[0022] In an optional implementation manner, the receiving the to-be-transferred heterogeneous data from the first data terminal includes:
[0023] Obtain the to-be-transferred heterogeneous data from the target message queue in the form of messages; wherein the target message queue is used to store the to-be-transferred heterogeneous data written from the first data end.
[0024] In a second aspect, the present disclosure provides a data processing device, the device comprising:
[0025] A receiving module, used for receiving heterogeneous data to be transferred from the first data terminal;
[0026] A processing module, configured to process the to-be-transferred heterogeneous data based on a heterogeneous data processing link corresponding to the first data end, and obtain processed data corresponding to the to-be-transferred heterogeneous data; wherein the heterogeneous data processing link includes a plurality of processing nodes having an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are configured to process the to-be-transferred heterogeneous data into data in a data storage format that conforms to a target database;
[0027] A storage module is used to store the processed data in the target database.
[0028] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a data processing method as provided in an embodiment of the present disclosure.
[0029] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the data processing method provided by the embodiment of the present disclosure.
[0030] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0031] In the data processing method provided by the embodiment of the present disclosure, heterogeneous data to be transferred is received from the first data end; based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed to obtain processed data corresponding to the heterogeneous data to be transferred, wherein the heterogeneous data processing link includes a plurality of processing nodes with an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are used to process the heterogeneous data to be transferred into data that conforms to the data storage format of the target database; and the processed data is stored in the target database. It can be seen that the embodiment of the present disclosure pre-configures the heterogeneous data processing link for the data end that needs to be connected to the target database in a configured manner, and processes the heterogeneous data to be transferred based on the processing nodes configured in the heterogeneous data processing link. Compared with realizing heterogeneous data writing by writing processing logic code, the configured method provided by the embodiment of the present disclosure can improve the efficiency of writing heterogeneous data to the target database. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0033] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;
[0034] Figure 2 A schematic diagram of a data filtering process provided by an embodiment of the present disclosure;
[0035] Figure 3 A data processing flow chart provided for an embodiment of the present disclosure;
[0036] Figure 4 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;
[0037] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0039] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0040] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0041] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0042] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0043] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0044] At present, for heterogeneous data from different data ends, it is usually necessary to write processing logic codes for different data storage formats, so as to transfer the heterogeneous data from different data ends to the target database by executing the processing logic codes. Obviously, this implementation method of writing heterogeneous data into the target database by writing processing logic codes leads to low efficiency in writing heterogeneous data.
[0045] Therefore, the embodiment of the present disclosure provides a data processing method, specifically, receiving heterogeneous data to be transferred from a first data end; based on the heterogeneous data processing link corresponding to the first data end, processing the heterogeneous data to be transferred to obtain processed data corresponding to the heterogeneous data to be transferred, wherein the heterogeneous data processing link includes a plurality of processing nodes with an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are used to process the heterogeneous data to be transferred into data that conforms to the data storage format of the target database; and storing the processed data in the target database. It can be seen that the embodiment of the present disclosure pre-configures the heterogeneous data processing link for the data end that needs to be connected to the target database in a configured manner, and processes the heterogeneous data to be transferred based on the processing nodes configured in the heterogeneous data processing link. Compared with realizing heterogeneous data writing by writing processing logic code, the configured method provided by the embodiment of the present disclosure can improve the efficiency of writing heterogeneous data to the target database.
[0046] Based on this, an embodiment of the present disclosure provides a data processing method, which is introduced below in conjunction with a specific embodiment.
[0047] Figure 1 The present invention provides a flowchart of a data processing method provided in an embodiment of the present invention. The method can be executed by a data processing device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:
[0048] S101: receiving heterogeneous data to be transferred from a first data terminal.
[0049] The data processing method provided in the embodiments of the present disclosure can be applied to any system or client with heterogeneous data transfer requirements. For example, the data processing method provided in the embodiments of the present disclosure can be used to transfer heterogeneous data from different data terminals to a target database corresponding to a retrieval system, so that users can use the target database to perform operations such as data retrieval.
[0050] In the embodiment of the present disclosure, the data storage format of the first data end is different from the data storage format of the target database. Therefore, after receiving the heterogeneous data to be transferred from the first data end, the heterogeneous data to be transferred needs to be processed. The embodiment of the present disclosure does not limit the storage method of the heterogeneous data to be transferred. The heterogeneous data to be transferred can be, for example, data stored in the database of the first data end.
[0051] In an optional implementation, the heterogeneous data to be transferred can also be obtained from the target message queue by means of messages. Exemplarily, asynchronous message transmission can be implemented using the target message queue and consumer mode, wherein the target message queue is used to store the heterogeneous data to be transferred written from the first data end, and the consumer is used to retrieve the data from the target message queue and process the heterogeneous data to be transferred based on the heterogeneous data processing link.
[0052] Specifically, multiple heterogeneous data written from the first data end are first stored in the target message queue in the form of messages, and then the consumer obtains the heterogeneous data to be transferred from the target message queue in the form of messages. Furthermore, based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed to obtain the processed data corresponding to the heterogeneous data to be transferred.
[0053] Among them, each heterogeneous data stored in the target message queue corresponds to a message. After the consumer obtains the heterogeneous data to be transferred from the messages corresponding to multiple heterogeneous data, the consumer can process the heterogeneous data to be transferred based on the pre-configured heterogeneous data processing link.
[0054] S102: Based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed to obtain processed data corresponding to the heterogeneous data to be transferred.
[0055] The heterogeneous data processing link includes a plurality of processing nodes pre-configured for the first data end and having an execution order relationship, and the plurality of processing nodes are used to process the heterogeneous data to be transferred into data conforming to the data storage format of the target database.
[0056] In the embodiment of the present disclosure, since a corresponding heterogeneous data processing link is pre-configured for the first data end, after receiving the heterogeneous data to be transferred from the first data end, the heterogeneous data processing link can be used to implement the transfer processing flow of the heterogeneous data to be transferred on the first data end.
[0057] In the embodiment of the present disclosure, the heterogeneous data processing link includes multiple processing nodes with an execution order relationship pre-configured for the first data end. By executing each processing node in the heterogeneous data processing link, the heterogeneous data to be transferred can be processed into data that conforms to the data storage format of the target database.
[0058] The target database may include a relational database, a non-relational database, a graph database, etc. Specifically, a relational database usually stores data based on a table structure, and a graph database stores data based on a graph structure.
[0059] In an optional implementation, since a plurality of processing nodes having an execution order relationship are pre-configured in the heterogeneous data processing link, the heterogeneous data to be transferred can be processed in turn based on each processing node based on the execution order relationship between the processing nodes in the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the heterogeneous data to be transferred.
[0060] Exemplarily, after receiving the heterogeneous data to be transferred from the first data end, assuming that the heterogeneous data processing link corresponding to the first data end includes three processing nodes having an execution order relationship, namely, a data parsing node, a data filtering node, and a field processing node, the heterogeneous data to be transferred can be transferred by sequentially executing the data parsing node, the data filtering node, and the field processing node to obtain processed data that conforms to the data storage format of the target database.
[0061] In the embodiment of the present disclosure, after the heterogeneous data to be transferred is processed into data conforming to the data storage format of the target database by each processing node in the heterogeneous data processing link, the processed data can be stored in the target database.
[0062] In an optional implementation, DAG (Directed Acyclic Graph) can be used to build heterogeneous data processing links, where DAG is a graph structure composed of points and edges, each edge is directed, and there is no loop in the entire graph. Therefore, the points on the DAG can be used to represent processing nodes, and the directed edges between the nodes can be used to represent the execution order relationship between the processing nodes, thereby realizing the construction and visualization of heterogeneous data processing links.
[0063] In actual applications, technicians can personalize the configuration of each processing node in the heterogeneous data processing link based on DAG based on the heterogeneous data characteristics of the first data end, so that the heterogeneous data processing link after personalization can better meet the heterogeneous data transfer requirements of the first data end.
[0064] S103: Storing the processed data in the target database.
[0065] In the embodiment of the present disclosure, after the heterogeneous data to be transferred at the first data end is processed using a heterogeneous data processing link to obtain processed data that conforms to the data storage format of the target database, the processed data can be stored in the target database, thereby realizing the function of transferring the heterogeneous data to be transferred to the target database in a configured manner.
[0066] In an optional implementation, the heterogeneous data processing link corresponding to the first data end may also define identification information of a target database corresponding to the processed data. After the heterogeneous data to be transferred is processed based on the heterogeneous data processing link corresponding to the first data end and the corresponding processed data is obtained, the target database may be determined from multiple databases based on the identification information, and the processed data may be stored in the target database.
[0067] In the data method provided by the embodiment of the present disclosure, heterogeneous data to be transferred is received from the first data end; based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed to obtain processed data corresponding to the heterogeneous data to be transferred, wherein the heterogeneous data processing link includes a plurality of processing nodes with an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are used to process the heterogeneous data to be transferred into data that conforms to the data storage format of the target database; and the processed data is stored in the target database. It can be seen that the embodiment of the present disclosure pre-configures the heterogeneous data processing link for the data end that needs to be connected to the target database in a configured manner, and processes the heterogeneous data to be transferred based on the processing nodes configured in the heterogeneous data processing link. Compared with realizing heterogeneous data writing by writing processing logic code, the configured method provided by the embodiment of the present disclosure can improve the efficiency of writing heterogeneous data to the target database.
[0068] In actual applications, since the data storage format of the first data end may be different from the data storage format of the target database, in order to successfully connect the heterogeneous data to be transferred from the first data end to the target database, after receiving the heterogeneous data to be transferred from the first data end, the data parsing node in the heterogeneous data processing link can also be used to parse and process the heterogeneous data to be transferred.
[0069] In the embodiment of the present disclosure, since the data parsing node is pre-configured with a data parsing module determined based on the data storage format of the first data end, in the process of processing the heterogeneous data to be transferred based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred can be parsed by calling the data parsing module corresponding to the data parsing node to obtain the parsed result data.
[0070] In the embodiment of the present disclosure, after parsing the heterogeneous data to be transferred based on the data parsing node and obtaining the parsing result data corresponding to the heterogeneous data to be transferred, the parsing result data can be further processed based on other processing nodes in the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the data to be transferred, and the processed data can be stored in the target database.
[0071] In the embodiment of the present disclosure, the corresponding data parsing module is pre-configured for the data parsing node, so that the function of parsing the heterogeneous data to be transferred can be realized by calling the data parsing module later. In addition, for different data terminals, the corresponding data parsing module can be configured for each data terminal based on the data storage format of each data terminal, and then the data parsing module is used to convert the heterogeneous data to be transferred of each data terminal into the data storage format common to the target database.
[0072] In an optional implementation, the multiple processing nodes of the heterogeneous data processing link further include a data filtering node, through which the to-be-transferred heterogeneous data that does not need to be stored in the target database can be filtered out.
[0073] In the disclosed embodiment, a corresponding data filter may be pre-configured for a data filtering node, wherein the data filter is used to perform data filtering processing on the heterogeneous data to be transferred based on the data filtering rules corresponding to the target database, so as to filter out unnecessary heterogeneous data.
[0074] Specifically, the heterogeneous data to be transferred can be filtered by calling the data filter corresponding to the data filtering node; if the heterogeneous data to be transferred successfully passes the data filtering process, the heterogeneous data to be transferred is processed based on other processing nodes according to the execution order relationship between the processing nodes in the heterogeneous data processing link to obtain the processed data corresponding to the heterogeneous data to be transferred, and the processed data is stored in the target database.
[0075] In an optional implementation, a universal data filter can be pre-designed, and corresponding data filters can be configured for each data terminal in a configurable manner, so as to use the corresponding data filter of the data terminal to filter the heterogeneous data to be transferred and stored on the data terminal. Among them, the data filtering rules can be defined based on strategies such as Boolean expression rule engine, random sampling, hash modulus, black and white lists, etc.
[0076] In an optional implementation, the multiple processing nodes of the heterogeneous data processing link also include a field processing node. Since the field processing node is pre-configured with a field processing module generated based on the field processing rules corresponding to the first data end, field processing can be performed on the heterogeneous data to be transferred by calling the field processing module.
[0077] Specifically, the field processing module corresponding to the field processing node in the heterogeneous data processing link is called, and the target field in the heterogeneous data to be transferred is processed based on the field processing rules to obtain field processed data; further, the field processed data is processed based on other processing nodes in the heterogeneous data processing link to obtain processed data corresponding to the heterogeneous data to be transferred out, and the processed data is stored in the target database.
[0078] In the embodiment of the present disclosure, the field processing rules corresponding to the first data end may include processing rules such as aggregation calculation and field conversion.
[0079] The aggregation calculation rule is used to map and transform multiple fields in the first data end to obtain mapped fields. Specifically, since some fields in the target database may be aggregated from multiple fields in the heterogeneous data to be transferred on the first data end, the aggregation calculation rule is configured for the field processing node in advance, so that when the field processing node is subsequently executed, the target field in the heterogeneous data to be transferred can be processed based on the aggregation calculation rule to obtain field processed data.
[0080] The field conversion rule is used to convert the target field in the heterogeneous data to be transferred into the field type corresponding to the target database, for example, converting the target field from an integer type to a string type.
[0081] In an optional implementation, the data access node may be used to receive the heterogeneous data to be transferred from the first data terminal, and the data output node may be used to store the processed data corresponding to the heterogeneous data to be transferred into the target database.
[0082] like Figure 2 FIG. 1 is a flow chart of a data filtering process provided by an embodiment of the present disclosure. The heterogeneous data processing link corresponding to the first data end may include multiple processing nodes executed in the following order, namely, a data parsing node, a data filtering node, and a field processing node.
[0083] Step 1: Receive heterogeneous data to be transferred from a first data terminal.
[0084] Step 2: Call the data parsing module corresponding to the data parsing node to parse the heterogeneous data to be transferred and obtain the parsing result data.
[0085] Step 3: Call the data filter corresponding to the data filtering node to filter the parsing result data corresponding to the heterogeneous data to be transferred.
[0086] Step 4: Determine whether the parsing result data successfully passes the data filtering. If yes (ie, not filtered out), execute step 5; if no, execute step 7.
[0087] Step 5: Call the field processing module corresponding to the field processing node, process the target field in the parsing result data based on the field processing rules, and obtain the field processed data.
[0088] Step 6: Store the processed field data into the target database.
[0089] Step 7: End execution.
[0090] It can be seen that the embodiment of the present disclosure pre-configures a heterogeneous data processing link for the data end that needs to be connected to the target database in a configured manner, and processes the heterogeneous data to be transferred based on the processing nodes configured in the heterogeneous data processing link, thereby improving the efficiency of writing heterogeneous data to the target database.
[0091] like Figure 3 As shown, it is a data processing flow chart provided by an embodiment of the present disclosure, which is specifically explained by taking a retrieval system as an example, wherein a write service is deployed in the retrieval system for storing the heterogeneous data to be transferred from the first data end to the target database of the retrieval system.
[0092] Specifically, during the startup of the write service, steps such as initializing the processing node, pulling and loading the configuration, and executing the heterogeneous data processing link are performed.
[0093] Among them, initializing the processing nodes specifically includes: initializing the pre-defined processing nodes, and initializing the data processing modules corresponding to each processing node, so that when processing is subsequently performed based on the heterogeneous data processing link, each processing node in the heterogeneous data processing link can be executed by calling the data processing module corresponding to the processing node.
[0094] Pulling and loading configuration refers to loading the configuration of the heterogeneous data processing link to initialize the heterogeneous data processing link. After initializing the heterogeneous data processing link corresponding to the first data end and the consumer corresponding to the target message queue, the consumer is used to retrieve and process the messages in the target message queue. Specifically, after the heterogeneous data A to be transferred from the first data end is stored in the target message queue in the form of a message, the consumer obtains the heterogeneous data A to be transferred from the target message queue in the form of a message, and processes the heterogeneous data A to be transferred based on the heterogeneous data processing link corresponding to the first data end to obtain the processed data corresponding to the heterogeneous data A to be transferred.
[0095] After pulling and loading the configuration of the heterogeneous data processing link, the heterogeneous data processing link is instantiated according to the configuration, so as to obtain processed data corresponding to the heterogeneous data to be transferred by executing the heterogeneous data processing link corresponding to the first data end, and store the processed data in the target database.
[0096] It can be seen that after initializing the heterogeneous data processing link, the embodiment of the present disclosure can process the heterogeneous data to be transferred at the first data end into data that conforms to the data storage format of the target database based on the processing nodes configured in the heterogeneous data processing link.
[0097] In an optional implementation, the heterogeneous data processing link corresponding to the first data end may include multiple processing nodes of different types. Specifically, the multiple processing nodes may include at least one first type of processing node, which is also called a general processing node, that is, a processing node that is common to different data ends.
[0098] In the disclosed embodiment, at least one first type processing node has a corresponding relationship with a data processing module generated based on a common code library. The common code library refers to a code library applicable to all data terminals and does not involve business-related codes. The first type processing node may include a data parsing node, a data filtering node, etc. Exemplarily, there is a corresponding relationship between the data parsing node and the data parsing module, and the data parsing module is generated based on the data parsing processing code in the common code library, and is used to parse and process the heterogeneous data to be transferred.
[0099] It can be seen that the embodiment of the present disclosure predefines general processing nodes and configures corresponding data processing modules for each general processing node, so that when processing is subsequently performed based on heterogeneous data processing links, heterogeneous data can be transferred and processed by calling the data processing modules corresponding to the processing nodes, thereby achieving the reusability of the general processing nodes.
[0100] The multiple processing nodes also include at least one second type processing node, which is also called a custom processing node, that is, a processing node defined for the first data end and is only suitable for processing heterogeneous data to be transferred from the first data end.
[0101] In the embodiment of the present disclosure, at least one second type of processing node can be generated based on the heterogeneous data features of the first data end. The second type of processing node can include, for example, a field processing node, etc. Exemplarily, the field processing node can be generated using the business code library of the first data end, and is used to aggregate business-related fields in the first data end. Since the heterogeneous data of different data ends have different characteristics, the processing node generated based on the heterogeneous data features of the first data end can only be used to process the heterogeneous data to be transferred from the first data end, and cannot be used to process the heterogeneous data of other data ends.
[0102] It can be seen that the embodiment of the present disclosure can meet the transfer requirements of heterogeneous data of different data terminals by configuring corresponding personalized processing nodes (also known as business-related nodes) for different data terminals. In addition, the configuration method can also reduce the cost of heterogeneous data transfer and maintenance, and improve the efficiency of writing heterogeneous data to the target database.
[0103] In order to implement the above embodiments, the present disclosure also proposes a data processing device. Figure 4The structure diagram of a data processing device provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 4 As shown, the device comprises:
[0104] The receiving module 401 is used to receive heterogeneous data to be transferred from the first data terminal;
[0105] A processing module 402 is used to process the to-be-transferred heterogeneous data based on the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the to-be-transferred heterogeneous data; wherein the heterogeneous data processing link includes a plurality of processing nodes having an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are used to process the to-be-transferred heterogeneous data into data in a data storage format that conforms to a target database;
[0106] The storage module 403 is used to store the processed data in the target database.
[0107] In an optional implementation, the processing module includes:
[0108] The first processing submodule is used to process the heterogeneous data to be transferred based on each processing node in turn based on the execution order relationship between each processing node in the heterogeneous data processing link corresponding to the first data end, so as to obtain processed data corresponding to the heterogeneous data to be transferred.
[0109] In an optional implementation, the multiple processing nodes include a data parsing node, the data parsing node is pre-configured with a data parsing module determined based on the data storage format of the first data end, and the processing module includes:
[0110] A parsing processing submodule, used for calling a data parsing module corresponding to the data parsing node in the heterogeneous data processing link, parsing the heterogeneous data to be transferred, and obtaining parsing result data;
[0111] The second processing submodule is used to process the parsing result data based on the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the heterogeneous data to be transferred.
[0112] In an optional implementation, the multiple processing nodes include a data filtering node, the data filtering node is pre-configured with a data filter, the data filter is used to filter data based on a data filtering rule corresponding to the target database, and the processing module includes:
[0113] A filtering processing submodule, used for calling a data filter corresponding to the data filtering node in the heterogeneous data processing link to perform data filtering on the heterogeneous data to be transferred;
[0114] The third processing submodule is used to process the heterogeneous data to be transferred based on the heterogeneous data processing link corresponding to the first data end if the heterogeneous data to be transferred successfully passes the data filtering, so as to obtain processed data corresponding to the heterogeneous data to be transferred.
[0115] In an optional implementation, the multiple processing nodes include a field processing node, and the field processing node is pre-configured with a field processing module generated based on a field processing rule corresponding to the first data end, and the processing module includes:
[0116] A field processing submodule, used for calling the field processing module corresponding to the field processing node in the heterogeneous data processing link, processing the target field in the to-be-transferred heterogeneous data based on the field processing rule, and obtaining field-processed data;
[0117] The fourth processing submodule is used to process the field processed data based on the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the heterogeneous data to be transferred.
[0118] In an optional implementation, the plurality of processing nodes include at least one first-type processing node, and there is a corresponding relationship between the at least one first-type processing node and a data processing module generated based on a common code library.
[0119] In an optional implementation, the plurality of processing nodes include at least one second-type processing node, and the at least one second-type processing node is generated based on heterogeneous data features of the first data end.
[0120] In an optional implementation manner, the receiving module includes:
[0121] The acquisition submodule is used to acquire the to-be-transferred heterogeneous data from the target message queue in the form of messages; wherein the target message queue is used to store the to-be-transferred heterogeneous data written from the first data end.
[0122] In the data processing device provided by the embodiment of the present disclosure, heterogeneous data to be transferred is received from the first data end; based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed to obtain processed data corresponding to the heterogeneous data to be transferred, wherein the heterogeneous data processing link includes a plurality of processing nodes with an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are used to process the heterogeneous data to be transferred into data that conforms to the data storage format of the target database; and the processed data is stored in the target database. It can be seen that the embodiment of the present disclosure pre-configures the heterogeneous data processing link for the data end that needs to be connected to the target database in a configured manner, and processes the heterogeneous data to be transferred based on the processing nodes configured in the heterogeneous data processing link. Compared with realizing heterogeneous data writing by writing processing logic code, the configured method provided by the embodiment of the present disclosure can improve the efficiency of writing heterogeneous data to the target database.
[0123] In order to implement the above embodiments, the present disclosure further proposes a computer program product, including a computer program / instruction, which implements the data processing method in the above embodiments when executed by a processor.
[0124] In addition, the present disclosure also provides a data processing device, see Figure 5 As shown, it may include:
[0125] Processor 501, memory 502, input device 503 and output device 504. The number of processors 501 in the data processing device can be one or more. Figure 5 In some embodiments of the present disclosure, the processor 501, the memory 502, the input device 503 and the output device 504 may be connected via a bus or other means, wherein: Figure 5 The example of connecting through bus is taken in the following.
[0126] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing of the data processing device by running the software programs and modules stored in the memory 502. The memory 502 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory 502 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 503 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the data processing device.
[0127] Specifically in this embodiment, the processor 501 will load the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 501 will run the applications stored in the memory 502, thereby realizing various functions of the above-mentioned data processing device.
[0128] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0129] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, characterized in that: include: Receiving heterogeneous data to be transferred from the first data terminal; Based on the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed to obtain processed data corresponding to the heterogeneous data to be transferred; wherein the heterogeneous data processing link includes a plurality of processing nodes with an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are used to process the heterogeneous data to be transferred into data that conforms to the data storage format of the target database; The processed data is stored in the target database.
2. The method according to claim 1, characterized in that: The processing of the to-be-transferred heterogeneous data based on the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the to-be-transferred heterogeneous data includes: Based on the execution order relationship between the processing nodes in the heterogeneous data processing link corresponding to the first data end, the heterogeneous data to be transferred is processed in turn based on each processing node to obtain processed data corresponding to the heterogeneous data to be transferred.
3. The method according to claim 1, characterized in that The multiple processing nodes include a data parsing node, the data parsing node is pre-configured with a data parsing module determined based on the data storage format of the first data end, and the heterogeneous data processing link corresponding to the first data end processes the heterogeneous data to be transferred to obtain processed data corresponding to the heterogeneous data to be transferred, including: Calling a data parsing module corresponding to the data parsing node in the heterogeneous data processing link to parse the heterogeneous data to be transferred and obtain parsing result data; Based on the heterogeneous data processing link corresponding to the first data end, the parsing result data is processed to obtain processed data corresponding to the heterogeneous data to be transferred.
4. The method according to claim 1, characterized in that The multiple processing nodes include a data filtering node, the data filtering node is pre-configured with a data filter, the data filter is used to filter data based on the data filtering rule corresponding to the target database, the heterogeneous data processing link corresponding to the first data end is used to process the heterogeneous data to be transferred, and the processed data corresponding to the heterogeneous data to be transferred is obtained, including: Calling a data filter corresponding to the data filtering node in the heterogeneous data processing link to perform data filtering on the heterogeneous data to be transferred; If the heterogeneous data to be transferred successfully passes the data filtering, the heterogeneous data to be transferred is processed based on the heterogeneous data processing link corresponding to the first data end to obtain processed data corresponding to the heterogeneous data to be transferred.
5. The method according to claim 1, characterized in that The multiple processing nodes include a field processing node, the field processing node is pre-configured with a field processing module generated based on a field processing rule corresponding to the first data end, and the heterogeneous data processing link corresponding to the first data end is used to process the heterogeneous data to be transferred to obtain processed data corresponding to the heterogeneous data to be transferred, including: Calling a field processing module corresponding to the field processing node in the heterogeneous data processing link, and processing the target field in the to-be-transferred heterogeneous data based on the field processing rule to obtain field-processed data; Based on the heterogeneous data processing link corresponding to the first data end, the field processed data is processed to obtain processed data corresponding to the heterogeneous data to be transferred.
6. The method according to claim 1, characterized in that The plurality of processing nodes include at least one first-type processing node, and a corresponding relationship exists between the at least one first-type processing node and a data processing module generated based on a common code library.
7. The method according to claim 1, characterized in that The plurality of processing nodes include at least one second-type processing node, and the at least one second-type processing node is generated based on heterogeneous data features of the first data end.
8. The method according to claim 1, characterized in that: The receiving the to-be-transferred heterogeneous data from the first data terminal includes: Obtain the to-be-transferred heterogeneous data from the target message queue in the form of messages; wherein the target message queue is used to store the to-be-transferred heterogeneous data written from the first data end.
9. A data processing device, characterized in that: The device comprises: A receiving module, used for receiving heterogeneous data to be transferred from the first data terminal; A processing module, configured to process the to-be-transferred heterogeneous data based on a heterogeneous data processing link corresponding to the first data end, and obtain processed data corresponding to the to-be-transferred heterogeneous data; wherein the heterogeneous data processing link includes a plurality of processing nodes having an execution order relationship pre-configured for the first data end, and the plurality of processing nodes are configured to process the to-be-transferred heterogeneous data into data in a data storage format that conforms to a target database; A storage module is used to store the processed data in the target database.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 8.
11. A data processing device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.
12. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.