Device and method for real-time processing of online production data in streaming mode

Abstract

The present invention discloses a device and method for real-time processing of online production data in a streaming manner. In the present invention, online production data of multiple data sources read in real time based on a streaming processing method can be transferred to a single data source sub-table in a key-value pair format, and the single data source sub-table in the key-value pair format of each data source can be periodically merged into a multi-data source wide table to achieve the integration of multiple data sources, thereby achieving the integration of multiple data sources with considerable consumption efficiency and under the premise of ensuring data integrity. Thus, the present invention can achieve the integration of multiple data sources with low development cost with considerable consumption efficiency and under the premise of ensuring data integrity to meet the demand for data real-time performance. Moreover, the multi-data source wide table can be provided to a distributed file system to achieve relatively simple data verification, and can also be fed back to a distributed publish-subscribe message system for downstream business use.

Description

Technical Field

[0001] The present invention relates to the field of data processing of big data, and in particular to a device for streaming real-time processing of online production data, a method for streaming real-time processing of online production data, a data processing device and a non-transient computer-readable storage medium. Background Art

[0002] With the development of big data, the amount of data that needs to be processed is also growing. Based on this situation, the traditional offline T+1 processing method is difficult to guarantee timeliness and difficult to meet the real-time data requirements from a business perspective. Summary of the invention

[0003] An embodiment of the present invention provides a device for real-time processing of online production data in a streaming manner, comprising:

[0004] A data access module, which reads online production data from multiple data sources from a distributed publish-subscribe messaging system based on a streaming processing method;

[0005] A data buffer module, which transfers the online production data of each data source into a single data source sub-table in a key-value format;

[0006] A data merging module is used to periodically merge the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table.

[0007] Optionally, the data access module reads online production data by calling a distributed processing engine for stream data and batch data.

[0008] Optionally, the online production data of each data source is stored in partitions according to primary key pairs in the distributed publish-subscribe message system, and the data access module reads the online production data of each data source from each partition in parallel.

[0009] Optionally, the data buffer module transfers the single data source sub-table in the key-value pair format in a distributed column-oriented database.

[0010] Optionally, the data buffer module sets a primary key identifier, a database row key, inter-table association fields, and whether there is a modification content field for an update operation in a corresponding single data source sub-table according to a configuration file pre-set for each data source.

[0011] Optionally, the data buffer module directly converts the newly added data in the online production data into a key-value pair format, converts the updated data into a key-value pair format after overwriting the data, and converts the deleted data into a key-value pair format after adding a deletion mark; and the data buffer module stores the newly added data, updated data, and deleted data in the key-value pair format into the corresponding single data source sub-table, wherein if there are multiple corresponding database row keys in the same single data source sub-table, the data buffer module splices the multiple corresponding database row keys; if there is a modification content field for the update operation in the single data source sub-table, the data buffer module makes an update decision.

[0012] Optionally, the data merging module merges multiple data sources in a relational database to obtain a wide table.

[0013] Optionally, the data merging module realizes merging of wide tables from multiple data sources by associating tables with each other based on structured query language.

[0014] Optionally, the data merging module periodically merges multiple data sources in a preset time window to obtain a wide table.

[0015] Optionally, it further includes: a data production module, which stores the multi-data source wide table into a distributed file system, and / or converts the multi-data source wide table into a log format and then provides it to a distributed publish-subscribe message system.

[0016] Another embodiment of the present invention provides a method for real-time processing of online production data in a streaming manner, comprising:

[0017] Read online production data from multiple data sources from a distributed publish-subscribe messaging system based on streaming processing;

[0018] Transfer the online production data of each data source into a single data source table in key-value format;

[0019] Periodically merge the single data source tables in the key-value pair format of each data source into a multi-data source wide table.

[0020] Optionally, reading online production data from multiple data sources includes: reading the online production data by calling a distributed processing engine for stream data and batch data.

[0021] Optionally, the online production data of each data source is stored in partitions according to primary keys in the distributed publish-subscribe message system, and reading the online production data of multiple data sources includes: reading the online production data of each data source from each partition in parallel.

[0022] Optionally, transferring the online production data of each data source into a single data source sub-table in a key-value pair format includes: transferring the single data source sub-table in a key-value pair format in a distributed column-oriented database.

[0023] Optionally, transferring the online production data of each data source into a single data source sub-table in a key-value pair format includes: setting a primary key identifier, a database row key, inter-table association fields, and whether there is a modification content field for an update operation in the corresponding single data source sub-table according to a configuration file pre-set for each data source; directly converting the newly added data in the online production data into a key-value pair format, converting the updated data into a key-value pair format after overwriting the data, and converting the deleted data into a key-value pair format after adding a deletion mark; storing the newly added data, updated data, and deleted data in the key-value pair format into the corresponding single data source sub-table, wherein, if there are multiple corresponding database row keys for the same single data source sub-table, the multiple corresponding database row keys are concatenated; and, if there is a modification content field for an update operation in the single data source sub-table, an update decision is made.

[0024] Optionally, merging the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table includes: merging in a relational database to obtain the multi-data source wide table.

[0025] Optionally, merging the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table includes: merging the multi-data source wide tables by associating tables between them based on structured query language.

[0026] Optionally, the method further includes: storing the multi-data source wide table in a distributed file system, and / or converting the multi-data source wide table into a log format and then providing it to a distributed publish-subscribe message system.

[0027] Another embodiment of the present invention provides a data processing device, comprising a processor, wherein the processor is configured to execute the steps of the above method.

[0028] Another embodiment of the present invention provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores instructions, wherein the instructions are used for the processor to execute the steps of the above method.

[0029] As can be seen above, based on the above embodiments, the online production data of multiple data sources read in real time based on the streaming processing method can be transferred to a single data source sub-table in the key-value pair format, and the single data source sub-table in the key-value pair format of each data source can be periodically merged into a multi-data source wide table to achieve the integration of multiple data sources, thereby achieving the integration of multiple data sources with considerable consumption efficiency and under the premise of ensuring data integrity to meet the demand for data real-time performance. Moreover, as a further continuation, the multi-data source wide table can be provided to the distributed file system to achieve relatively simple data verification, and can also be fed back to the distributed publish-subscribe message system for downstream business use. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of an exemplary structure of a device for real-time streaming processing of online production data in one embodiment;

[0031] Figure 2 For Figure 1 A schematic diagram of the principle of the data access module in the device shown;

[0032] Figure 3 For Figure 1 A schematic diagram of the principle of the data buffer module in the device shown;

[0033] Figure 4 For Figure 1 A schematic diagram of the principle of the data merging module in the device shown;

[0034] Figure 5 It is a schematic diagram of an exemplary flow chart of a method for real-time streaming processing of online production data in one embodiment;

[0035] Figure 6 For Figure 5 An extended flow diagram of the method shown;

[0036] Figure 7 For Figure 5 A flowchart of a specific example of the method shown;

[0037] Figure 8 FIG. 4 is a schematic diagram of the structure of a data processing device in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0039] Figure 1 FIG. 1 is a schematic diagram of an exemplary structure of a device for real-time streaming processing of online production data in an embodiment. Figure 1 In one embodiment, the device for real-time streaming processing of online production data may include: a data access module 11, a data buffer module 12, a data merging module 13, and a data production module 14.

[0040] The data access module 11 can read online production data from multiple data sources from the distributed publish-subscribe messaging system 20 based on a streaming processing method.

[0041] Among them, the above-mentioned distributed publish-subscribe messaging system 20 can be an open source stream processing platform with high throughput such as Kafka, which has the ability to process all action stream data from multiple data sources. The online production data accessible in the distributed publish-subscribe messaging system 20 can be in log format, such as the binary log (binlog) included in the conversation (Topic) data.

[0042] Moreover, the data access module 11 can call a distributed processing engine for stream data and batch data, such as Fast Flexible Connection (Flink), to read online production data from the distributed publish-subscribe messaging system 20 through the called distributed processing engine. Compared with Storm, which is commonly used in traditional stream processing, Flink has more functions, such as machine learning, batch processing, stream processing, etc., and Flink has higher data consumption efficiency.

[0043] In addition, the distributed publish-subscribe messaging system 20 can partition and store the online production data of each data source according to the primary key, and accordingly, read the online production data of each data source in parallel from each partition to improve the efficiency of data consumption and ensure the integrity and consistency of data with the same primary key.

[0044] Figure 2 For Figure 1 The schematic diagram of the data access module in the device shown in FIG. Figure 2 And combined with Figure 1 Assuming that the distributed publish-subscribe messaging system 20 is connected to M data sources 20_1 to 20_M (M is a positive integer greater than 1), and the online production data of each of the M data sources 20_1 to 20_M has at most N primary keys (N is a positive integer greater than 1), then for each of the M data sources 20_1 to 20_M, there are at most N primary key partitions in the distributed publish-subscribe messaging system 20. Accordingly, for the online production data of each of the M data sources 20_1 to 20_M, the data access module 11 can read the corresponding partitions in parallel.

[0045] Figure 3 For Figure 1 Schematic diagram of the data buffer module in the device shown. Figure 3 And combined with Figure 1, the data buffer module 12 can transfer the online production data of each data source into a single data source sub-table in a key-value format. The data buffer module 12 can transfer the single data source sub-table in a key-value format in a distributed column-oriented database such as HBase. The online production data of each data source can correspond to a single data source sub-table, so the number of single data source sub-tables can be M.

[0046] For the single data source sub-table 400 corresponding to each data source, the corresponding configuration file 40 can be processed in advance, and the data buffer module can set the primary key identifier 41, the database row key (Rowkey) 42, the modification content field (UpdateCol) 43 for the update operation, and the inter-table association field 44 in the corresponding single data source sub-table 400 according to the configuration file 40 pre-set for each data source. Among them, the modification content field 43 for the update operation is an optional configuration item, and whether the corresponding data needs to be modified can be judged later based on whether this configuration item is in the single data source sub-table 400.

[0047] The configuration files 40 of different data sources may contain the same configuration items, but the specific contents of the configuration items may be different. For example, the configuration files 50 of each data source all set the configuration item of the primary key for the single data source table, but the values ​​of the primary key in the configuration item may be different.

[0048] By configuring the single data source sub-table 400 by loading the configuration file 40, it is easy to adjust the single data source sub-table 400 according to the needs. That is, the single data source sub-table 400 can be flexibly adjusted by only changing the configuration file 40 without changing the program settings of the data buffer module 12.

[0049] The single data source sub-table 400 configured according to the configuration file 40 may be stored in a cache medium 121 such as Google Cache for subsequent reading.

[0050] Still see Figure 3 , taking HBase as an example for the distributed column-oriented database 120, and the operation types corresponding to the online production data 50 may include three types: insert, update, and delete, wherein the delete operation may be a logical delete, and accordingly:

[0051] For the newly added data corresponding to the adding operation in the online production data 50, the data buffer module 12 can directly perform key-value pair format conversion 53 on it;

[0052] For the update data involved in the update operation in the online production data 50, the data buffer module 12 may perform the key-value pair format conversion 53 after performing the data overwriting 51 on it, wherein the data overwriting 51 means overwriting the original data with the latest data;

[0053] For deleted data, the data buffer module 12 may perform key-value pair format conversion after adding a deletion mark 52 to the deleted data.

[0054] After the above format conversion, the data buffer module 12 reads the single data source sub-table 400 from the cache medium 121 to the database 120, and can store the newly added data, updated data, and deleted data in the key-value pair format into the corresponding single data source sub-table 400 stored in the distributed column-oriented database 120.

[0055] If the database row key 42 configured for the same single data source sub-table 400 contains more than one Rowkey, in order to match the storage format of HBase, the data buffer module 12 needs to perform row key concatenation 54, that is, concatenate multiple corresponding Rowkeys together through symbols such as underscore "_".

[0056] In addition, if there is a modification content field 43 for the update operation in the single data source sub-table 400, the data buffer module 12 also needs to perform an update judgment 55. For example, the data buffer module 12 first determines whether the business operation time corresponding to the current data is later than or equal to the business operation time of the data stored in the corresponding single data source sub-table 400 in HBase 120. If so, it indicates that the current data is the latest data, and the stored data in the corresponding single data source sub-table 400 in HBase 120 is updated to keep the data consistent; otherwise, the corresponding single data source sub-table 400 in HBase 120 is not updated, and at the same time, it can be considered that the current data is dirty data that should be overwritten and can be discarded.

[0057] By analogy, HBase 120 will store all single-data source shard tables 400 in real time in full according to this rule.

[0058] The above business operation time can be determined by comparing the "mid sequence" in the key-value pair format. For details, please refer to the following example:

[0059] Assume that the primary key of online production data 1 in key-value pair format is key=16777921_80463060328_8226277, and the key value of the primary key includes a mid sequence and an opt field. The value of the mid sequence is "200135521945", and the opt field is "INSERT" indicating a new operation.

[0060] Assume again that the primary key of online production data 2 in key-value pair format is key=16777921_80463060328_8226277, and the key value of the primary key also includes the mid sequence and opt field. The value of the mid sequence is "200135521951", and the opt field is "UPDATE" indicating an update operation.

[0061] The primary key keys of the production data 1 and 2 on the above two lines are both composed of the primary key identifier, the old identifier, and the product ID, and the values ​​of the primary key keys of the production data 1 and 2 on the above two lines are the same.

[0062] The key values ​​of the above two online production data 1 and 2 are both in JSON (JavaScript Object Notation, JS object notation) string format, which contains the content of online production data 1 and 2.

[0063] As can be seen above, the opt field of online production data 1 is "INSERT" indicating a new operation, and the opt field of online production data 2 is "UPDATE" indicating an update operation. Through the modified content field (UpdateCol) for the update operation, the data buffer module 12 can determine whether the online production data 2 is indeed new data through the mid sequence that increases over time, that is, update judgment 55.

[0064] Since the business operation time represented by the mid sequence "200135521951" of the online production data 2 is later than the business operation time represented by the mid sequence "200135521945" of the online production data 1 stored in the corresponding single data source sub-table 400 in HBase 120, for the above example, the data buffer module 12 can determine that the online production data 2 is new data through the update judgment 55 of the mid sequence, and use it to update the online production data 1 stored in the corresponding single data source sub-table 400 in HBase 120.

[0065] However, if the business operation time represented by the mid sequence of the online production data 2 is not later than, or even earlier than, the business operation time represented by the mid sequence of the online production data 1 stored in the corresponding single data source sub-table 400 in HBase 120, then even if the opt field of the online production data 2 is "UPDATE" representing an update operation, the data buffer module 12 may be able to determine that the online production data 2 is dirty data that should be overwritten through the update judgment 55 of the mid sequence.

[0066] Figure 4 For Figure 1 Schematic diagram of the data merging module in the device shown. Figure 4 And combined with Figure 1 The data merging module 13 can periodically merge the single data source sub-tables 400 in the key-value pair format in each partition into a multi-data source wide table 600 .

[0067] Specifically, the data merging module 13 can periodically obtain the data stored in the single data source sub-table 400 from the distributed column-oriented database (such as HBase) 120 in a preset time window (such as a Flink time window), and then merge the data in a relational database such as an H2 in-memory database to obtain a multi-data source wide table 600, and finally insert the multi-data source wide table 600 into the distributed column-oriented database (such as HBase) 120.

[0068] The length of the time window can be set arbitrarily to adjust the degree of immediacy of "real-time processing". The length of the time window can be measured based on time or data volume. Usually, the length of the time window is adapted to the processing speed of milliseconds.

[0069] Moreover, the merging in relational databases such as H2 memory database can be realized by implementing the inter-table association between each single data source sub-table 400 based on Structured Query Language (SQL). Compared with the merging method using Java to associate tables, the SQL-based association method can save development costs and shorten the development cycle. Moreover, considering that offline data is also processed using SQL, using SQL to process online generated data can better keep the real-time data consistent with the offline data.

[0070] In addition, the association between the single data source sub-tables 400 can be determined by the inter-table association fields mentioned above, that is, the single data source sub-tables 400 corresponding to some primary keys of some data sources can be associated and merged into a multi-data source wide table 600, or all the single data source sub-tables 400 corresponding to all primary keys of all the single data source sub-tables 400 can be associated and merged into a multi-data source wide table 600.

[0071] Please look back Figure 1The data production module 14 can store the multi-data source wide table into the distributed file system 30, that is, write the operation into the distributed file system 30 for verification with offline data. For example, the distributed file system 30 can select the Hadoop Distributed File System (HDFS); and / or, the data production module 14 can convert the multi-data source wide table into a log format and provide it to the distributed publish-subscribe messaging system (such as Kafka) 20 and display it on a large screen, so as to provide it to downstream businesses for continued use through the continued production of the distributed publish-subscribe messaging system (such as Kafka) 20.

[0072] Figure 5 FIG. 1 is an exemplary flow chart of a method for real-time streaming processing of online production data in an embodiment. Figure 5 In one embodiment, a method for real-time streaming processing of online production data may include:

[0073] S71, reading online production data from multiple data sources from a distributed publish-subscribe messaging system based on a streaming processing method;

[0074] S72, transferring the online production data of each data source into a single data source sub-table in a key-value pair format;

[0075] S73, periodically merging the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table.

[0076] Based on the above process, the online production data of multiple data sources read through S71 can be transferred to a single data source sub-table in the key-value pair format in S72, and the single data source sub-table in the key-value pair format of each data source can be periodically merged into a multi-data source wide table through S73 to achieve integration of multiple data sources, thereby achieving integration of multiple data sources with considerable consumption efficiency and under the premise of ensuring data integrity.

[0077] Figure 6 For Figure 5 Schematic diagram of the extended process of the method shown. Figure 5 ,like Figure 6 The process shown further includes the following steps after S73:

[0078] S75, storing the multi-data source wide table into the distributed file system;

[0079] S76, converting the multi-data source wide table into a log format and providing it to the distributed publish-subscribe messaging system.

[0080] Therefore, as a Figure 5As a further extension of the process shown, the wide table of multiple data sources can be provided to the distributed file system to achieve relatively simple data verification, and can also be fed back to the distributed publish-subscribe messaging system for downstream business use.

[0081] Figure 7 For Figure 5 A flowchart of a specific example of the method shown. Figure 7 In this embodiment, the method for real-time processing of online production data in a streaming manner can be specifically manifested as comprising the following steps:

[0082] S710. By calling the distributed processing engine for stream data and batch data, online production data stored in each data source according to primary key partitions is read in parallel from the distributed publish-subscribe message system.

[0083] S720: Transfer the online production data of each data source into a single data source sub-table in a key-value pair format in a distributed column-oriented database.

[0084] Among them, S720 may specifically include:

[0085] S721, setting the primary key identifier, database row key, inter-table association fields, and whether there is a modification content field for the update operation in the corresponding single data source sub-table according to the configuration file pre-set for each data source;

[0086] S722, directly converting the newly added data in the online production data into a key-value pair format, converting the updated data into a key-value pair format after overwriting the updated data, and converting the deleted data into a key-value pair format after adding a deletion mark;

[0087] S723, store the newly added data, updated data, and deleted data in the key-value pair format into the corresponding single data source sub-table. If there are multiple corresponding database row keys in the same single data source sub-table, the multiple corresponding database row keys are concatenated. If there is a modification content field for the update operation in the single data source sub-table, an update decision is made.

[0088] S730 , by associating tables based on SQL, merge the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table in the relational database.

[0089] Figure 8 FIG. 1 is a schematic diagram of the structure of a data processing device in one embodiment. Figure 8 As shown, in one embodiment, a data processing device includes a processor 81 and a non-transitory computer-readable storage medium 82, wherein the non-transitory computer-readable storage medium 82 stores instructions executable by the processor 81, so that the processor 81 executes the following Figure 5 or Figure 6 or Figure 7 The steps of the method shown are, for example:

[0090] Read online production data from multiple data sources from a distributed publish-subscribe messaging system based on streaming processing (for example, by calling a distributed processing engine for stream data and batch data), and partition and store online production data from multiple data sources according to primary keys;

[0091] Transfer the online production data of each data source (for example, in a distributed column-oriented database) to a single data source table in a key-value pair format. The specific transfer method can refer to the method described above.

[0092] Periodically merge the single data source sub-tables in the key-value pair format of each data source (for example, SQL-based associations in a relational database) into a multi-data source wide table.

[0093] Furthermore, when the instructions stored in the non-transitory computer-readable storage medium 82 are executed by the processor 81, the processor 81 may further store the multi-data source wide table into a distributed file system, and / or convert the multi-data source wide table into a log format and provide it to a distributed publish-subscribe messaging system.

[0094] In addition, the data processing device may also include a large-capacity storage medium 84 for storing distributed column-oriented databases such as HBase, and relational databases such as H2 in-memory databases.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A device for real-time processing of online production data in a streaming manner, characterized in that: include: A data access module, which reads online production data from multiple data sources from a distributed publish-subscribe messaging system based on a streaming processing method; A data buffer module, which transfers the online production data of each data source into a single data source sub-table in a key-value format; A data merging module, which periodically merges the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table; The data buffer module sets the primary key identifier, database row key, inter-table association fields, and whether there is a modification content field for the update operation in the corresponding single data source sub-table according to the configuration file pre-set for each data source; The data buffer module directly converts the newly added data in the online production data into a key-value pair format, converts the updated data into a key-value pair format after overwriting the data, and converts the deleted data into a key-value pair format after adding a deletion mark; and the data buffer module stores the newly added data, updated data, and deleted data in the key-value pair format into the corresponding single data source sub-table, wherein, If there are multiple corresponding database row keys in the same single data source sub-table, the data buffer module concatenates the multiple corresponding database row keys; If there is a modification content field for the update operation in the single data source sub-table, the data buffer module performs an update decision.

2. The device according to claim 1, characterized in that The data access module reads online production data by calling a distributed processing engine for stream data and batch data.

3. The device according to claim 1, characterized in that The online production data of each data source is stored in partitions according to primary key pairs in the distributed publish-subscribe message system, and the data access module reads the online production data of each data source from each partition in parallel.

4. The device according to claim 1, characterized in that The data buffer module transfers the single data source sub-table in the key-value pair format in the distributed column-oriented database.

5. The device according to claim 1, characterized in that The data merging module merges multiple data sources in a relational database to obtain a wide table.

6. The device according to claim 1, characterized in that The data merging module realizes the merging of wide tables from multiple data sources through association between tables based on structured query language.

7. The device according to claim 1, characterized in that The data merging module periodically merges multiple data sources in a preset time window to obtain a wide table.

8. The device according to claim 1, characterized in that Further including: A data production module stores the multi-data source wide table into a distributed file system, and / or converts the multi-data source wide table into a log format and then provides it to a distributed publish-subscribe messaging system.

9. A method for real-time processing of online production data in a streaming manner, characterized in that: include: Read online production data from multiple data sources from a distributed publish-subscribe messaging system based on streaming processing; Transfer the online production data of each data source into a single data source table in key-value pair format; Periodically merge the single data source tables in the key-value pair format of each data source into a multi-data source wide table; The method of transferring the online production data of each data source into a single data source sub-table in a key-value pair format includes: According to the configuration files pre-set for each data source, set the primary key identifier, database row key, inter-table association fields, and whether there is a modification content field for the update operation in the corresponding single data source sub-table; Directly convert the newly added data in the online production data into the key-value pair format, convert the updated data into the key-value pair format after overwriting it, and convert the deleted data into the key-value pair format after adding a deletion mark; The newly added data, updated data, and deleted data in the key-value pair format are stored in the corresponding single data source sub-table. If there are multiple corresponding database row keys in the same single data source sub-table, the multiple corresponding database row keys are concatenated; and if there is a modification content field for the update operation in the single data source sub-table, an update decision is made.

10. The method according to claim 9, characterized in that Reading online production data from multiple data sources includes: reading the online production data by calling a distributed processing engine for stream data and batch data.

11. The method according to claim 9, characterized in that The online production data of each data source is partitioned and stored according to the primary key in the distributed publish-subscribe message system, and reading the online production data of multiple data sources includes: reading the online production data of each data source from each partition in parallel.

12. The method according to claim 9, characterized in that Transferring the online production data of each data source into a single data source sub-table in a key-value pair format includes: transferring the single data source sub-table in a key-value pair format in a distributed column-oriented database.

13. The method according to claim 9, characterized in that Merging the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table includes: merging in a relational database to obtain the multi-data source wide table.

14. The method according to claim 9, characterized in that Merging the single data source sub-tables in the key-value pair format of each data source into a multi-data source wide table includes: merging the multi-data source wide tables through table association based on structured query language.

15. The method according to claim 9, characterized in that Further including: The multi-data source wide table is stored in a distributed file system, and / or the multi-data source wide table is converted into a log format and then provided to a distributed publish-subscribe message system.

16. A data processing device, characterized in that: comprising a processor configured to execute the steps of the method as claimed in any one of claims 9 to 15.

17. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores instructions for a processor to execute the steps of the method as claimed in any one of claims 9 to 15.

Images