Real-time data processing method, device, equipment and medium

Abstract

The invention discloses a real-time data processing method and device, equipment and a medium, and relates to the technical field of computers. Comprising the following steps: calling an initialization program to initialize stock log data in a source database; acquiring newly added log data from the source database; the newly added log data comprises driving table data and driven table data; calling a real-time synchronization program to write the driven table data into a cache database; calling a real-time processing program to determine target driven table data matched with the associated key value from the cache database; and writing target data obtained by performing association processing on the target driven table data into a data warehouse so as to complete a real-time data processing process, and performing management scheduling on the related programs through a preset management scheduling program so as to control the processing process. Therefore, the real-time subscription and real-time processing of the data are completed for the complex business scene of the data warehouse, and the data integrity and accuracy are ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a real-time data processing method, device, equipment and medium. Background Art

[0002] Real-time data processing for complex business scenarios in data warehouses involves scenarios where multiple data tables are associated and data is not synchronized between data tables. Whether it is supporting analysis systems or supporting production systems, there are problems such as complex business logic, reliance on multiple data tables and asynchrony between data tables. Real-time data calculation for complex business scenarios is required.

[0003] At present, the above problems are mainly solved by combining multiple statements to query real-time data, using ETL (Extract-Transform-Load, a data extraction, transformation and loading process) tools with centralized architecture, using distributed stream processing tools and using data real-time synchronization tools. Among them, the solution of combining multiple statements to query real-time data can solve the problem of low timeliness, but some source tables have large data and low query efficiency, which cannot be applied to high-concurrency scenarios and scenarios with insufficient computing resources; the solution of using ETL tools with centralized architecture can only achieve high-frequency processing, and consumes a lot of resources in the data warehouse, and the processing timeliness cannot fully meet the needs; the solution of using distributed stream processing tools can well solve the problem of timeliness and single-table calculation, but cannot solve the problem of multi-stream association, and there will be data loss due to the asynchrony of data associated with each stream table; the solution of using real-time data synchronization tools can well solve the problem of sub-second synchronization of single tables, but cannot solve the problem of single-table calculation, let alone the problem of complex calculation.

[0004] From the above, it can be seen that how to complete real-time subscription and real-time processing of data for complex business scenarios in data warehouses and ensure data integrity and accuracy is an urgent problem to be solved. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a real-time data processing method, device, equipment and medium, which can complete real-time subscription and real-time processing of data for complex business scenarios of data warehouses, and ensure data integrity and accuracy. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a real-time data processing method, comprising:

[0007] Call the initialization program to initialize the existing log data in the source database and write the obtained initialization data into the data warehouse;

[0008] Acquire new log data from the source database, and store the new log data in the message synchronization middleware; the new log data includes driving table data and driven table data;

[0009] Calling a real-time synchronization program to write the driven table data in the message synchronization middleware into a cache database;

[0010] Calling a real-time processing program to determine target driven table data matching the associated key value from the cache database using the associated key value of the driving table data, and performing associated processing on the target driven table data to obtain target data;

[0011] Writing the target data into the data warehouse to complete the real-time data processing process;

[0012] The initialization program, the real-time synchronization program and the real-time processing program are all managed and scheduled by a preset management and scheduling program to achieve control over the real-time data processing process.

[0013] Optionally, acquiring the newly added log data from the source database and storing the newly added log data in the message synchronization middleware includes:

[0014] Acquire the newly added log data from the source database, and store the newly added log data in different storage units in the message synchronization middleware according to the data type of the newly added log data; the storage unit includes a driving table storage unit for storing the driving table data and a driven table storage unit for storing the driven table data;

[0015] Correspondingly, the calling of the real-time synchronization program to write the driven table data in the message synchronization middleware into the cache database includes:

[0016] Calling a real-time synchronization program to obtain a driven table storage unit in the message synchronization middleware, and writing the driven table data in the driven table storage unit into a cache database;

[0017] Correspondingly, the calling of the real-time processing program determines the target driven table data matching the associated key value from the cache database using the associated key value of the driver table data, including:

[0018] The real-time processing program is called to obtain the driver table storage unit in the message synchronization middleware, and the associated key value of the driver table data in the driver table storage unit is used to determine the target driven table data matching the associated key value from the cache database.

[0019] Optionally, the calling of the real-time processing program uses the association key value of the driver table data to determine the target driven table data matching the association key value from the cache database, and performs association processing on the target driven table data to obtain the target data, including:

[0020] Calling a real-time processing program to obtain the driver table data from the message synchronization middleware, and using the associated key value of the driver table data to determine whether there is target driven table data matching the associated key value in the cache database;

[0021] If the target driven table data matching the associated key value exists in the cache database, the target driven table data is processed to obtain the target data;

[0022] If the target driven table data matching the associated key value does not exist in the cache database, the target driven table data corresponding to the associated key value is written to the message processing middleware, and the target driven table data matching the associated key value is determined from the cache database based on a preset delayed consumption time node.

[0023] Optionally, the determining, from the cache database, target driven table data matching the associated key value based on a preset delayed consumption time node includes:

[0024] Based on the preset delayed consumption time node, using the associated key value of the target driver table data, determining whether there is target driven table data matching the associated key value in the cache database;

[0025] If the target driven table data matching the associated key value exists in the cache database, the target driven table data is processed to obtain the target data;

[0026] If the target driven table data matching the associated key value does not exist in the cache database, the target driven table data is rewritten into the message processing middleware.

[0027] Optionally, the real-time data processing method further includes:

[0028] The compensation processing program is called to obtain the target driving table data satisfying the preset time condition in the message processing middleware, and the target driving table data is compensated and processed to obtain the target data.

[0029] Optionally, the calling of the compensation processing program to obtain the target driving table data satisfying the preset time condition in the message processing middleware, and performing compensation processing on the target driving table data to obtain the target data, includes:

[0030] Calling a compensation processing program to obtain first target driver table data that meets a first preset time condition in the message processing middleware, and using an associated key value of the first target driver table data to search for target driven table data that matches the associated key value from the source database;

[0031] Performing association processing on the target driven table data to obtain target data;

[0032] Or, calling a compensation processing program to obtain second target driver table data satisfying a second preset time condition in the message processing middleware, and recording the associated key value of the second target driver table data in an exception list in a preset processing configuration library;

[0033] The data in the source database is calculated according to the associated key value recorded in the exception list by calling a manual compensation program to obtain target data.

[0034] Optionally, the real-time data processing method further includes:

[0035] A plurality of data task execution nodes are created through the preset management scheduling program, and the initialization program, the real-time synchronization program, the real-time processing program and the compensation processing program are called based on the data task execution nodes to realize the control of the real-time data processing process.

[0036] In a second aspect, the present application provides a real-time data processing device, comprising:

[0037] The data initialization module is used to call the initialization program to initialize the stock log data in the source database and write the obtained initialization data into the data warehouse;

[0038] A data acquisition module, used to acquire newly added log data from the source database and store the newly added log data in the message synchronization middleware; the newly added log data includes driving table data and driven table data;

[0039] A data synchronization module, used for calling a real-time synchronization program to write the driven table data in the message synchronization middleware into a cache database;

[0040] A data processing module, used for calling a real-time processing program to determine target driven table data matching the associated key value from the cache database using the associated key value of the driving table data, and performing associated processing on the target driven table data to obtain target data;

[0041] A data writing module, used to write the target data into the data warehouse to complete the real-time data processing process;

[0042] The initialization program, the real-time synchronization program and the real-time processing program are all managed and scheduled by a preset management and scheduling program to achieve control over the real-time data processing process.

[0043] In a third aspect, the present application provides an electronic device, including:

[0044] Memory, used to store computer programs;

[0045] The processor is used to execute the computer program to implement the above-mentioned real-time data processing method.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned real-time data processing method when executed by a processor.

[0047] The present application provides a real-time data processing method, which includes first acquiring newly added log data from the source database, and storing the newly added log data in a message synchronization middleware; the newly added log data includes driving table data and driven table data; then calling a real-time synchronization program to write the driven table data in the message synchronization middleware into a cache database; then calling a real-time processing program to determine target driven table data matching the associated key value from the cache database using the associated key value of the driving table data, and performing associated processing on the target driven table data to obtain target data; finally, writing the target data into the data warehouse to complete the real-time data processing process.

[0048] As can be seen from the above, the present application obtains the driven table data generated in the source database through a real-time synchronization program and stores it in a cache database. By calling a real-time processing program, the driving table data is associated and matched with the driven table data, thereby avoiding the problem of data invalidation during data processing due to the asynchronization of data in multiple business data tables, and ensuring the real-time and accuracy of data processing during data processing. As a result, it is possible to complete real-time subscription and real-time processing of data for complex business scenarios in data warehouses, ensuring data integrity and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 A flow chart of a real-time data processing method disclosed in this application;

[0051] Figure 2 A schematic diagram of the structure of a specific real-time data processing method disclosed in this application;

[0052] Figure 3 A schematic diagram of a data structure disclosed in this application;

[0053] Figure 4 A schematic diagram of a real-time data processing device disclosed in this application;

[0054] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Real-time data processing for complex business scenarios in data warehouses involves scenarios where multiple data tables are associated and data is not synchronized between data tables. Whether it is supporting analysis systems or supporting production systems, there are problems with complex business logic, reliance on multiple data tables, and the data tables are not synchronized. Real-time data calculation for complex business scenarios is required, but the main data processing methods currently have their own shortcomings and defects and cannot adapt to complex business scenarios. To this end, this application provides a real-time data processing solution that can complete real-time subscription and real-time processing of data for complex business scenarios in data warehouses to ensure data integrity and accuracy.

[0057] See also Figure 1 As shown, the embodiment of the present application discloses a real-time data processing method, including:

[0058] Step S11, calling the initialization program to initialize the stock log data in the source database, and writing the obtained initialization data into the data warehouse.

[0059] In this embodiment, the stock log data in the source database needs to be stored in the data warehouse. Specifically, the initialization program is called to initialize the stock log data in the source database, and the obtained initialization data is written into the data warehouse. That is, by calling the initialization program, the stock data is initialized and processed, and the initialization data generated after the processing is written into the data warehouse, data loss can be avoided and the integrity of the data during the data processing process can be guaranteed.

[0060] Step S12: Acquire the newly added log data from the source database, and store the newly added log data in the message synchronization middleware.

[0061] In this embodiment, the message synchronization middleware accesses the source database to obtain the newly added log data, and stores the newly added log data in different storage units according to different tables. The newly added log data includes driver table data and driven table data, and each table data has a unique corresponding associated key value, and the corresponding driver table data and driven table data can be associated and matched through the key value. Correspondingly, the storage unit includes a driver table storage unit for storing the driver table data and a driven table storage unit for storing the driven table data.

[0062] It should be noted that the message synchronization middleware is a data management platform, and the specific type of the data management platform can be determined based on the needs of the actual application scenario. For example, in a specific implementation, Kafka (an open source distributed event stream platform) can be selected as the message synchronization middleware; accordingly, the newly added log data is stored in different Topics (the basic unit of data publishing and subscription in Kafka) according to different tables. Specifically, the acquisition of newly added log data from the source database and the storage of the newly added log data in the message synchronization middleware may include: acquiring the newly added log data from the source database and storing the newly added log data in different topics in the synchronized Kafka according to the data type of the newly added log data; correspondingly, the calling of the real-time synchronization program to write the driven table data in the message synchronization middleware into the cache database may include: calling the real-time synchronization program to subscribe to the driven table topic in the synchronized Kafka and writing the driven table data in the driven table topic into the cache database; correspondingly, the calling of the real-time processing program to determine the target driven table data matching the associated key value from the cache database using the associated key value of the driving table data may include: calling the real-time processing program to obtain the driving table topic in the synchronized Kafka and determining the target driven table data matching the associated key value from the cache database using the associated key value of the driving table data in the driving table topic. It can be understood that, in addition to using Kafka as the message synchronization middleware, other data management platforms having the same data structure and capable of data interaction may also be used as the message synchronization middleware, which is not specifically limited here.

[0063] Step S13: calling a real-time synchronization program to write the driven table data in the message synchronization middleware into a cache database.

[0064] In this embodiment, the real-time synchronization program writes the driving table data into the cache database by subscribing to the driven table storage unit in the message synchronization middleware. The cache database can use a hash data structure to store the driven table data, and the load capacity of the cache database can be enhanced by changing the data format of the driven table data.

[0065] Furthermore, in order to avoid a huge load pressure caused by an excessive amount of data in the cache database, the cache database can delete the data stored in itself based on a preset time, which can be determined according to actual application conditions.

[0066] Step S14, calling the real-time processing program to determine the target driven table data matching the associated key value from the cache database using the associated key value of the driving table data, and performing associated processing on the target driven table data to obtain the target data.

[0067] In this embodiment, the real-time processing program obtains the driver table data by subscribing to the driver table storage unit in the message synchronization middleware, and searches for the target driven table data corresponding to the associated key value from the cache database according to the associated key value corresponding to the driver table data. Specifically, the calling of the real-time processing program uses the associated key value of the driver table data to determine the target driven table data matching the associated key value from the cache database, and performs associated processing on the target driven table data to obtain the target data, which may include: calling the real-time processing program to obtain the driver table data from the message synchronization middleware, and using the associated key value of the driver table data to search for the target driven table data matching the associated key value from the cache database; performing associated processing on the target driven table data and the initialization data corresponding to the target driven table to obtain the target data.

[0068] In this embodiment, the real-time processing program is called to use the associated key value of the driver table data to search the target driven table data that matches the associated key value from the cache database. If the target driven table data that matches the associated key value does not exist in the cache database, the target driver table data corresponding to the associated key value is written into the message processing middleware. Wherein, while storing the target driver table data, the message processing middleware also needs to record the specific time when the target driver table data is first written into the message processing middleware, so as to determine the target driven table data that matches the associated key value from the cache database based on the preset delayed consumption time node. For example, in a specific embodiment, the real-time processing program can determine the target driven table data that matches the associated key value from the cache database according to the preset progressive time node. It can be understood that the preset progressive time node can be specifically determined according to the actual application situation, and no specific limitation is made here.

[0069] Specifically, the determining the target driven table data matching the associated key value from the cache database based on the preset delayed consumption time node may include: judging whether there is target driven table data matching the associated key value in the cache database based on the preset progressive time node using the associated key value of the target driven table data; if there is target driven table data matching the associated key value in the cache database, processing the target driven table data to obtain the target data; if there is no target driven table data matching the associated key value in the cache database, rewriting the target driven table data into the message processing middleware. That is, by storing the target driven table data that failed to successfully perform the associated match into the message processing middleware, and rematching the target driven table data based on the preset progressive time node, data processing errors caused by the failure of the target driven table data and the driving table data to be synchronized in real time can be avoided, thereby improving the accuracy of the real-time data processing process.

[0070] In this embodiment, if the retention of the target driver table data in the message processing middleware meets the preset time condition, the target driver table data is obtained from the message processing middleware by calling the compensation processing program. The preset time condition is the time condition for the driver table data to be retained in the message processing middleware. For example, in a specific embodiment, the retention of 1 hour is set as the preset time condition, the compensation processing program is called to obtain the target driver table data with a retention time greater than 1 hour and failed to be successfully associated and matched from the message processing middleware, and the driver table data is compensated. Specifically, the calling of the compensation processing program to obtain the target driver table data in the message processing middleware that meets the preset time condition, and the compensation processing of the target driver table data to obtain the target data may include: calling the compensation processing program to obtain the target driver table data in the message processing middleware that has a retention time greater than 1 hour and failed to be successfully associated and matched, and using the associated key value of the target driver table data to find the target driven table data matching the associated key value from the source database; and performing associated processing on the target driven table data to obtain the target data. That is, by setting the preset time condition in advance, compensatory processing is performed on the driving table data that has been retained in the message processing middleware for too long, and the corresponding driven table data is directly found from the source database for association matching, thereby avoiding data processing errors caused by data loss.

[0071] Further, if the driven table data that matches the associated key value of the driving table data cannot be determined from the source database within a certain period of time, manual intervention is required by relevant personnel. For example, in a specific embodiment, the retention of 24 hours is set as a preset time condition, the compensation processing program is called to obtain the target driving table data with a retention time greater than 24 hours and failed to be successfully associated and matched from the message processing middleware, and the driving table data is recorded in the exception list, so that the relevant personnel can manually compensate for it. Specifically, the calling of the compensation processing program to obtain the target driving table data that meets the preset time condition in the message processing middleware, and the compensation processing of the target driving table data to obtain the target data may include: calling the compensation processing program to obtain the target driving table data with a retention time greater than 24 hours and failed to be successfully associated and matched in the message processing middleware, and recording the associated key value of the target driving table data in the exception list in the preset processing configuration library; by calling the manual compensation program, the data in the source database is calculated according to the associated key value recorded in the exception list to obtain the target data. That is, the driver table data in the exception list is processed by the manual compensation program to avoid data processing errors caused by data anomalies in the source database, thereby improving the data integrity and accuracy of real-time data processing. It should be noted that the specific compensation processing method includes but is not limited to recalculating the driver table data and associating and matching the driver table data with new driven table data.

[0072] Step S15: writing the target data into the data warehouse to complete the real-time data processing process.

[0073] In this embodiment, the data format of the target data is changed and the target data is stored in the data warehouse, wherein the data format can be determined according to the actual application scenario.

[0074] It should be noted that the preset management scheduling program can be used to configure data computing tasks, arrange dependencies and accurately schedule, deploy several data task execution nodes, and send corresponding task processing commands and control commands to the task execution nodes by the preset management scheduling program, so as to access the relevant source database, message synchronization middleware, and cache database to perform corresponding data initialization processing, real-time synchronization, real-time processing and other operations to complete the complete process of data processing. Specifically, the real-time data processing method may also include: creating several data task execution nodes through the preset management scheduling program, and calling the initialization program, the real-time synchronization program, the real-time processing program and the compensation processing program based on the data task execution node to realize the control of the real-time data processing process. That is, by uniformly creating data task nodes through the management mobilization program, the entire real-time data processing process can be managed and monitored, the number of errors in the processing process can be reduced, and the location of the error can be located in time and corrected when an error occurs.

[0075] As can be seen from the above, the embodiment of the present application writes the stock data in the source database into the data warehouse by calling the initialization program, avoiding processing errors that may be caused by data loss, and ensuring the integrity of the data to the greatest extent. The driven table data generated in the source database is obtained by the real-time synchronization program and stored in the cache database. The driver table data is associated and matched with the driven table data by calling the real-time processing program, avoiding the problem of data failure in the data processing process caused by the data in multiple business data tables being out of sync, and ensuring the real-time and accuracy of data processing in the data processing process. Further, by performing multiple delayed matching on the driver table that failed to successfully perform the association matching for the first time, the data processing error caused by the failure of the driver table data and the driver table data to synchronize in real time can be avoided, and the driver table data with a long retention time is compensated for, avoiding data processing errors caused by data anomalies and data loss in the source database, thereby improving the data integrity and accuracy of real-time data processing. At the same time, the management and scheduling program manages and monitors each step of the processing process, which can reduce the number of errors in the processing process, and can locate the location of the error in time and correct it when an error occurs. As a result, real-time subscription and real-time processing of data can be completed for complex business scenarios in the data warehouse to ensure data integrity and accuracy.

[0076] See also Figure 2 As shown, the embodiment of the present application provides a specific real-time data processing method, including:

[0077] In this embodiment, for the real-time data processing scenarios of complex business scenarios in data warehouses, data computing task configuration, dependency orchestration and precise scheduling are performed in the management scheduler, and several data task execution nodes are deployed. The management scheduler sends the corresponding task processing commands and control commands to the task execution nodes, and the task execution nodes access the relevant data sources: message middleware, cache database, source database, to perform corresponding data initialization processing, real-time synchronization, real-time processing, compensation processing and other operations to complete the complete data processing process.

[0078] See also Figure 2 As shown, the embodiment of the present application includes a data processing link and a management link, wherein link 1 is an initialization processing link, which completes the initialization processing by calling an initialization program to obtain stock data from the source database and write the data into the data warehouse.

[0079] Link 2 is a real-time synchronization link, which accesses the source database through the message synchronization middleware to obtain the newly added log data, and stores it in different storage units according to different tables; the real-time synchronization program subscribes to the storage unit of the driven table and writes the data into the cache database.

[0080] Link 3 is a real-time processing link. By calling the real-time processing program, the storage unit of the driver table is subscribed from the message synchronization middleware, and the driven table data corresponding to the association key is searched in the cache database in sequence according to the association key value, the driven table data and related records in the cache database are obtained for association processing, and the processed data is written to the data warehouse; if the association key value does not exist in the cache database, the data of the driver table is marked as delayed consumption and written to the message processing middleware. The real-time processing program retrieves the delayed consumption driver table data from the message processing middleware according to progressive time periods (5 seconds, 10 seconds, 30 seconds, 2 minutes, 5 minutes, 30 minutes, 60 minutes, 100 minutes), and the driven table data corresponding to the association key is searched in the cache database in sequence according to the association key value, the driven table data and related records in the cache database are obtained for association processing, and the processed data is written to the data warehouse. If it does not exist in the cache database, the data of the driver table is continued to be written back to the message processing middleware.

[0081] Link 4 is the compensation processing link. By mobilizing the compensation processing program, the driving table data that has not been successfully associated within 100 minutes in the message processing middleware is obtained, and the driven table data corresponding to the association key is searched in the source database in turn according to the association key value, and the driven table data and related records are obtained for association processing, and the processed data is written to the data warehouse. If the data is still not successfully associated after 24 hours, the association key value is recorded in the exception list of the processing configuration library, and then the manual compensation program is manually called to obtain the relevant data from the source data warehouse in turn according to the association key value in the exception list for calculation or add the specified association key value for recalculation.

[0082] Link 5 is the management and scheduling link, which manages the synchronization, processing logic, and task dependencies of all processing tasks through the management scheduler, and coordinates the orderly operation of various tasks; establishes a development framework to reduce the difficulty of development, configures the mapping relationship of processing tasks, and then encapsulates the processing scripts by the computing engine.

[0083] It should be noted that see Figure 3 As shown, the data structures involved in the embodiments of the present application include message middleware, cache database, and processing configuration library. Among them, the message middleware stores the data of the driving table and the driven table at the same time during the data processing process, one storage unit for each table, using the association key as the partition key, and adding a time field to record the current time of the first write. The cache database stores the driven table data during the data processing process, one association key for each table, and adopts the data structure of the hash table; for the convenience of retrieval, the recorded association key value uses the primary key of the table, and the recorded value uses the non-primary key of the table to assemble into json (JavaScript Object Notation, a lightweight data exchange format) format. To avoid unlimited data expansion, the cache database will delete the driven table data that is more than 24 hours old. The processing configuration library stores the processed exception list during the data processing process, which mainly includes the association key of the driving table with the exception, and records the time when the association key is first written into the exception list.

[0084] As can be seen from the above, the embodiment of the present application uses a management scheduler to perform real-time data processing and small task collaboration for complex business scenarios that require data warehouses, and performs unified task configuration, dependency arrangement, and precise collaborative scheduling to ensure data integrity, real-time, and accuracy. The initialization program, real-time synchronization program, real-time processing program, compensation processing program, and supporting data structures deployed at the task execution node are responsible for starting and stopping according to the control commands of the management scheduler, and executing corresponding data subscription, search, logical processing, and corresponding data writing operations according to the processing commands. This makes it possible to complete real-time subscription and real-time processing of data for complex business scenarios in data warehouses, ensuring data integrity and accuracy.

[0085] See also Figure 4 As shown, the embodiment of the present application provides a real-time data processing device, including:

[0086] The data initialization module 11 is used to call the initialization program to initialize the stock log data in the source database and write the obtained initialization data into the data warehouse;

[0087] A data acquisition module 12 is used to acquire newly added log data from the source database and store the newly added log data in the message synchronization middleware; the newly added log data includes driving table data and driven table data;

[0088] A data synchronization module 13, used for calling a real-time synchronization program to write the driven table data in the message synchronization middleware into a cache database;

[0089] The data processing module 14 is used to call the real-time processing program to determine the target driven table data matching the associated key value from the cache database using the associated key value of the driving table data, and to perform associated processing on the target driven table data to obtain the target data;

[0090] The data writing module 15 is used to write the target data into the data warehouse to complete the real-time data processing process.

[0091] As can be seen from the above, the embodiment of the present application obtains the driven table data generated in the source database through a real-time synchronization program and stores it in a cache database. By calling a real-time processing program, the driving table data is associated and matched with the driven table data, thereby avoiding the problem of data invalidation during the data processing process due to the asynchronization of data in multiple business data tables, and ensuring the real-time and accuracy of data processing during the data processing process. As a result, it is possible to complete real-time subscription and real-time processing of data for complex business scenarios in data warehouses, ensuring data integrity and accuracy.

[0092] In some specific implementations, the data acquisition module 12 may specifically include:

[0093] A data storage unit, used to obtain the newly added log data from the source database, and store the newly added log data in different storage units in the message synchronization middleware according to the data type of the newly added log data; the storage unit includes a driving table storage unit for storing the driving table data and a driven table storage unit for storing the driven table data;

[0094] Accordingly, the data synchronization module 13 may specifically include:

[0095] A driven table data acquisition unit, used for calling a real-time synchronization program to acquire the driven table storage unit in the message synchronization middleware, and writing the driven table data in the driven table storage unit into a cache database;

[0096] Accordingly, the data processing module 14 may specifically include:

[0097] The driving table data acquisition unit is used to call the real-time processing program to obtain the driving table storage unit in the message synchronization middleware, and use the associated key value of the driving table data in the driving table storage unit to determine the target driven table data matching the associated key value from the cache database.

[0098] In some specific implementations, the data processing module 14 may specifically include:

[0099] A first associated key value matching unit is used to call a real-time processing program to obtain the driver table data from the message synchronization middleware, and use the associated key value of the driver table data to determine whether there is target driven table data matching the associated key value in the cache database;

[0100] a first data processing unit, configured to process the target driven table data to obtain target data if target driven table data matching the associated key value exists in the cache database;

[0101] The data delay processing submodule is used to write the target driven table data corresponding to the associated key value into the message processing middleware if there is no target driven table data matching the associated key value in the cache database, and determine the target driven table data matching the associated key value from the cache database based on a preset delayed consumption time node.

[0102] Furthermore, in some specific implementations, the data cache submodule may specifically include:

[0103] A second associated key value matching unit is used to determine whether there is target driven table data matching the associated key value in the cache database based on a preset delayed consumption time node by using the associated key value of the target driver table data;

[0104] a second data processing unit, configured to process the target driven table data to obtain target data if target driven table data matching the associated key value exists in the cache database;

[0105] The data cache unit is used to rewrite the target driven table data into the message processing middleware if the target driven table data matching the associated key value does not exist in the cache database.

[0106] In some specific implementations, the real-time data processing device may further include:

[0107] A data compensation processing module is used to call a compensation processing program to obtain target drive table data that meets a preset time condition in the message processing middleware, and to perform compensation processing on the target drive table data to obtain target data;

[0108] The program management module is used to create a number of data task execution nodes through the preset management scheduling program, and call the initialization program, the real-time synchronization program, the real-time processing program and the compensation processing program based on the data task execution nodes to realize the control of the real-time data processing process.

[0109] Furthermore, in some specific implementations, the data compensation processing module may specifically include:

[0110] A first data compensation unit is used to call a compensation processing program to obtain first target driver table data that meets a first preset time condition in the message processing middleware, and use an associated key value of the first target driver table data to search for target driven table data that matches the associated key value from the source database;

[0111] A second data compensation unit is used to call a compensation processing program to obtain second target driver table data that meets a second preset time condition in the message processing middleware, and record the associated key value of the second target driver table data in an exception list in a preset processing configuration library;

[0112] The third data processing unit is used to calculate the data in the source database according to the associated key value recorded in the exception list by calling a manual compensation program to obtain target data.

[0113] Furthermore, the present application also discloses an electronic device. Figure 5 : This is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the real-time data processing method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0114] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0115] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0116] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the real-time data processing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0117] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the real-time data processing method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0119] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0120] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0121] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0122] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A real-time data processing method, characterized in that: include: Call the initialization program to initialize the existing log data in the source database and write the obtained initialization data into the data warehouse; Acquire new log data from the source database, and store the new log data in the message synchronization middleware; the new log data includes driving table data and driven table data; Calling a real-time synchronization program to write the driven table data in the message synchronization middleware into a cache database; Calling a real-time processing program to determine target driven table data matching the associated key value from the cache database using the associated key value of the driving table data, and performing associated processing on the target driven table data to obtain target data; Writing the target data into the data warehouse to complete the real-time data processing process; The initialization program, the real-time synchronization program and the real-time processing program are all managed and scheduled by a preset management and scheduling program to achieve control over the real-time data processing process.

2. The real-time data processing method according to claim 1, characterized in that: The acquiring of the newly added log data from the source database and storing the newly added log data in the message synchronization middleware includes: Acquire the newly added log data from the source database, and store the newly added log data in different storage units in the message synchronization middleware according to the data type of the newly added log data; the storage unit includes a driving table storage unit for storing the driving table data and a driven table storage unit for storing the driven table data; Correspondingly, the calling of the real-time synchronization program to write the driven table data in the message synchronization middleware into the cache database includes: Calling a real-time synchronization program to obtain a driven table storage unit in the message synchronization middleware, and writing the driven table data in the driven table storage unit into a cache database; Correspondingly, the calling of the real-time processing program determines the target driven table data matching the associated key value from the cache database using the associated key value of the driver table data, including: The real-time processing program is called to obtain the driver table storage unit in the message synchronization middleware, and the associated key value of the driver table data in the driver table storage unit is used to determine the target driven table data matching the associated key value from the cache database.

3. The real-time data processing method according to claim 1 or 2, characterized in that: The calling real-time processing program determines the target driven table data matching the associated key value from the cache database by using the associated key value of the driving table data, and performs associated processing on the target driven table data to obtain the target data, including: Calling a real-time processing program to obtain the driver table data from the message synchronization middleware, and using the associated key value of the driver table data to determine whether there is target driven table data matching the associated key value in the cache database; If the target driven table data matching the associated key value exists in the cache database, the target driven table data is processed to obtain the target data; If the target driven table data matching the associated key value does not exist in the cache database, the target driven table data corresponding to the associated key value is written to the message processing middleware, and the target driven table data matching the associated key value is determined from the cache database based on a preset delayed consumption time node.

4. The real-time data processing method according to claim 3, characterized in that: The determining, based on the preset delayed consumption time node, from the cache database the target driven table data matching the associated key value comprises: Based on the preset delayed consumption time node, using the associated key value of the target driver table data, determining whether there is target driven table data matching the associated key value in the cache database; If the target driven table data matching the associated key value exists in the cache database, the target driven table data is processed to obtain the target data; If the target driven table data matching the associated key value does not exist in the cache database, the target driven table data is rewritten into the message processing middleware.

5. The real-time data processing method according to claim 3, characterized in that: Also includes: The compensation processing program is called to obtain the target driving table data satisfying the preset time condition in the message processing middleware, and the target driving table data is compensated and processed to obtain the target data.

6. The real-time data processing method according to claim 5, characterized in that: The calling of the compensation processing program to obtain the target driving table data satisfying the preset time condition in the message processing middleware, and performing compensation processing on the target driving table data to obtain the target data, includes: Calling a compensation processing program to obtain first target driver table data that meets a first preset time condition in the message processing middleware, and using an associated key value of the first target driver table data to search for target driven table data that matches the associated key value from the source database; Performing association processing on the target driven table data to obtain target data; Or, calling a compensation processing program to obtain second target driver table data satisfying a second preset time condition in the message processing middleware, and recording the associated key value of the second target driver table data in an exception list in a preset processing configuration library; The data in the source database is calculated according to the associated key value recorded in the exception list by calling a manual compensation program to obtain target data.

7. The real-time data processing method according to claim 6, characterized in that: Also includes: A plurality of data task execution nodes are created through the preset management scheduling program, and the initialization program, the real-time synchronization program, the real-time processing program and the compensation processing program are called based on the data task execution nodes to realize the control of the real-time data processing process.

8. A real-time data processing device, characterized in that: include: The data initialization module is used to call the initialization program to initialize the stock log data in the source database and write the obtained initialization data into the data warehouse; A data acquisition module, used to acquire newly added log data from the source database and store the newly added log data in the message synchronization middleware; the newly added log data includes driving table data and driven table data; A data synchronization module, used for calling a real-time synchronization program to write the driven table data in the message synchronization middleware into a cache database; A data processing module, used for calling a real-time processing program to determine target driven table data matching the associated key value from the cache database using the associated key value of the driving table data, and performing associated processing on the target driven table data to obtain target data; A data writing module, used to write the target data into the data warehouse to complete the real-time data processing process; The initialization program, the real-time synchronization program and the real-time processing program are all managed and scheduled by a preset management and scheduling program to achieve control over the real-time data processing process.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the real-time data processing method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the real-time data processing method according to any one of claims 1 to 7 is implemented.

Images