Data processing method and device, electronic equipment and computer program product

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a computer program product, and relates to the technical field of big data. The method comprises the steps that to-be-processed stream data is acquired, a service database table corresponding to the to-be-processed stream data is determined, and the service database table is used for generating an association dimension table; subscribing the incremental data broadcast message according to the data mapping information between the service database table and the message queue theme; in response to the received incremental data broadcast message, determining service database table incremental data corresponding to the service database table, and synchronizing the service database table incremental data to a local cache of the dimension table association logic; and executing the dimension table association task based on the local cache of the dimension table association logic to obtain a real-time association dimension table. According to the method and the device, dimension table association is carried out through data broadcasting and local cache, millisecond-level updating of data can be realized on the premise of ensuring task performance, the purpose of real-time data updating of services is achieved, and stability and consistency of dimension table data are ensured.

Description

Background Art

[0002] This section aims to provide background or context for the embodiments of the present disclosure described in the claims. The description herein is not admitted to be prior art merely by virtue of its inclusion in this section.

[0003] A streaming task, also known as a real-time task or a stream processing task, is a computational task for processing unbounded data sets. An unbounded data set refers to a data set that has a start but no end. They do not terminate and provide data when generated. Unbounded streams must be processed continuously, that is, relevant events must be processed immediately after acquisition. For unbounded data streams, it is usually not possible to wait for all data to arrive because the input is unbounded and will not be completed at any point in time. Processing unbounded data usually requires obtaining events in a specific order (such as the order in which events occur) in order to infer the integrity of the results.

[0004] In the field of stream data processing, tasks often need to perform dimension table association to expand the dimension information of relevant data and then perform subsequent processing logic after broadening the relevant data. Dimension table data is generally stored in a business database, and big data stream data processing tasks need to obtain relevant information by reading the business database. Summary of the Invention

[0005] In related stream processing tasks, the update of the associated dimension table can be achieved by adding a layer of local cache to the dimension table association logic. However, the above solution has the problem of data inconsistency caused by the timing of cache update and elimination. During the update interval period between the local cache and the data in the database, the data is inconsistent and cannot be applied to business scenarios that are sensitive to data accuracy.

[0006] Therefore, the present disclosure proposes an improved data processing method to achieve data update at the millisecond level while ensuring task performance, achieving the purpose of real-time data update for the business, and ensuring the stability and consistency of dimension table data.

[0007] In this context, embodiments of the present disclosure are expected to provide a data processing method, a data processing device, a computer-readable storage medium, an electronic device, and a computer program product.

[0008] In a first aspect of the embodiments of the present disclosure, a data processing method is provided, including: obtaining to-be-processed streaming data, determining a service database table corresponding to the to-be-processed streaming data, where the service database table is used to generate an associated dimension table; subscribing to an incremental data broadcast message according to data mapping information between the service database table and a message queue topic; in response to the received incremental data broadcast message, determining incremental data of the service database table corresponding to the service database table, and synchronizing the incremental data of the service database table to a local cache of a dimension table association logic; and performing a dimension table association task based on the local cache of the dimension table association logic to obtain a real-time associated dimension table.

[0009] In an embodiment of the present disclosure, the service database table includes a data stream table and a to-be-associated dimension table, and determining the service database table corresponding to the to-be-processed streaming data includes: determining a service processing requirement of the to-be-processed streaming data, and determining a message queue topic corresponding to the to-be-processed streaming data in a stream processing task according to the service processing requirement; and determining the data stream table and the to-be-associated dimension table from a service database according to the message queue topic.

[0010] In an embodiment of the present disclosure, subscribing to an incremental data broadcast message according to data mapping information between the service database table and a message queue topic includes: obtaining a pre-constructed meta database, where the meta database is used to store the pre-written data mapping information; obtaining the data mapping information based on the meta database; and subscribing to the incremental data broadcast message according to the data mapping information.

[0011] In an embodiment of the present disclosure, before subscribing to the incremental data broadcast message according to the data mapping information, the method further includes: receiving a broadcast start operation for the service database table, generating a broadcast start configuration statement based on the broadcast start operation; and starting an incremental data broadcast function of the service database table according to the broadcast start configuration statement, where the incremental data broadcast function is used to support a message subscription operation of the incremental data broadcast message.

[0012] In an embodiment of the present disclosure, synchronizing the incremental data of the service database table to a local cache of a dimension table association logic includes: determining the incremental data of the service database table, and writing the incremental data of the service database table to an external data storage component; writing the incremental data of the service database table to a message queue topic corresponding to an incremental message queue based on the incremental data broadcast message; and synchronizing the incremental data of the service database table to the local cache of the dimension table association logic through the incremental message queue.

[0013] In one embodiment of the present disclosure, performing a dimension table association task based on the local cache of the dimension table association task to obtain a real-time associated dimension table, including: obtaining a message queue topic corresponding to the incremental data of the business library table from a pre-constructed meta database; starting an incremental data writing thread, and synchronizing the incremental data of the business library table in the message queue topic to the local cache through the incremental data writing thread; updating the business database table in the local cache to obtain a real-time business database table; performing a dimension table association task based on the real-time business database table to obtain the real-time associated dimension table.

[0014] In one embodiment of the present disclosure, the real-time business database table includes a real-time data flow table and a real-time dimension table to be associated. Performing a dimension table association task based on the real-time business database table to obtain the real-time associated dimension table, including: determining the data flow table fields corresponding to the real-time data flow table and the dimension table fields to be associated corresponding to the real-time dimension table to be associated; determining a data table connection statement based on a business processing request; and performing an association process on the data flow table fields and the dimension table fields to be associated according to the data table connection statement to obtain the real-time associated dimension table.

[0015] In one embodiment of the present disclosure, the above method further includes: obtaining monitoring metric data corresponding to the real-time associated dimension table, where the monitoring metric data includes one or more of the incremental data consumption quantity, data consumption latency, updated data volume, and data health; obtaining pre-configured monitoring and warning conditions, where the monitoring and warning conditions include monitoring metric reference thresholds corresponding to each of the monitoring metric data; determining a task processing status according to the monitoring metric data and the monitoring and warning conditions; and when the task processing status is an insecure state, generating an alarm prompt message based on the monitoring metric data and displaying the alarm prompt message.

[0016] In a second aspect of the embodiments of the present disclosure, a data processing apparatus is provided, including: a business table determination module, configured to obtain to-be-processed flow data and determine a business database table corresponding to the to-be-processed flow data, where the business database table is used to generate an associated dimension table; a message subscription module, configured to subscribe to an incremental data broadcast message according to data mapping information between the business database table and a message queue topic; a data synchronization module, configured to, in response to the received incremental data broadcast message, determine incremental data of the business library table corresponding to the business database table and synchronize the incremental data of the business library table to a local cache of a dimension table association logic; and a dimension table association module, configured to perform a dimension table association task based on the local cache of the dimension table association logic to obtain a real-time associated dimension table.

[0017] In one embodiment of the present disclosure, the service database table includes a data flow table and a dimension table to be associated, and the service table determination module includes a service table determination unit, which is configured to: determine the service processing requirements of the to-be-processed flow data, and determine the message queue topic corresponding to the to-be-processed flow data in the flow processing task according to the service processing requirements; determine the data flow table and the dimension table to be associated from the service database according to the message queue topic.

[0018] In one embodiment of the present disclosure, the message subscription module includes a message subscription unit, which is configured to: obtain a pre-constructed meta database for storing the pre-written data mapping information; obtain the data mapping information based on the meta database; subscribe to the incremental data broadcast message according to the data mapping information.

[0019] In one embodiment of the present disclosure, the data processing device further includes a broadcast function enabling module, which is configured to: receive a broadcast enabling operation for the service database table, generate a broadcast enabling configuration statement based on the broadcast enabling operation; enable the incremental data broadcast function of the service database table according to the broadcast enabling configuration statement, and the incremental data broadcast function is used to support the message subscription operation of the incremental data broadcast message.

[0020] In one embodiment of the present disclosure, the data synchronization module includes a data synchronization unit, which is configured to: determine the incremental data of the service database table, write the incremental data of the service database table into an external data storage component; write the incremental data of the service database table into the message queue topic corresponding to the incremental message queue based on the incremental data broadcast message; synchronize the incremental data of the service database table to the local cache of the dimension table association logic through the incremental message queue.

[0021] In one embodiment of the present disclosure, the dimension table association module includes a dimension table association unit, which is configured to: obtain the message queue topic corresponding to the incremental data of the service database table from a pre-constructed meta database; start an incremental data writing thread, and synchronize the incremental data of the service database table in the message queue topic to the local cache through the incremental data writing thread; update the service database table in the local cache to obtain a real-time service database table; perform a dimension table association task based on the real-time service database table to obtain the real-time associated dimension table.

[0022] In one embodiment of the present disclosure, the real-time service database table includes a real-time data flow table and a real-time dimension table to be associated. The dimension table association unit includes a dimension table association subunit for: determining the data flow table fields corresponding to the real-time data flow table and the dimension table fields to be associated corresponding to the real-time dimension table to be associated; determining a data table connection statement based on a service processing request; and performing an association process on the data flow table fields and the dimension table fields to be associated according to the data table connection statement to obtain the real-time associated dimension table.

[0023] In one embodiment of the present disclosure, the data processing device further includes a monitoring and alarming module for: obtaining monitoring metric data corresponding to the real-time associated dimension table, where the monitoring metric data includes one or more of an incremental data consumption quantity, a data consumption delay time, an updated data quantity, and a data health degree; obtaining pre-configured monitoring and alarming conditions, where the monitoring and alarming conditions include monitoring metric reference thresholds corresponding to the respective monitoring metric data; determining a task processing status according to the monitoring metric data and the monitoring and alarming conditions; and when the task processing status is an insecure status, generating an alarm prompt message based on the monitoring metric data and displaying the alarm prompt message.

[0024] In the third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method as described above is implemented.

[0025] In the fourth aspect of the embodiments of the present disclosure, an electronic device is provided, including: a processor; and a memory on which computer-readable instructions are stored. When the computer-readable instructions are executed by the processor, the data processing method as described above is implemented.

[0026] According to the fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program which, when executed by a processor, implements the data processing method as described above.

[0027] According to the technical solution of the embodiments of the present disclosure, on the one hand, by subscribing to broadcast messages, relevant data changes are monitored in real time, and the changed incremental data is synchronized to the local cache. On the premise of ensuring task performance, data can be updated at the millisecond level, achieving the purpose of real-time business data update. On the other hand, by performing dimension table association through data broadcasting and local caching, the stability and consistency of the dimension table data can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the drawings, several embodiments of the present disclosure are shown by way of illustration and not limitation, wherein:

[0029] Figure 1 shows a schematic flowchart of the dimension table association processing in the related solution;

[0030] Figure 2 schematically shows a schematic block diagram of the system architecture of an exemplary application scenario according to some embodiments of the present disclosure;

[0031] Figure 3 schematically shows a schematic flowchart of a data processing method according to some embodiments of the present disclosure;

[0032] Figure 4 schematically shows a logical implementation diagram of a real-time dimension table association solution based on broadcasting according to some embodiments of the present disclosure;

[0033] Figure 5 schematically shows a processing logic diagram of the incremental subscription function in the dimension table association solution according to some embodiments of the present disclosure;

[0034] Figure 6 schematically shows a processing logic diagram of real-time dimension table association in the dimension table association solution according to some embodiments of the present disclosure;

[0035] Figure 7 schematically shows a schematic block diagram of a data processing device according to some embodiments of the present disclosure;

[0036] Figure 8 schematically shows a schematic diagram of a storage medium according to an exemplary embodiment of the present disclosure;

[0037] Figure 9 schematically shows a block diagram of an electronic device according to an exemplary embodiment of the invention.

[0038] In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed embodiments

[0039] Next, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and then implement the present disclosure, and do not limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to be able to fully convey the scope of the present disclosure to those skilled in the art.

[0040] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, device, equipment, method, or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0041] According to the embodiments of the present disclosure, a data processing method, a data processing device, a computer-readable storage medium, an electronic device, and a computer program product are provided.

[0042] In this context, it should be understood that the terms involved, such as unbounded data sets, refer to data sets that have a start but no end. They do not terminate and provide data when generated. Unbounded streams must be processed continuously, that is, events must be processed immediately after acquisition. For unbounded data streams, we cannot wait for all data to arrive because the input is unbounded and will not be completed at any point in time. Processing unbounded data usually requires obtaining events in a specific order (such as the order in which events occur) in order to infer result integrity.

[0043] A streaming task, also known as a real-time task or a stream processing task, is a computational task for processing unbounded data sets. In a master-slave database, master-slave replication is an essential technology. The master database is responsible for receiving all write operations, while the slave database is responsible for replicating these changes in real time. When the master database executes an INSERT, UPDATE, or DELETE statement, it generates a transaction log (binlog), which is like a detailed operation guide telling the slave database how to update the data. Through reasonable configuration and maintenance, data consistency can be ensured, and at the same time, the availability and load balancing of the system can be improved by using the slave database.

[0044] Dimensional table association and real-time query of the dimensional table mean that the user directly accesses an external database in a Flink operator, for example, using a database such as MySQL for association. This method is a synchronous method, and the data is guaranteed to be up-to-date. Flink is an open-source stream processing framework, and its core is a distributed stream data flow engine written in programming languages Java and Scala. Flink executes any stream data program in a data parallel and pipelined manner, and Flink's pipeline runtime system can execute batch processing and stream processing programs.

[0045] The Remote Dictionary Server (Redis) database is a key-value store database that supports networking, is memory-based, distributed, and optionally persistent. HBase is an open-source non-relational distributed database (Not Only SQL, NoSQL) that runs on top of the Hadoop Distributed FileSystem (HDFS). It can provide extremely high fault tolerance for sparse files. TiDB is a distributed relational database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is compatible with MySQL and can provide horizontal scalability, strong consistency, and high availability.

[0046] In addition, the number of any elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0047] Next, with reference to several representative embodiments of the present disclosure, the principles and spirit of the present disclosure will be elaborated in detail. Summary of the Invention

[0049] In the field of stream data processing, tasks often need to perform dimension table joins to expand the dimensional information of relevant data, widen the relevant data, and then perform subsequent processing logic. Dimension table data is generally stored in the business database, and big data stream data processing tasks need to obtain relevant information by reading the business database.

[0050] For example, in the advertising user behavior log, it contains end-user behavior data such as user identification (ID), resource position, schedule ID, click count, exposure count, etc. When performing relevant behavior statistics in the real-time processing link, more dimensional information of the user and the schedule is required. At this time, the dimension table join technology needs to be used to join the user information table and the schedule information table in the database through the user ID and the schedule ID, and then perform subsequent data processing.

[0051] In a related solution, the dimension table join processing is performed by directly associating and querying the business database. For example, in a stream processing task, directly query the business database to obtain other dimensional information through primary keys such as user ID and schedule ID. Generally, query the slave database. Querying the master database may affect the online business. Refer to Figure 1 , Figure 1 shows a schematic flow diagram of the dimension table join processing in the related solution. In this solution, each record is processed in the stream processing task, and the business slave database is directly queried to obtain the corresponding dimension table information.

[0052] However, the above solutions have the following problems: 1) Querying the slave database will affect the performance of the database slave, causing abnormal or delayed synchronization of the slave; 2) The resource cost of the database is relatively high; 3) The concurrent query ability of the database is relatively limited and cannot meet the query requirements in large-traffic scenarios.

[0053] To solve the problems of Technical Solution 1, generally, the data in the business database can be synchronized to a third-party storage, such as HBase / Redis, etc., and then the data in this storage can be queried in big data tasks, shifting the pressure from the business database to the third-party storage, specifically as Figure 1 shown in Technical Solution 2. The data in the business database is synchronized to the third-party storage in full and incremental ways, and the dimension table is associated to query the third-party storage. A layer of storage, such as Redis / HBase / TiDB, etc., is set up between the big data task and the business database to achieve the purpose of decoupling and performance improvement. The data in the business database is synchronized to a storage with stronger concurrent query ability, such as Redis / HBase / TiDB, etc., through a third-party synchronization tool. Then, in the stream task, the third-party storage is queried to obtain the corresponding data.

[0054] Adopting Technical Solution 2, when the query volume per second increases, the following two problems will arise: 1) The pressure on the external storage engine increases and the query latency increases; 2) The query latency increases and cannot meet the query performance requirements. Based on the problems of Technical Solution 2, usually, a local cache can be added in the dimension table association logic to reduce the query pressure and improve the query performance. The query results are cached in the local memory and strategies such as LRU are used for elimination and update. The core logic is as Figure 1 shown in Technical Solution 3.

[0055] A relatively big problem with Technical Solution 3 is the issue of the timing of cache update and elimination and the resulting data inconsistency problem. During the update interval period, the data in the local cache and the database is inconsistent, and this solution cannot be used in business scenarios that are sensitive to data accuracy.

[0056] Based on the above, the basic idea of the present disclosure is to obtain the to-be-processed stream data, determine the business database table corresponding to the to-be-processed stream data, where the business database table is used to generate an associated dimension table; subscribe to the incremental data broadcast message according to the data mapping information between the business database table and the message queue topic; in response to the received incremental data broadcast message, determine the incremental data of the business database table corresponding to the business database table, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic; perform the dimension table association task based on the local cache of the dimension table association logic to obtain the real-time associated dimension table. The present disclosure performs dimension table association through data broadcast and local cache, and on the premise of ensuring the task performance, can achieve data update at the millisecond level, achieve the purpose of real-time business data update, and ensure the stability and consistency of the dimension table data.

[0057] After introducing the basic principle of the present disclosure, the following specifically introduces various non-limiting implementation manners of the present disclosure.

[0058] Overview of Application Scenarios

[0059] First, refer to Figure 2 , Figure 2 which shows a schematic block diagram of the system architecture of an exemplary application scenario of a data processing method and apparatus to which the embodiments of the present disclosure can be applied.

[0060] As Figure 2 shown, the system architecture 200 may include one or more of the terminal devices 210, a network, and a server 220. The network is used to provide a medium for a communication link between the terminal device 210 and the server 220. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. The terminal device 210 may be various electronic devices with a display screen, including but not limited to desktop computers, portable computers, smartphones, and tablet computers, etc. It should be understood that Figure 2 the number of terminal devices, networks, and servers in

[0061] The data processing method provided by the embodiments of the present disclosure is generally executed by the server 220. Correspondingly, the data processing device is generally disposed in the server 220. However, those skilled in the art can easily understand that the data processing method provided by the embodiments of the present disclosure can also be executed by the terminal device 210. Correspondingly, the data processing device can also be disposed in the terminal device 210. No special limitation is made in this exemplary embodiment. For example, in an exemplary embodiment, a user may perform a service operation through the terminal device 210, and upload the data generated by the service operation to the server 220. Since a large number of users continuously perform service operations through the terminal device, the generated service data will be transmitted as streaming data to the stream processing task. The server performs real-time maintenance table association processing on the relevant service database tables according to the real-time service requirements by using the data processing method provided by the embodiments of the present disclosure, obtains a real-time associated maintenance table, and returns the real-time associated maintenance table to the terminal device 210, etc., so that the terminal device 210 can display the relevant data in the real-time associated maintenance table to the user.

[0062] It should be understood that Figure 2 the application scenario shown is only an example in which the embodiments of the present disclosure can be implemented. The scope of application of the embodiments of the present disclosure is not limited by any aspect of this application scenario.

[0063] Exemplary Method

[0064] Next, in conjunction with Figure 2 the application scenario of Figure 3 the data processing method according to the exemplary embodiments of the present disclosure will be described with reference to

[0065] It should be noted that the above application scenario is only shown for the convenience of understanding the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0065] The present disclosure first provides a data processing method. The execution subject of this method can be a terminal device or a server. The present disclosure does not make special limitations in this regard. In this exemplary embodiment, the case where the server executes this method is taken as an example for illustration.

[0066] Referring to Figure 3 shown, the data processing method may include the following steps S310 to S340:

[0067] Step S310, obtain the to-be-processed streaming data, and determine the service database table corresponding to the to-be-processed streaming data, where the service database table is used to generate an associated maintenance table;

[0068] Step S320, subscribe to the incremental data broadcast message according to the data mapping information between the service database table and the message queue topic;

[0069] Step S330: In response to the received incremental data broadcast message, determine the incremental data of the business database table corresponding to the business database table, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic;

[0070] Step S340: Perform a dimension table association task based on the local cache of the dimension table association logic to obtain a real-time associated dimension table.

[0071] In the data processing method provided by this exemplary embodiment, on the one hand, by subscribing to the broadcast message, relevant data changes are monitored in real time, and the changed incremental data is synchronized to the local cache. On the premise of ensuring the task performance, data can be updated at the millisecond level, achieving the purpose of real-time business data update. On the other hand, by performing dimension table association through data broadcast and local cache, the stability and consistency of the dimension table data can be guaranteed.

[0072] Next, the above steps of this exemplary embodiment will be described in more detail.

[0073] In step S310, obtain the to-be-processed stream data, and determine the business database table corresponding to the to-be-processed stream data. The business database table is used to generate an associated dimension table.

[0074] In an embodiment of the present disclosure, the to-be-processed stream data may be the data to be processed in a stream processing task. The business database table may be a data table in the business database related to the to-be-processed stream data. The associated dimension table may be a data table obtained by performing dimension table association processing on the table information of relevant dimensions in the data tables in different business databases.

[0075] Refer to Figure 4 , Figure 4 which schematically shows a logical implementation diagram of a broadcast-based real-time dimension table association scheme according to some embodiments of the present disclosure. Figure 4 The log receiving service in

[0076] continuously receives the to-be-processed data stream and sends the to-be-processed data stream to the stream processing task through the message queue. After receiving the to-be-processed stream data, the stream processing task can determine the business database table corresponding to the to-be-processed stream data according to the real-time business requirements.

[0077] When sending the stream data, the data producer may add information such as the business module and table name to which the data belongs in the message header or specific fields. By parsing these metadata, the business database table corresponding to the to-be-processed stream data can be directly obtained in combination with the real-time business requirements.

[0078] In one embodiment of the present disclosure, a message queue topic can be a logical message classification mechanism in stream task processing. A message queue topic can be an identifier used to distinguish different types of messages in a message queue. Data mapping information can be the mapping relationship between a business database table and a message queue topic. An incremental data broadcast message can be a broadcast message sent due to incremental data generated in a business data table.

[0079] The stream processing task is performed based on multiple business databases, and a business database can contain multiple data tables. Different business database tables can send incremental data broadcasts to corresponding message queue topics, and the stream processing task can subscribe to incremental data broadcast messages through an incremental message queue. A message queue topic is like a classification label that groups messages with the same nature or source into one category. For example, in an e-commerce system, there may be different topics such as "order creation", "product inventory change", "user login", etc., which are used to process messages generated in different business scenarios.

[0080] Therefore, in order for the stream processing task to subscribe to incremental data broadcast messages of different business database tables, in the data synchronization processing logic, the data mapping relationship between the business database table and the message queue topic is pre-synchronized to the meta-database. The stream processing task can start a thread to subscribe to the incremental data broadcast messages corresponding to the relevant business database tables through the data mapping information in the meta-database.

[0081] In step S330, in response to the received incremental data broadcast message, determine the incremental data of the business database table corresponding to the business database table, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic.

[0082] In one embodiment of the present disclosure, the incremental data of the business database table can be the incremental data generated by the data table in the business database. The dimension table association logic can be the processing logic for performing dimension table association operations. The local cache of the dimension table association logic can be a part of the physical memory allocated locally in the dimension table association logic for caching real-time data.

[0083] After receiving the incremental data broadcast message, the stream processing task can determine the incremental data of the business database table corresponding to the business data table, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic under the stream processing task, so as to update the business database table in the local cache based on the incremental data of the business database table, and make the data of the business data table in the local cache of the dimension table association logic synchronized with the data in the business database.

[0084] If the local cache needs to be enabled for the dimension table association logic, a configuration Structured Query Language (SQL) statement needs to be added. For example, the SQL statement can be configured as: SET ad_schedule_detail.connector.lookup.cache.ttl = 5000.

[0085] In step S340, perform the dimension table association task based on the local cache of the dimension table association logic to obtain the real-time associated dimension table.

[0086] In an embodiment of the present disclosure, the dimension table association task may be an operation task for associating data tables in different business databases. The real-time associated dimension table may be a real-time dimension table obtained by executing the dimension table association task.

[0087] Due to the data synchronization between the local cache of the dimension table association logic and the data tables in the business database, when performing the dimension table association task based on the local cache of the dimension table association logic, the real-time data synchronized with the business database will be used to perform the dimension table association operation. Therefore, a real-time associated dimension table can be obtained.

[0088] Based on the business scenario requirements, the present disclosure designs a real-time dimension table association solution based on broadcasting. To achieve the purpose of real-time data synchronization to the local cache, while synchronizing the data to the external storage component, the real-time changed data is pushed to the incremental message queue in a broadcast manner. In the dimension table association module of the stream processing task, subscribe to the information of the incremental message queue, monitor the relevant data changes in real time, and update the relevant changed data to the local cache. Through the above method, the data can be updated at the millisecond level, achieving the purpose of real-time business data update.

[0089] In an embodiment of the present disclosure, for step S310, determining the business database table corresponding to the to-be-processed stream data includes: determining the business processing requirements of the to-be-processed stream data, and determining the message queue topic corresponding to the to-be-processed stream data in the stream processing task according to the business processing requirements; according to the message queue topic, determining the data stream table and the to-be-associated dimension table from the business database.

[0090] Among them, the business processing requirements may be business requirements determined according to the specific application scenario of the stream processing task. For example, according to the business processing requirements, the business data tables that need to be associated in the stream processing task can be clarified.

[0091] After receiving the stream data to be processed, the stream processing task can determine the business processing requirements of the stream data to be processed. For example, taking the business processing requirements of advertising user behavior statistics as an example, the advertising user behavior log contains end-user behavior data such as user ID, resource position, schedule ID, click count, and exposure count. When performing relevant behavior statistics in the real-time processing link, more dimensional information about users and schedules is required. At this time, the dimension table association technology needs to be used to associate the user information table and the schedule information table in the database through the user ID and the schedule ID, and then perform subsequent data processing. Refer to Figure 4 , in the business database in the advertising user behavior statistics scenario, it can include a user information database and a schedule information database, and each of the user information database and the schedule information database can contain one or more data tables.

[0092] Since the stream data to be processed received by the log receiving service is passed to the stream processing task through a message queue, the message queue topic corresponding to the stream data to be processed in the stream processing task can be determined according to the business processing requirements. For example, the message queue topic can include "advertising schedule", etc. The stream processing task can query the corresponding data stream table and the dimension table to be associated from the business database according to the message queue topic corresponding to the received stream data to be processed.

[0093] The data stream table can be a table structure based on stream data, used to store and process real-time flowing data; it can be regarded as a dynamic table, where the data is continuously updated and changed over time. The data stream table has characteristics such as data real-time, sequentiality, and unboundedness. For example, the data stream table in the "advertising schedule" scenario can include but is not limited to the user behavior data table.

[0094] The associated dimension table is a relatively static data table, used to store dimension information related to the stream data. It usually contains some attributes and metadata used to describe the characteristics of the stream data, and this information can be used to enrich the stream data for more in-depth analysis and processing.

[0095] The main role of the associated dimension table is to provide additional context information for the stream data. By associating the data in the stream table with the associated dimension table, more abundant information can be obtained, so as to better understand and process the stream data. For example, when analyzing order stream data, by associating with the product information dimension table, the detailed information of the products involved in each order can be learned, such as the brand and specifications of the products, which is very helpful for accurate sales analysis and marketing strategy formulation.

[0096] The dimension table to be associated in this disclosure can be an associated dimension table to be associated with the data flow table. For example, the dimension table to be associated in the "advertising schedule" scenario can include, but is not limited to, the schedule information table. By determining the data flow table corresponding to the data stream to be processed and the dimension table to be associated, subsequent dimension table association operations can be performed based on the above data tables.

[0097] In an embodiment of the present disclosure, for step S320, subscribing to the incremental data broadcast message according to the data mapping information between the business database table and the message queue topic includes: obtaining the pre-constructed meta database, which is used to store the pre-written data mapping information; obtaining the data mapping information based on the meta database; and subscribing to the incremental data broadcast message according to the data mapping information.

[0098] Among them, the meta database is a database specifically used to store and manage metadata. For example, this disclosure can store the data mapping information in the meta database.

[0099] To solve the problem of data real-time in dimension table association processing, this disclosure proposes to synchronize the data tables in the business database to the dimension table association logic in real time by means of broadcasting. Refer to Figure 5 , Figure 5 Schematically shows a processing logic diagram of the incremental subscription function in the dimension table association scheme according to some embodiments of the present disclosure. When realizing data synchronization based on broadcasting, the data mapping relationship between the business database table and the message queue topic can be written into the meta database in advance for external query and subscription. For example, the dimension table association logic in the stream processing task can obtain the data mapping information through the meta database.

[0100] The dimension table association logic starts a thread to subscribe to the incremental data broadcast message corresponding to the business database table according to the data mapping information obtained from the meta database, and obtains the incremental data in the incremental message queue, so as to update the incremental data to the local cache in real time. Subscribing to the incremental data broadcast message through the data mapping information enables the dimension table association logic to obtain the incremental data of the business data table in real time to achieve data synchronization.

[0101] In an embodiment of the present disclosure, before subscribing to the incremental data broadcast message according to the data mapping information, receive the broadcast opening operation for the business database table, generate a broadcast opening configuration statement based on the broadcast opening operation; and according to the broadcast opening configuration statement, enable the incremental data broadcast function of the business database table, where the incremental data broadcast function is used to support the message subscription operation of the incremental data broadcast message.

[0102] Among them, the broadcast start operation can be an operation to start a data table in a certain business database for the message broadcast function. The broadcast start configuration statement can be a database configuration statement used to control the broadcast start operation. The incremental data broadcast function can be a function for a data table in a certain business database to broadcast incremental data messages. The message subscription operation can be an operation for a data consumer to subscribe to incremental data broadcast messages.

[0103] To solve the data real-time problem through broadcasting, the present disclosure provides an incremental data broadcast function for business database tables. For example, for a data table in a business database, a user can perform a broadcast start operation on the business database table, and generate a broadcast start configuration statement for the business database table based on the broadcast start operation. After adding an SQL statement for configuring the broadcast start in the business database table, the incremental data broadcast function of the business database table can be started.

[0104] In addition, an SQL statement can also be added to the stream processing task: SET ad_schedule_detail.connector.lookup.listen = true to enable this function in the stream processing task, and support the message subscription operation of the incremental data broadcast message for the dimension table association logic in the stream processing task. At the same time, Figure 4 By enabling the broadcast function in the incremental data synchronization module in, the real-time dimension table association ability with a second-level delay can be used to solve the data synchronization problem between the business database and the dimension table association logic. Through simple parameter configuration by the user, the broadcast message subscription function can be enabled, and the broadcast subscription start function has ease of use.

[0105] In an embodiment of the present disclosure, for step S330, synchronizing the incremental data of the business database table to the local cache of the dimension table association logic includes: determining the incremental data of the business database table, and writing the incremental data of the business database table into an external data storage component; based on the incremental data broadcast message, writing the incremental data of the business database table into the message queue topic corresponding to the incremental message queue; synchronizing the incremental data of the business database table to the local cache of the dimension table association logic through the incremental message queue.

[0106] Among them, the external data storage component can be a storage component outside the stream processing task and the business database. The incremental message queue can be an asynchronous communication container for storing incremental data. The incremental message queue of the present disclosure can be used to store the incremental data that needs to be transmitted between the associated dimension table logic and the business database.

[0107] When there is an update to the data table in the business database, the updated data in the above data table can be used as the incremental data of the business database table. After determining the incremental data of the business database table, it is written into an external data storage component. For example, the external data storage component can include, but is not limited to, databases such as Redis, HBase, and TiDB.

[0108] For the incremental data of the business database table, if the user enables the broadcast ability of the specified business database table, the incremental subscription function is responsible for writing the incremental data of the database table into the incremental message queue for the dimension table association logic to consume in a timely manner. Continue to refer to Figure 4 And Figure 5 , the incremental data of the business database table is written into the message queue topic corresponding to the incremental message queue. If the dimension table association logic subscribes to the incremental data broadcast message of the business database table, the incremental message queue can synchronize the incremental data of the business database table to the local cache of the dimension table association logic. The incremental message queue writes the incremental data of the business database table into the local cache of the dimension table association logic based on the incremental data broadcast message, which can realize the data synchronization between the dimension table association logic and the business database.

[0109] It is easy for those skilled in the art to understand that the present disclosure includes passing the to-be-processed stream data in the log receiving service to the message queue of the stream processing task, and also includes the incremental message queue in the incremental subscription function. The two can be two different message queues, which are containers for passing messages or data between different data producers and data consumers for asynchronous communication.

[0110] In an embodiment of the present disclosure, for step S340, performing the dimension table association task based on the local cache of the dimension table association task to obtain the real-time associated dimension table includes: obtaining the message queue topic corresponding to the incremental data of the business database table from the pre-constructed meta database; starting an incremental data writing thread, and synchronizing the incremental data of the business database table in the message queue topic to the local cache through the incremental data writing thread; updating the business database table in the local cache to obtain the real-time business database table; and performing the dimension table association task based on the real-time business database table to obtain the real-time associated dimension table.

[0111] Among them, the incremental data writing thread can be a thread for performing incremental data writing operations, which is used to write the incremental data of the business database table in the relevant message queue topic of the incremental message queue into the local cache of the dimension table association logic. The real-time business database table can be a business data table after real-time data update processing.

[0112] Refer to Figure 6 , Figure 6 Schematically shows the processing logic diagram of real-time dimension table association in the dimension table association scheme according to some embodiments of the present disclosure. Refer to Figure 6, if the user enables the data monitoring function, the dimension table association logic can start the following core functions: Since the new data mapping information is pre-stored in the meta-database, according to the data mapping relationship between the business database table and the message queue topic, the message queue topic corresponding to the incremental data of the specified business library table can be obtained from the pre-constructed meta-database; and a background thread is started, such as an incremental data writing thread, which consumes the incremental data of the business library table in the specified message queue topic in real time through the incremental data writing thread, and writes the relevant data to the local cache after processing.

[0113] Writing the processed incremental data of the business library table to the local cache can realize the data update of the specified business database table in the local cache, obtain the real-time business database table, and subsequent perform the dimension table association task based on the real-time business database table to obtain the real-time associated dimension table, ensuring the stability and consistency of the dimension table data.

[0114] In an embodiment of the present disclosure, performing a dimension table association task based on the real-time business database table to obtain a real-time associated dimension table includes: determining the data stream table fields corresponding to the real-time data stream table and the to-be-associated dimension table fields corresponding to the real-time to-be-associated dimension table; determining a data table connection statement based on a business processing request; and associating and processing the data stream table fields with the to-be-associated dimension table fields according to the data table connection statement to obtain a real-time associated dimension table.

[0115] Among them, the real-time data stream table can be a stream table after real-time data update processing. The real-time to-be-associated dimension table can be a to-be-associated dimension table after real-time data update processing.

[0116] The real-time business database table can include a real-time data stream table and a real-time to-be-associated dimension table, and respectively determine the data stream table fields corresponding to the real-time data stream table and the to-be-associated dimension table fields corresponding to the real-time to-be-associated dimension table.

[0117] Taking the "advertising schedule" scenario of advertising user behavior statistics as an example, the data stream table can be a user behavior data table, and the definition of the data stream table (such as the user behavior data table) is as follows:

[0118]

[0119]

[0120] The to-be-associated dimension table can be a schedule information table, and the detailed definition of the to-be-associated dimension table (such as the schedule information table) is as follows:

[0121]

[0122] Based on the above definitions of the data stream table and the to-be-associated dimension table, a data table connection statement for performing an association operation on the two data tables can be determined based on a business processing request, as follows:

[0123] Dimensional table association real-time SQL:

[0124]

[0125] LEFT JOIN catalog.db.ad_schedule_detail FOR SYSTEM_TIME AS OF t1.proctime AS t2

[0126] ON t1.schedule_id = t2.schedule_id;

[0127] In the above way, a Kafka stream table ad_user_action and a MySQL dimensional table ad_schedule_detail are defined, and then the stream table is associated with the dimensional table through the LEFT JOIN syntax. Kafka is an open-source stream processing platform, whose goal is to provide a unified, high-throughput, and low-latency platform for processing real-time data.

[0128] The above data table connection statement associates the data stream table fields with the fields of the dimensional table to be associated, and obtains a real-time associated dimensional table after managing and processing the two data tables. Based on the real-time business database table for dimensional table association operations, it is possible to use incremental data in the dimensional table association in a timely manner, thereby ensuring data accuracy and achieving end-to-end data latency within seconds.

[0129] In an embodiment of the present disclosure, monitor metric data corresponding to the real-time associated dimensional table is obtained. The monitor metric data includes one or more of the incremental data consumption quantity, data consumption latency time, updated data volume, and data health. A pre-configured monitoring and warning condition is obtained. The monitoring and warning condition includes a monitoring metric reference threshold corresponding to each monitor metric data. The task processing status is determined based on the monitor metric data and the monitoring and warning condition. When the task processing status is an insecure state, an alarm prompt message is generated based on the monitor metric data, and the alarm prompt message is displayed.

[0130] Among them, the monitor metric data can be specific data used to measure and reflect the task execution status and performance of the stream processing task. The monitoring and warning condition can be a determination condition for triggering the generation of an alarm prompt message. The monitoring metric reference threshold can be a reference threshold used to compare with the specific value of the monitor metric data. The task processing status can be used to reflect the health status of the stream processing task. The alarm prompt message can be a message automatically generated and displayed by the system when an abnormal situation occurs in the stream processing task, and is used to prompt information that needs attention and corresponding measures to be taken during the execution of the stream processing task.

[0131] In the entire data processing solution, when the incremental subscription function writes the incremental data of the business library table into the incremental message queue, it can also write the relevant monitoring metric data of the incremental data into the monitoring and alerting system. The monitoring metric data related to the incremental data reported by the incremental subscription function can include, but is not limited to, the updated data volume and data health, etc. The updated data volume, that is, the number of incremental data records, can be the specific quantity of the incremental data. The data health can be an indicator reflecting the health status of the relevant data in the stream processing task.

[0132] When the dimension table association logic writes the incremental data of the business library table into the local cache, it can also report the relevant monitoring data of the incremental data to the monitoring and alerting system. The monitoring metric data related to the incremental data reported by the dimension table association logic can include, but is not limited to, the incremental data consumption quantity and data consumption latency, etc. The incremental data consumption quantity can be the incremental data that has been consumed by the data consumer. The data consumption latency can be the latency time generated when the data consumer consumes the incremental data.

[0133] The monitoring and alerting system performs full-link monitoring and alerting based on the received monitoring metric data to ensure the accurate and stable operation of the data and tasks. When relevant problems occur, it can promptly detect and notify relevant personnel for handling. Through the monitoring and alerting system, the monitoring and alerting conditions can be pre-configured. The monitoring and alerting conditions include the monitoring metric reference thresholds corresponding to each monitoring metric data, and then judge whether the operating status of each part is normal according to the monitoring metric reference thresholds.

[0134] For example, the monitoring and alerting conditions can include, but are not limited to: the health status of the incremental subscription module and the dimension table association module, whether they are operating normally; the latency of the real-time consumption threads of the dimension table association module; the matching situation of the incremental data writing and consumption levels, whether they are consistent, to ensure data accuracy; the end-to-end latency of the data from production to final use, etc.

[0135] According to the monitoring metric data and the monitoring and alerting conditions, the task processing status of the stream processing task is monitored in real time; if it is detected that the task processing status is in an insecure state, an alert prompt message will be generated based on the comparison result of the monitoring metric data and the monitoring and alerting conditions, and the alert prompt message will be sent to relevant personnel so that relevant personnel can handle it in a timely manner. By equipping a perfect monitoring and alerting mechanism, it is ensured that the incremental data is processed in a timely and effective manner, and the stability and operability of the entire data link are guaranteed.

[0136] In summary, for the data processing method of the present disclosure, the to-be-processed streaming data is obtained, and the business database table corresponding to the to-be-processed streaming data is determined. The business database table is used to generate an associated dimension table. The incremental data broadcast message is subscribed according to the data mapping information between the business database table and the message queue topic. In response to the received incremental data broadcast message, the incremental data of the business database table corresponding to the business database table is determined, and the incremental data of the business database table is synchronized to the local cache of the dimension table association logic. The dimension table association task is executed based on the local cache of the dimension table association logic to obtain a real-time associated dimension table. On the one hand, by subscribing to the broadcast message, relevant data changes can be monitored in real time, and the changed incremental data is synchronized to the local cache. On the premise of ensuring the task performance, data can be updated at the millisecond level, ensuring that the end-to-end delay of the data reaches the second level, achieving the ability to update the dimension table data at the second level. On the other hand, by performing dimension table association through data broadcast and local cache, the stability and consistency of the dimension table data can be guaranteed. On the further hand, users can enable the broadcast message subscription function through simple parameter configuration, and the broadcast subscription enabling function is easy to use.

[0137] Exemplary Apparatus

[0138] After introducing the method of the exemplary embodiment of the present disclosure, next, reference is made to Figure 7 to describe the data processing device of the exemplary embodiment of the present disclosure.

[0139] In Figure 7 , the data processing device 700 may include: a service table determination module 710, a message subscription module 720, a data synchronization module 730, and a dimension table association module 740.

[0140] Among them, the service table determination module 710 is used to obtain the to-be-processed streaming data and determine the business database table corresponding to the to-be-processed streaming data. The business database table is used to generate an associated dimension table. The message subscription module 720 is used to subscribe to the incremental data broadcast message according to the data mapping information between the business database table and the message queue topic. The data synchronization module 730 is used to determine the incremental data of the business database table corresponding to the business database table in response to the received incremental data broadcast message, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic. The dimension table association module 740 is used to execute the dimension table association task based on the local cache of the dimension table association logic to obtain a real-time associated dimension table.

[0141] In an embodiment of the present disclosure, the business database table includes a data flow table and a to-be-associated dimension table. The service table determination module 710 includes a service table determination unit, which is used to: determine the business processing requirements of the to-be-processed streaming data, and determine the message queue topic corresponding to the to-be-processed streaming data in the stream processing task according to the business processing requirements; determine the data flow table and the to-be-associated dimension table from the business database according to the message queue topic.

[0142] In one embodiment of the present disclosure, the message subscription module 720 includes a message subscription unit for: obtaining a pre-constructed metadata database for storing pre-written data mapping information; obtaining data mapping information based on the metadata database; and subscribing to incremental data broadcast messages according to the data mapping information.

[0143] In one embodiment of the present disclosure, the data processing device 700 further includes a broadcast function enabling module for: receiving a broadcast enabling operation for a service database table, generating a broadcast enabling configuration statement based on the broadcast enabling operation; and enabling the incremental data broadcast function of the service database table according to the broadcast enabling configuration statement, where the incremental data broadcast function is used to support the message subscription operation of incremental data broadcast messages.

[0144] In one embodiment of the present disclosure, the data synchronization module 730 includes a data synchronization unit for: determining incremental data of a service database table, writing the incremental data of the service database table to an external data storage component; writing the incremental data of the service database table to a message queue topic corresponding to an incremental message queue based on the incremental data broadcast message; and synchronizing the incremental data of the service database table to the local cache of the dimension table association logic through the incremental message queue.

[0145] In one embodiment of the present disclosure, the dimension table association module 740 includes a dimension table association unit for: obtaining a message queue topic corresponding to incremental data of a service database table from a pre-constructed metadata database; starting an incremental data writing thread, and synchronizing the incremental data of the service database table in the message queue topic to the local cache through the incremental data writing thread; updating the service database table in the local cache to obtain a real-time service database table; and performing a dimension table association task based on the real-time service database table to obtain a real-time associated dimension table.

[0146] In one embodiment of the present disclosure, the real-time service database table includes a real-time data flow table and a real-time dimension table to be associated, and the dimension table association unit includes a dimension table association sub-unit for: determining a data flow table field corresponding to the real-time data flow table and an associated dimension table field corresponding to the real-time dimension table to be associated; determining a data table connection statement based on a service processing request; and performing an association process on the data flow table field and the associated dimension table field according to the data table connection statement to obtain a real-time associated dimension table.

[0147] In one embodiment of the present disclosure, the data processing device 700 further includes a monitoring and alarming module, which is configured to: obtain monitoring metric data corresponding to a real-time associated dimension table, where the monitoring metric data includes one or more of the incremental data consumption quantity, data consumption latency, updated data volume, and data health; obtain pre-configured monitoring and alarming conditions, where the monitoring and alarming conditions include monitoring metric reference thresholds corresponding to the respective monitoring metric data; determine a task processing status based on the monitoring metric data and the monitoring and alarming conditions; and when the task processing status is an insecure status, generate an alarm prompt message based on the monitoring metric data and display the alarm prompt message.

[0148] Since each functional module of the data processing device in the exemplary embodiments of the present disclosure corresponds to the steps in the exemplary embodiments of the above data processing method, for details not disclosed in the device embodiments of the present disclosure, please refer to the embodiments of the above data processing method of the present disclosure, which will not be elaborated herein.

[0149] It should be noted that although several modules or units of the data processing device are mentioned in the above detailed description, such a division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0150] Exemplary Medium

[0151] After introducing the device of the exemplary embodiments of the present disclosure, next, reference is made to Figure 8 to describe the storage medium of the exemplary embodiments of the present disclosure.

[0152] In some embodiments, various aspects of the present disclosure can also be implemented as a medium having program code stored thereon, which, when executed by a processor of a device, is used to implement the steps in the data processing method according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0153] For example, when the processor of the device executes the program code, it can implement as Figure 3In step S310 described above, obtain the data stream to be processed, and determine the business database table corresponding to the data stream to be processed. The business database table is used to generate an associated dimension table; step S320, subscribe to the incremental data broadcast message according to the data mapping information between the business database table and the message queue topic; step S330, in response to the received incremental data broadcast message, determine the incremental data of the business database table corresponding to the business database table, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic; step S340, perform a dimension table association task based on the local cache of the dimension table association logic to obtain a real-time associated dimension table.

[0154] Reference Figure 8 As shown, a program product 800 for implementing the above data processing method or implementing the above data processing method according to an embodiment of the present disclosure is described. It may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this.

[0155] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0156] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than the readable storage medium.

[0157] Program code for performing the operations of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0158] Exemplary Computing Device

[0159] After introducing the data processing method, data processing device, and storage medium of the exemplary embodiments of the present disclosure, next, reference is made to Figure 9 to describe the electronic device of the exemplary embodiments of the present disclosure.

[0160] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, method, or program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module", or "system" here.

[0161] In some possible embodiments, the electronic device according to the present disclosure may at least include at least one processing unit and at least one storage unit. Among them, the storage unit stores program code, and when the program code is executed by the processing unit, the processing unit is caused to execute the steps in the data processing method according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the processing unit can execute steps such as Figure 3 shown in S310, obtain the stream data to be processed, determine the business database table corresponding to the stream data to be processed, and the business database table is used to generate an associated dimension table; step S320, subscribe to the incremental data broadcast message according to the data mapping information between the business database table and the message queue topic; step S330, in response to the received incremental data broadcast message, determine the incremental data of the business database table corresponding to the business database table, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic; step S340, perform the dimension table association task based on the local cache of the dimension table association logic to obtain the real-time associated dimension table.

[0162] Next, reference is made to Figure 9 to describe the electronic device 900 according to the exemplary embodiments of the present disclosure. Figure 9The illustrated electronic device 900 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present disclosure.

[0163] As Figure 9 shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one of the above-mentioned processing units 901, at least one of the above-mentioned storage units 902, a bus 903 connecting different system components (including the storage unit 902 and the processing unit 901), and a display unit 907.

[0164] The bus 903 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures.

[0165] The storage unit 902 may include a readable medium in the form of volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922, and may further include a read-only memory (ROM) 923.

[0166] The storage unit 902 may also include a program / utilities 925 having a set (at least one) of program modules 924. Such program modules 924 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0167] The electronic device 900 may also communicate with one or more external devices 904 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or may communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface 905. Moreover, the electronic device 900 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 906. As shown in the figure, the network adapter 906 communicates with other modules of the electronic device 900 through the bus 903. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0168] It should be noted that although several units / modules or sub-units / modules of the data processing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / modules. Conversely, the features and functions of one unit / modules described above can be further divided and embodied by multiple units / modules.

[0169] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0170] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefits. This division is only for the convenience of expression. The present disclosure aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data processing method, characterized in that, Including: Obtain the data stream to be processed, and determine the business database table corresponding to the data stream to be processed, where the business database table is used to generate an associated dimension table; Subscribe to the incremental data broadcast message according to the data mapping information between the business database table and the message queue topic; In response to the received incremental data broadcast message, determine the incremental data of the business database table corresponding to the business database table, and synchronize the incremental data of the business database table to the local cache of the dimension table association logic; Execute the dimension table association task based on the local cache of the dimension table association logic to obtain a real-time associated dimension table.

2. The method according to claim 1, characterized in that, The business database table includes a data stream table and a dimension table to be associated. Determining the business database table corresponding to the data stream to be processed includes: Determine the business processing requirements of the data stream to be processed, and determine the message queue topic corresponding to the data stream to be processed in the stream processing task according to the business processing requirements; According to the message queue topic, determine the data stream table and the dimension table to be associated from the business database.

3. The method according to claim 1, wherein The subscribing to the incremental data broadcast message according to the data mapping information between the business database table and the message queue topic includes: Obtain a pre-constructed meta database, where the meta database is used to store the pre-written data mapping information; Obtain the data mapping information based on the meta database; Subscribe to the incremental data broadcast message according to the data mapping information.

4. The method according to claim 3, characterized in that, Before subscribing to the incremental data broadcast message according to the data mapping information, the method further includes: Receive a broadcast start operation for the business database table, and generate a broadcast start configuration statement based on the broadcast start operation; According to the broadcast start configuration statement, enable the incremental data broadcast function of the business database table, where the incremental data broadcast function is used to support the message subscription operation of the incremental data broadcast message.

5. The method according to claim 1, characterized in that, The synchronizing the incremental data of the business database table to the local cache of the dimension table association logic includes: Determine the incremental data of the business database table, and write the incremental data of the business database table to an external data storage component; Based on the incremental data broadcast message, write the incremental data of the business database table to the message queue topic corresponding to the incremental message queue; Synchronize the incremental data of the business database table to the local cache of the dimension table association logic through the incremental message queue.

6. The method according to claim 1, wherein The executing the dimension table association task based on the local cache of the dimension table association task to obtain a real-time associated dimension table includes: Obtain the message queue topic corresponding to the incremental data of the business database table from a pre-constructed meta database; Start an incremental data writing thread, and synchronize the incremental data of the business database table in the message queue topic to the local cache through the incremental data writing thread; Update the business database table in the local cache to obtain a real-time business database table; Execute the dimension table association task based on the real-time business database table to obtain the real-time associated dimension table.

7. The method according to claim 6, characterized in that, The real-time business database table includes a real-time data stream table and a real-time dimension table to be associated. Executing the dimension table association task based on the real-time business database table to obtain the real-time associated dimension table includes: Determine the data flow table fields corresponding to the real-time data flow table and the to-be-associated dimension table fields corresponding to the real-time to-be-associated dimension table; Determine the data table connection statement based on the service processing request; Associate the data flow table fields with the to-be-associated dimension table fields according to the data table connection statement to obtain the real-time associated dimension table.

8. A data processing device, characterized in that, Including: A service table determination module, configured to obtain the to-be-processed flow data and determine the service database table corresponding to the to-be-processed flow data, where the service database table is used to generate an associated dimension table; A message subscription module, configured to subscribe to an incremental data broadcast message according to the data mapping information between the service database table and the message queue topic; A data synchronization module, configured to, in response to the received incremental data broadcast message, determine the incremental data of the service database table corresponding to the service database table and synchronize the incremental data of the service database table to the local cache of the dimension table association logic; A dimension table association module, configured to execute a dimension table association task based on the local cache of the dimension table association logic to obtain a real-time associated dimension table.

9. An electronic device, characterized in that, Including: A processor; And A memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the data processing method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images