Data association method, device, equipment and storage medium thereof

Abstract

The embodiments of the present application belong to the field of financial technology and are applied to scenarios of data interaction and processing in financial business. They involve a data association method, apparatus, device, and storage medium thereof, including creating and initializing a data dimension table in Apache Hudi; monitoring data changes that occur during the interaction process; identifying changed dimension data; incrementally updating the changed dimension data to the data dimension table; sending the data changes and the changed dimension data to the Apache Flink real-time computing engine; performing association operations; and completing the real-time association of data in the target interaction process based on the results of the association operations. By utilizing the feature of Apache Hudi that supports incremental calculations, the amount of data loaded into the dimension table each time during the real-time association calculation process is reduced, thereby improving the efficiency of the association calculation. A clearing strategy for dimension table cache data is designed through the LinkedHashMap linked list, which reduces the memory pressure of real-time tasks, makes the caching method more reasonable, and improves the stability of real-time tasks.

Description

Technical Field

[0001] The present application relates to the field of financial technology and is applied to scenarios of financial business data interaction processing, and in particular to a data association method, apparatus, device and storage medium thereof. Background Art

[0002] With the rapid development of the internet, various industries are seeking breakthroughs through the internet. In recent years, the financial industry has also been expanding its online business around the internet. Due to the large volume of business and data involved in the financial industry, and the increasing demand for products from users, the timeliness requirements for data processing have also increased. Currently, many business scenarios require real-time data processing and real-time delivery of processed results to users. Real-time processing technology in the field of big data has also been rapidly developing in recent years. However, the data processing details of real-time streaming technology differ significantly from traditional offline technology, and migrating traditional offline processing logic to real-time data stream processing scenarios presents many difficulties.

[0003] Apache Flink, the industry's mainstream real-time computing processing engine, still has many problems with associating dimension table information during the calculation process:

[0004] First, if dimension table information is stored in a relational database, which supports transactions and indexing services, dimension table information can be quickly read and written. However, relational databases are high-end storage, and the hardware and operation and maintenance costs associated with them are very high. If a relational database is abandoned and Apache HBase is selected as the dimension table storage system, the operation and maintenance of a set of open source components will also be required.

[0005] Second, if Apache Hive, an essential basic storage component for big data, is used, Flink cannot dynamically monitor dimension table changes at a high frequency. The official minimum recommended monitoring interval is one hour. A frequency higher than one hour would put significant pressure on the Hive metadata service. However, an update timeliness of one hour clearly does not meet the timeliness requirements of real-time data.

[0006] Third, each time Flink reads or updates the contents of a Hive dimension table, it needs to read the entire dimension table data into memory and cache it, replacing the previous version of the dimension table data with the newly read dimension table data. If the dimension table data is too large, the cached dimension table data will put enormous pressure on the task memory, easily causing task memory overflow exceptions.

[0007] Therefore, the existing use of the Flink computing engine for big data computing still has problems such as excessive amount of data loaded into dimension tables each time during real-time association calculations, resulting in low computing efficiency and unreasonable caching methods. Summary of the Invention

[0008] The purpose of the embodiments of the present application is to propose a data association method, apparatus, device, and storage medium thereof to address the problems in the prior art of using the Flink computing engine for big data computing, such as excessive amount of data loaded into dimension tables each time during real-time association calculations, resulting in low computing efficiency and unreasonable caching.

[0009] In order to solve the above technical problems, the present application provides a data association method, which adopts the following technical solutions:

[0010] A data association method comprises the following steps:

[0011] In response to the dimension table creation instruction sent by the data maintenance end, create a data dimension table in Apache Hudi according to the dimension table creation instruction;

[0012] In response to the dimension data initialization instruction sent by the data maintenance end, the dimension data in the target source end database is initialized into the data dimension table, and the initialized data dimension table is obtained;

[0013] According to the preset real-time monitoring component, the data changes occurring in the target source database during the target interaction process are monitored and the data change results are obtained;

[0014] According to the data change result, identifying the dimension data corresponding to the change in the target interaction process as the target dimension data;

[0015] Reading the target dimension data from the target source database, and incrementally updating the target dimension data to the initialized data dimension table using data synchronization technology to obtain an incrementally updated data dimension table;

[0016] Using Apache Kafka message streams, data changes in the target source database are sent as message stream data to the Apache Flink real-time computing engine.

[0017] Querying the target dimension data that is synchronously updated in the incrementally updated data dimension table through an incremental query method, and sending the target dimension data to the Apache Flink real-time computing engine;

[0018] Performing an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data;

[0019] The real-time association of data in the target interaction process is completed according to the mapping relationship between the message stream data and the target dimension data.

[0020] Furthermore, the step of incrementally updating the target dimension data into the initialized data dimension table using data synchronization technology to obtain the incrementally updated data dimension table specifically includes:

[0021] The target dimension data is incrementally updated to the initialized data dimension table using Sqoop offline synchronization or Flink CDC real-time synchronization to obtain an incrementally updated data dimension table.

[0022] Furthermore, after executing the step of sending the data changes occurring in the target source database as message stream data to the Apache Flink real-time computing engine through the Apache Kafka message stream, the method further includes:

[0023] Start the real-time reading task preset in the Apache Flink real-time computing engine;

[0024] Based on the real-time reading task, the message stream data sent in the Apache Kafka message stream mode is read, and after reading the message stream data, the preset associated computing task in the Apache Flink real-time computing engine is triggered and started according to the preset trigger component.

[0025] Furthermore, before executing the step of triggering and starting the preset associated computing task in the Apache Flink real-time computing engine according to the preset trigger component, the method further includes:

[0026] According to a preset judgment component, it is judged whether the associated calculation task has preloaded the initialized data dimension table;

[0027] If the associated calculation task has not pre-loaded the initialized data dimension table, the preset loading component is started to load the initialized data dimension table into the memory corresponding to the associated calculation task, and the data in the initialized data dimension table is cached in the memory in the form of a LinkedHashMap linked list.

[0028] Furthermore, the step of querying the synchronously updated target dimension data in the incrementally updated data dimension table by incremental query specifically includes:

[0029] By using the preset incremental query component in the Apache Flink real-time computing engine, the initialized data dimension table is used as a reference table, and the changed data content of the incrementally updated data dimension table relative to the reference table is queried through a comparative query method;

[0030] Using the data content changed relative to the reference table as the target dimension data to be synchronously updated in the incrementally updated data dimension table;

[0031] After executing the step of querying the synchronously updated target dimension data in the incrementally updated data dimension table by incremental query, the method further includes:

[0032] Identify the LinkedHashMap linked list corresponding to the initialized data dimension table;

[0033] The target dimension data is added as a new element of the LinkHashMap linked list to the head position of the queue of the LinkHashMap linked list.

[0034] Furthermore, after executing the step of performing an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data, the method further includes:

[0035] The position information of the target dimension data in the LinkedHashMap linked list is updated from the head position of the queue to the tail position of the queue of the LinkedHashMap linked list.

[0036] Furthermore, after executing the step of updating the position information of the target dimension data in the LinkedHashMap linked list from the head position of the queue to the tail position of the queue of the LinkedHashMap linked list, the method further includes:

[0037] The data volume calculation component preset in the Apache Flink real-time computing engine is used to calculate the data volume in the LinkedHashMap table corresponding to the current data dimension table, as the memory required for caching the current data dimension table.

[0038] Use comparison to determine whether the memory required for caching the current data dimension table has reached the preset cleanup threshold;

[0039] If the memory required for caching the current data dimension table has reached the cleaning threshold, a preset proportion of linked list data is deleted from the queue head position to the queue tail position of the LinkedHashMap linked list.

[0040] In order to solve the above technical problems, the embodiment of the present application also provides a data association device, which adopts the following technical solutions:

[0041] A data association device, comprising:

[0042] A data dimension table creation module is used to create a data dimension table in Apache Hudi in response to a dimension table creation instruction sent by the data maintenance terminal according to the dimension table creation instruction;

[0043] A data dimension table initialization module is configured to initialize the dimension data in the target source database into the data dimension table in response to a dimension data initialization instruction sent by the data maintenance terminal, and obtain the initialized data dimension table;

[0044] A real-time monitoring module is used to monitor data changes in the target source database during the target interaction process according to a preset real-time monitoring component and obtain data change results;

[0045] a target dimension data identification module, configured to identify, based on the data change result, dimension data corresponding to the change in the target interaction process as target dimension data;

[0046] A data dimension table update module is configured to read the target dimension data from the target source database and incrementally update the target dimension data to the initialized data dimension table using data synchronization technology to obtain an incrementally updated data dimension table;

[0047] A message stream sending module is used to send data changes occurring in the target source database as message stream data to the Apache Flink real-time computing engine through the Apache Kafka message stream method;

[0048] A target dimension data sending module is used to query the target dimension data that is synchronously updated in the incrementally updated data dimension table through an incremental query method, and send the target dimension data to the Apache Flink real-time computing engine;

[0049] An engine association operation module, configured to perform an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data, to obtain a mapping relationship between the message stream data and the target dimension data;

[0050] The real-time data association module is used to complete the real-time data association in the target interaction process according to the mapping relationship between the message stream data and the target dimension data.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0052] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data association method described above when executing the computer-readable instructions.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0054] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data association method described above.

[0055] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0056] The data association method described in the embodiment of the present application is as follows: creating a data dimension table in Apache Hudi; initializing the dimension data in the target source database into the data dimension table; monitoring the data changes in the target source database during the target interaction process; identifying the dimension data corresponding to the change in the target interaction process; incrementally updating the changed dimension data into the data dimension table; sending the data changes in the target source database as message stream data to the Apache Flink real-time computing engine; obtaining and sending the changed dimension data to the Apache Flink real-time computing engine through incremental query; performing association operations in the Apache Flink real-time computing engine based on the message stream data and the changed dimension data to obtain a mapping relationship between the message stream data and the changed dimension data; and completing the real-time association of data during the target interaction process based on the mapping relationship. By utilizing the feature of Apache Hudi that supports incremental calculation, the amount of data loaded in the dimension table each time during the real-time association calculation process is reduced, thereby improving the task efficiency of the real-time association calculation. An expiration elimination strategy for the dimension table memory data is designed through the LinkedHashMap linked list, thereby reducing the memory pressure of the real-time task and improving the stability of the real-time task. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0059] Figure 2 is a flow chart of an embodiment of a data association method according to the present application;

[0060] Figure 3 is a structural diagram of an embodiment of a data association device according to the present application;

[0061] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0063] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0065] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0066] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0067] Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0068] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0069] It should be noted that the data association method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the data association device is generally set in the server / terminal device.

[0070] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0071] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data association method according to the present application. The data association method comprises the following steps:

[0072] Step 201: In response to a dimension table creation instruction sent by a data maintenance terminal, a data dimension table is created in Apache Hudi according to the dimension table creation instruction.

[0073] In this embodiment, the data maintenance terminal includes a financial business data maintenance terminal, which is generally equipped with corresponding maintenance personnel or an automated maintenance system. The dimension table creation instructions include command line instructions when creating a form in the Apache Hudi data warehouse. The data dimension table represents identification data that distinguishes each piece of data, such as time ID, customer ID, product ID, supplier ID, device ID, etc. The data dimension table generally does not contain data that can be repeated between multiple pieces of data, such as unit price, age, etc.

[0074] Step 202 : In response to the dimension data initialization instruction sent by the data maintenance end, the dimension data in the target source end database is initialized into the data dimension table, and an initialized data dimension table is obtained.

[0075] In essence, the initialization is to initialize the dimension data in the target source database into the data dimension table, thereby separating the data dimension table from the source data and facilitating subsequent data association.

[0076] Step 203: According to a preset real-time monitoring component, data changes occurring in the target source database during the target interaction process are monitored to obtain data change results.

[0077] Specifically, the monitoring involves monitoring data changes occurring in the target source database during target interaction, including writing new data, modifying old data, deleting data, and invoking data. The purpose of monitoring via the real-time monitoring component is to ensure that, upon data changes occurring in the target source database, the financial service processing system corresponding to the target source database can promptly make business adjustments based on the data changes.

[0078] Step 204 : According to the data change result, the dimensional data corresponding to the change in the target interaction process is identified as the target dimensional data.

[0079] For example, the interactive process of entering card application information for a credit card user will result in the addition of new data in the corresponding target source database. The newly added data includes data such as the credit card number, user ID information, and user ID. These data are unique and can be used as target dimension data. However, data such as name and card activation limit can be the same for different users, so such data is not used as target dimension data.

[0080] Step 205 : Read the target dimension data from the target source database, and incrementally update the target dimension data to the initialized data dimension table using data synchronization technology to obtain an incrementally updated data dimension table.

[0081] In this embodiment, the step of using data synchronization technology to incrementally update the target dimension data to the initialized data dimension table to obtain the incrementally updated data dimension table specifically includes: using Sqoop offline synchronization method or Flink CDC real-time synchronization method to incrementally update the target dimension data to the initialized data dimension table to obtain the incrementally updated data dimension table.

[0082] Sqoop is a commonly used data migration tool, mainly used to import and export data between different storage systems. Its principle is to convert execution commands into MapReduce jobs to achieve data migration and synchronize the changed data to the associated operation engine in a timely manner.

[0083] Flink CDC is an independent open source project, with the project code hosted on GitHub. The Flink community has developed the flink-cdc-connectors component, a source component that can directly read full and incremental data from databases such as MySQL and PostgreSQL. The core of this technology is the real-time synchronization and processing of full and incremental data within a table, allowing users to easily obtain a consistent, real-time snapshot of each table. Flink CDC enables real-time data synchronization, promptly synchronizing changed data to the relational computing engine.

[0084] The target dimension data is read from the target source database, and the target dimension data is incrementally updated to the initialized data dimension table using data synchronization technology to obtain the incrementally updated data dimension table. The purpose is to update the target dimension data to the data dimension table in a timely manner when the dimension data in the target source database changes.

[0085] Step 206: Send the data changes occurring in the target source database as message stream data to the Apache Flink real-time computing engine through the Apache Kafka message stream method.

[0086] In this embodiment, after executing the step of sending the data changes occurring in the target source database as message stream data to the Apache Flink real-time computing engine through the Apache Kafka message stream method, the method further includes: starting a real-time reading task preset in the Apache Flink real-time computing engine; based on the real-time reading task, reading the message stream data sent in the Apache Kafka message stream method, and after reading the message stream data, triggering and starting the associated computing task preset in the Apache Flink real-time computing engine according to a preset trigger component.

[0087] Specifically, the message flow data includes specific business data changes other than dimensional data, such as amount increase, amount decrease, and other such data.

[0088] In this embodiment, before executing the step of triggering and starting the preset associated computing task in the Apache Flink real-time computing engine according to the preset trigger component, the method further includes: judging, according to a preset judgment component, whether the associated computing task has preloaded the initialized data dimension table; if the associated computing task has not preloaded the initialized data dimension table, starting the preset loading component to load the initialized data dimension table into the memory corresponding to the associated computing task, and caching the data in the initialized data dimension table in the memory in the form of a LinkedHashMap linked list.

[0089] By loading the initialized data dimension table into the memory corresponding to the associated calculation task, it is convenient to directly obtain the corresponding dimension table data from the memory corresponding to the associated calculation task when performing the associated calculation. LinkedHashMap is a combination of HashMap and a two-way linked list, that is, a HashMap that links all Entry nodes into a two-way linked list, wherein each Entry node contains 3 attributes for representing the linked list queue relationship, namely the next pointer and before, After. The next pointer is used to maintain the order of the Entry nodes connected to the specified table position of the HashMap, and before, After are used to maintain the order of insertion of the Entry nodes. The purpose of caching the data in the initialized data dimension table in the form of a LinkedHashMap linked list in the memory is to facilitate the identification of the order of the incremental data based on the next pointer and before, After attributes in the LinkedHashMap linked list.

[0090] Step 207: query the synchronously updated target dimension data in the incrementally updated data dimension table through an incremental query method, and send the target dimension data to the Apache Flink real-time computing engine.

[0091] In this embodiment, the step of querying the target dimension data that is synchronously updated in the incrementally updated data dimension table through an incremental query method specifically includes: using the incremental query component preset in the Apache Flink real-time computing engine, using the initialized data dimension table as a reference table, and querying the data content that has changed in the incrementally updated data dimension table relative to the reference table through a comparative query method; and using the data content that has changed relative to the reference table as the target dimension data that is synchronously updated in the incrementally updated data dimension table.

[0092] In this embodiment, after executing the step of querying the target dimension data that is synchronously updated in the incrementally updated data dimension table through incremental query, the method also includes: identifying the LinkedHashMap linked list corresponding to the initialized data dimension table; and adding the target dimension data as a new element of the LinkHashMap linked list to the queue head position of the LinkHashMap linked list.

[0093] By adding the target dimension data as a new element of the LinkHashMap linked list to the head position of the queue of the LinkHashMap linked list, it is indicated that the target dimension data will be used, which facilitates obtaining the target dimension data from the LinkHashMap linked list.

[0094] Step 208: Perform an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data.

[0095] Specifically, obtaining the mapping relationship between the message stream data and the target dimension data is essentially obtaining the mapping relationship between the identification data corresponding to the target dimension data and the specific business data corresponding to the message stream data.

[0096] In this embodiment, after executing the step of performing an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data, the method further includes: updating the position information of the target dimension data in the LinkedHashMap linked list from the head position of the queue to the tail position of the queue of the LinkedHashMap linked list.

[0097] By updating the position information of the target dimension data in the LinkedHashMap linked list from the head position of the queue to the tail position of the queue of the LinkedHashMap linked list, it is indicated that the target dimension data has been used.

[0098] In this embodiment, after executing the step of updating the position information of the target dimension data in the LinkedHashMap linked list from the queue head position to the queue tail position of the LinkedHashMap linked list, the method also includes: calculating the amount of data in the LinkedHashMap linked list corresponding to the current data dimension table according to the data amount calculation component preset in the Apache Flink real-time computing engine, as the memory required for caching the current data dimension table; determining whether the memory required for caching the current data dimension table has reached a preset cleanup threshold; if the memory required for caching the current data dimension table has reached the cleanup threshold, deleting a preset proportion of linked list data from the queue head position to the queue tail position of the LinkedHashMap linked list.

[0099] Through the data volume calculation component and the preset clearing threshold, it is convenient to clear the data volume in the LinkedHashMap linked list, so that the memory occupied by the LinkedHashMap linked list is within a reasonable range. Specifically, the clearing method can be cleared using the LinkedHashMap.size()*0.2 method, where LinkedHashMap.size() represents the number of data nodes currently cached in the LinkedHashMap linked list, and 0.2 represents a multiple. The LinkedHashMap.size()*0.2 method is used for clearing, that is, clearing 20% ​​of the number of currently cached data nodes, deleting the linked list data from the queue head position to the queue tail position of the LinkedHashMap linked list, that is, deleting the earliest inserted linked list data and retaining the latest inserted linked list data. An expiration elimination strategy for dimension table memory data is designed through the LinkedHashMap linked list, which reduces the memory pressure of real-time tasks and improves the stability of real-time tasks.

[0100] Step 209: completing real-time association of data in the target interaction process according to the mapping relationship between the message stream data and the target dimension data.

[0101] By obtaining the mapping relationship between the identification data corresponding to the target dimension data and the specific business data corresponding to the message flow data, a specific business processing table is generated according to the mapping relationship between the identification data and the specific business data, thereby completing the real-time association of data in the target interaction process.

[0102] The present application creates a data dimension table in Apache Hudi; initializes the dimension data in the target source database into the data dimension table; monitors the data changes in the target source database during the target interaction process; identifies the dimension data corresponding to the target interaction process; incrementally updates the changed dimension data to the data dimension table; sends the data changes in the target source database as message stream data to the Apache Flink real-time computing engine; obtains and sends the changed dimension data to the Apache Flink real-time computing engine through incremental query; performs association operations in the Apache Flink real-time computing engine based on the message stream data and the changed dimension data to obtain the mapping relationship between the message stream data and the changed dimension data; and completes the real-time association of data in the target interaction process based on the mapping relationship. By utilizing the feature of Apache Hudi that supports incremental calculation, the amount of data loaded in the dimension table each time during the real-time association calculation process is reduced, thereby improving the task efficiency of the real-time association calculation. An expiration elimination strategy for the dimension table memory data is designed through the LinkedHashMap linked list, thereby reducing the memory pressure of the real-time task and improving the stability of the real-time task.

[0103] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0104] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data correlation technologies, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0105] In an embodiment of the present application, a data dimension table is created in Apache Hudi; dimension data in the target source database is initialized into the data dimension table; data changes in the target source database during the target interaction process are monitored; dimension data corresponding to the target interaction process is identified; the changed dimension data is incrementally updated to the data dimension table; data changes in the target source database are sent to the Apache Flink real-time computing engine as message stream data; the changed dimension data is acquired and sent to the Apache Flink real-time computing engine through incremental query; association operations are performed in the Apache Flink real-time computing engine based on the message stream data and the changed dimension data to obtain a mapping relationship between the message stream data and the changed dimension data; and real-time association of data in the target interaction process is completed based on the mapping relationship. By utilizing the feature of Apache Hudi that supports incremental calculations, the amount of data loaded in the dimension table each time during the real-time association calculation process is reduced, thereby improving the task efficiency of the real-time association calculation. An expiration elimination strategy for dimension table memory data is designed through the LinkedHashMap linked list, thereby reducing the memory pressure of the real-time task and improving the stability of the real-time task.

[0106] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in FIG, the present application provides an embodiment of a data association device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0107] like Figure 3 As shown, the data association device 300 described in this embodiment includes: a data dimension table creation module 301, a data dimension table initialization module 302, a real-time monitoring module 303, a target dimension data identification module 304, a data dimension table update module 305, a message flow sending module 306, a target dimension data sending module 307, an engine association operation module 308 and a data real-time association module 309.

[0108] in:

[0109] A data dimension table creation module 301 is configured to create a data dimension table in Apache Hudi in response to a dimension table creation instruction sent by a data maintenance terminal according to the dimension table creation instruction;

[0110] A data dimension table initialization module 302 is configured to initialize the dimension data in the target source database into the data dimension table in response to a dimension data initialization instruction sent by the data maintenance end, thereby obtaining an initialized data dimension table;

[0111] A real-time monitoring module 303 is configured to monitor data changes occurring in the target source database during target interaction according to a preset real-time monitoring component, and obtain data change results;

[0112] A target dimension data identification module 304 is configured to identify the dimension data corresponding to the target interaction process according to the data change result, as target dimension data;

[0113] The data dimension table update module 305 is configured to read the target dimension data from the target source database and incrementally update the target dimension data to the initialized data dimension table using data synchronization technology to obtain an incrementally updated data dimension table.

[0114] A message stream sending module 306 is configured to send data changes occurring in the target source database as message stream data to the Apache Flink real-time computing engine via an Apache Kafka message stream.

[0115] The target dimension data sending module 307 is used to query the target dimension data that is synchronously updated in the incrementally updated data dimension table through an incremental query method, and send the target dimension data to the Apache Flink real-time computing engine;

[0116] An engine association operation module 308 is configured to perform an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data;

[0117] The real-time data association module 309 is configured to complete the real-time data association in the target interaction process according to the mapping relationship between the message stream data and the target dimension data.

[0118] The present application creates a data dimension table in Apache Hudi; initializes the dimension data in the target source database into the data dimension table; monitors the data changes in the target source database during the target interaction process; identifies the dimension data corresponding to the target interaction process; incrementally updates the changed dimension data to the data dimension table; sends the data changes in the target source database as message stream data to the Apache Flink real-time computing engine; obtains and sends the changed dimension data to the Apache Flink real-time computing engine through incremental query; performs association operations in the Apache Flink real-time computing engine based on the message stream data and the changed dimension data to obtain the mapping relationship between the message stream data and the changed dimension data; and completes the real-time association of data in the target interaction process based on the mapping relationship. By utilizing the feature of Apache Hudi that supports incremental calculation, the amount of data loaded in the dimension table each time during the real-time association calculation process is reduced, thereby improving the task efficiency of the real-time association calculation. An expiration elimination strategy for the dimension table memory data is designed through the LinkedHashMap linked list, thereby reducing the memory pressure of the real-time task and improving the stability of the real-time task.

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0121] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0122] The computer device 4 includes a memory 4a, a processor 4b, and a network interface 4c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 having components 4a-4c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0123] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0124] The memory 4a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 4a can be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 4a can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 4a can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 4a is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a data association method. In addition, the memory 4a can also be used to temporarily store various types of data that have been output or are to be output.

[0125] In some embodiments, the processor 4b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data association chip. The processor 4b is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 4b is used to execute computer-readable instructions stored in the memory 4a or process data, such as computer-readable instructions for executing the data association method.

[0126] The network interface 4c may include a wireless network interface or a wired network interface. The network interface 4c is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0127] The computer device proposed in this embodiment belongs to the field of financial technology and is applied to financial business data interaction processing scenarios. This application creates a data dimension table in Apache Hudi; initializes dimension data from a target source database into the data dimension table; monitors data changes in the target source database during a target interaction; identifies the dimension data that changes during the target interaction; incrementally updates the changed dimension data into the data dimension table; transmits the data changes in the target source database as message stream data to the Apache Flink real-time computing engine; retrieves and transmits the changed dimension data to the Apache Flink real-time computing engine through incremental query; performs a correlation operation within the Apache Flink real-time computing engine based on the message stream data and the changed dimension data to obtain a mapping relationship between the message stream data and the changed dimension data; and completes real-time data correlation during the target interaction based on this mapping relationship. By leveraging Apache Hudi's support for incremental computation, the amount of data loaded into the dimension table during each real-time correlation calculation is reduced, thereby improving the efficiency of the real-time correlation calculation task. An expiration and elimination strategy for dimension table memory data is designed using the LinkedHashMap linked list, reducing memory pressure on real-time tasks and improving their stability.

[0128] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the data association method as described above.

[0129] The computer-readable storage medium proposed in this embodiment belongs to the field of financial technology and is applied to financial business data interaction processing scenarios. This application creates a data dimension table in Apache Hudi; initializes dimension data from a target source database into the data dimension table; monitors data changes in the target source database during a target interaction; identifies the dimension data that changes during the target interaction; incrementally updates the changed dimension data into the data dimension table; transmits the data changes in the target source database as message stream data to the Apache Flink real-time computing engine; retrieves and transmits the changed dimension data to the Apache Flink real-time computing engine through incremental query; performs a correlation operation within the Apache Flink real-time computing engine based on the message stream data and the changed dimension data to obtain a mapping relationship between the message stream data and the changed dimension data; and completes real-time data correlation during the target interaction based on this mapping relationship. By leveraging Apache Hudi's support for incremental computation, the amount of data loaded into the dimension table during each real-time correlation calculation is reduced, thereby improving the efficiency of the real-time correlation calculation task. An expiration policy for dimension table memory data is designed using the LinkedHashMap linked list, reducing memory pressure on real-time tasks and improving their stability.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0131] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A data association method, characterized in that: The steps include: In response to the dimension table creation instruction sent by the data maintenance end, create a data dimension table in Apache Hudi according to the dimension table creation instruction; In response to the dimension data initialization instruction sent by the data maintenance end, the dimension data in the target source end database is initialized into the data dimension table, and the initialized data dimension table is obtained; According to the preset real-time monitoring component, the data changes occurring in the target source database during the target interaction process are monitored and the data change results are obtained; According to the data change result, identifying the dimension data corresponding to the change in the target interaction process as the target dimension data; Reading the target dimension data from the target source database, and incrementally updating the target dimension data to the initialized data dimension table using data synchronization technology to obtain an incrementally updated data dimension table; Using Apache Kafka message streams, data changes in the target source database are sent as message stream data to the Apache Flink real-time computing engine. Querying the synchronously updated target dimension data in the incrementally updated data dimension table through an incremental query method, and sending the target dimension data to the Apache Flink real-time computing engine. After executing the step of querying the synchronously updated target dimension data in the incrementally updated data dimension table through an incremental query method, the method further includes: Identify the LinkedHashMap linked list corresponding to the initialized data dimension table; Add the target dimension data as a new element of the LinkedHashMap linked list to the head position of the queue of the LinkedHashMap linked list; Performing an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data; The real-time association of data in the target interaction process is completed according to the mapping relationship between the message stream data and the target dimension data.

2. The data association method according to claim 1, wherein: The step of incrementally updating the target dimension data into the initialized data dimension table using data synchronization technology to obtain the incrementally updated data dimension table specifically includes: The target dimension data is incrementally updated to the initialized data dimension table using Sqoop offline synchronization or Flink CDC real-time synchronization to obtain an incrementally updated data dimension table.

3. The data association method according to claim 1, wherein: After executing the step of sending the data changes occurring in the target source database as message stream data to the Apache Flink real-time computing engine through the Apache Kafka message stream, the method further includes: Start the real-time reading task preset in the Apache Flink real-time computing engine; Based on the real-time reading task, the message stream data sent in the Apache Kafka message stream mode is read, and after reading the message stream data, the preset associated computing task in the Apache Flink real-time computing engine is triggered and started according to the preset trigger component.

4. The data association method according to claim 3, wherein: Before executing the step of triggering and starting the preset associated computing task in the Apache Flink real-time computing engine according to the preset trigger component, the method further includes: According to a preset judgment component, it is judged whether the associated calculation task has preloaded the initialized data dimension table; If the associated calculation task has not pre-loaded the initialized data dimension table, the preset loading component is started to load the initialized data dimension table into the memory corresponding to the associated calculation task, and the data in the initialized data dimension table is cached in the memory in the form of a LinkedHashMap linked list.

5. The data association method according to claim 1 or 4, characterized in that: The step of querying the target dimension data synchronously updated in the incrementally updated data dimension table by incremental query specifically includes: By using the incremental query component preset in the Apache Flink real-time computing engine, the initialized data dimension table is used as a reference table, and the changed data content of the incrementally updated data dimension table relative to the reference table is queried through a comparative query method; The data content changed relative to the reference table is used as the target dimension data for synchronous update in the incrementally updated data dimension table.

6. The data association method according to claim 1, wherein: After executing the step of performing an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data, the method further includes: The position information of the target dimension data in the LinkedHashMap linked list is updated from the head position of the queue to the tail position of the queue of the LinkedHashMap linked list.

7. The data association method according to claim 6, characterized in that: After executing the step of updating the position information of the target dimension data in the LinkedHashMap linked list from the head position of the queue to the tail position of the queue of the LinkedHashMap linked list, the method further includes: The data volume calculation component preset in the Apache Flink real-time computing engine is used to calculate the data volume in the LinkedHashMap table corresponding to the current data dimension table, as the memory required for caching the current data dimension table. Use comparison to determine whether the memory required for caching the current data dimension table has reached the preset cleanup threshold; If the memory required for caching the current data dimension table has reached the cleaning threshold, a preset proportion of linked list data is deleted from the queue head position to the queue tail position of the LinkedHashMap linked list.

8. A data association device, characterized in that: The data association device is used to implement the steps of the data association method according to any one of claims 1 to 7, and the data association device includes: A data dimension table creation module is used to create a data dimension table in Apache Hudi in response to a dimension table creation instruction sent by the data maintenance terminal according to the dimension table creation instruction; A data dimension table initialization module is configured to initialize the dimension data in the target source database into the data dimension table in response to a dimension data initialization instruction sent by the data maintenance terminal, and obtain the initialized data dimension table; A real-time monitoring module is used to monitor data changes in the target source database during the target interaction process according to a preset real-time monitoring component and obtain data change results; a target dimension data identification module, configured to identify, based on the data change result, dimension data corresponding to the change in the target interaction process as target dimension data; A data dimension table update module is configured to read the target dimension data from the target source database and incrementally update the target dimension data to the initialized data dimension table using data synchronization technology to obtain an incrementally updated data dimension table; A message stream sending module is used to send data changes occurring in the target source database as message stream data to the Apache Flink real-time computing engine through the Apache Kafka message stream method; A target dimension data sending module is configured to query the target dimension data that is synchronously updated in the incrementally updated data dimension table through an incremental query method, and send the target dimension data to the Apache Flink real-time computing engine. After executing the step of querying the target dimension data that is synchronously updated in the incrementally updated data dimension table through an incremental query method, the method further includes: Identify the LinkedHashMap linked list corresponding to the initialized data dimension table; Add the target dimension data as a new element of the LinkedHashMap linked list to the head position of the queue of the LinkedHashMap linked list; An engine association operation module is used to perform an association operation in the Apache Flink real-time computing engine based on the message stream data and the target dimension data to obtain a mapping relationship between the message stream data and the target dimension data; The real-time data association module is used to complete the real-time data association in the target interaction process according to the mapping relationship between the message stream data and the target dimension data.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data association method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data association method according to any one of claims 1 to 7.

Images