Data processing method, device, equipment, storage medium and program product

Abstract

The present application discloses a data processing method, device, equipment, storage medium and program product, which relates to the field of database management technology. The method includes: receiving a transaction query request through a gateway; responding to and processing the transaction query request and obtaining transaction data according to the type of the transaction query request and the transaction processing database; synchronizing the transaction data in each of the transaction processing databases according to a preset synchronization task. The scheme enables multiple nodes to process query requests in parallel through a preset splitting strategy, greatly improving the concurrent processing capability of the system. Based on the transaction processing database, the scheme can not only store massive amounts of data, but also meet the needs of fast query, while ensuring the accuracy and integrity of the data. In addition, through synchronization tasks, the scheme ensures the synchronization and consistency of transaction data, thereby improving the performance and reliability of the entire system.

Description

Technical Field

[0001] The present application relates to the field of database management technology, and in particular to a data processing method, apparatus, device, storage medium and program product. Background Art

[0002] With the increasing vitality of economic activities and the deepening of digital transformation, the amount of transaction data is growing rapidly. In this case, the storage and query capabilities of transaction data have become key factors in system stability and customer satisfaction. In order to meet this challenge, it is necessary to build an efficient massive data storage and query system. At the same time, the channels and scenarios for face-to-face inquiries are also constantly enriched and expanded. This means that it is necessary not only to achieve stable storage of massive transaction data, but also to ensure that the data can support fast queries in multiple channels and scenarios.

[0003] In current database management practices, it is common to store and query operations on the transaction business table of the ORACLE database. In order to maintain the performance of the business table, the final transaction data is regularly migrated to a dedicated history table to reduce the storage and performance pressure of the business table. However, as time goes by, the amount of data in the history table gradually increases, which will again lead to a decrease in query efficiency. Therefore, when the history table also evolves into a large table, a more efficient solution is sought to cope with the continued growth of data volume and query optimization needs. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, device, equipment, storage medium and program product, aiming to solve the technical problem of how to ensure stable storage and efficient query of massive transaction data.

[0005] To achieve the above objectives, the present application proposes a data processing method, which includes:

[0006] Receive transaction query requests through the gateway;

[0007] Respond to and process the transaction query request according to the type of the transaction query request and the transaction processing database, and obtain transaction data;

[0008] According to the preset synchronization task, the transaction data in each of the transaction processing databases are synchronized.

[0009] In one embodiment, the transaction processing database includes one or more of a transaction service database, a transaction query database, and a distributed database. The step of responding to and processing the transaction query request and obtaining transaction data through a preset segmentation strategy and a transaction processing database includes:

[0010] If the transaction query request is a high-frequency query request, calling the transaction service library, and summarizing and querying transaction data from the transaction service table and the history table in the transaction service library;

[0011] If the transaction query request is a medium frequency query request, calling the transaction query library and obtaining the processed transaction data in the transaction query library;

[0012] If the transaction query request is a low-frequency query request, the distributed database is called and historical transaction data is obtained from the distributed database.

[0013] In one embodiment, before the step of responding to and processing the transaction query request and acquiring transaction data according to the type of transaction query request and the transaction processing database, the step further includes:

[0014] According to a preset segmentation strategy, the transaction query request is segmented into the high-frequency query request, the medium-frequency query request and the low-frequency query request.

[0015] In one embodiment, the preset segmentation strategy uses the time interval as the key basis for segmenting the transaction query request. After the step of segmenting the transaction query request into high-frequency query requests, medium-frequency query requests and low-frequency query requests according to the preset segmentation strategy, the step further includes:

[0016] According to user behavior and channel display, the slices of the high-frequency query request, the medium-frequency query request and the low-frequency query request are dynamically adjusted.

[0017] In one embodiment, the method is applied to a query architecture, wherein the transaction business library, the transaction query library and the distributed database constitute a three-layer query architecture, and at least two of the transaction business library, the transaction query library and the distributed database constitute a two-layer query architecture. When a failure is detected in at least one of the transaction business library, the transaction query library or the distributed database, the three-layer query architecture is switched to the two-layer query architecture.

[0018] In one embodiment, the preset synchronization tasks include end-of-day applications and scheduled tasks, and the step of synchronizing the transaction data in each of the transaction processing databases according to the preset synchronization tasks includes:

[0019] querying the transaction service library according to the end-of-day application to obtain transaction data in the transaction service library;

[0020] Integrate the transaction data in the transaction service library into the preset month partitions, and retrieve the transaction data in the preset partitions and insert them into the transaction query library; and / or

[0021] The transaction service library is queried according to the scheduled task to obtain the transaction data in the transaction service library, and the transaction data in the transaction service library is inserted into the distributed database.

[0022] In one embodiment, the step of synchronizing the transaction data in each of the transaction processing databases according to the preset synchronization task further includes:

[0023] Send synchronization tasks to each pre-set partition through the task distribution application, and submit the synchronization tasks in each partition to the asynchronous thread pool;

[0024] When the synchronous writing application monitors the synchronous task in the asynchronous thread pool, the transaction data in each partition is written to the transaction processing database.

[0025] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:

[0026] A receiving module, used for receiving a transaction query request through a gateway;

[0027] A processing module, used to respond to and process the transaction query request according to the type of the transaction query request and the transaction processing database, and obtain transaction data;

[0028] The synchronization module is used to synchronize the transaction data in each of the transaction processing databases according to a preset synchronization task.

[0029] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0030] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0032] The embodiment of the present application proposes a data processing method, device, equipment, storage medium and program product, the method comprising receiving a transaction query request through a gateway; responding to and processing the transaction query request and obtaining transaction data according to the type of the transaction query request and the transaction processing database; synchronizing the transaction data in each of the transaction processing databases according to a preset synchronization task. The scheme enables multiple nodes to process query requests in parallel through a preset splitting strategy, greatly improving the concurrent processing capability of the system. Based on the transaction processing database, the scheme can not only store massive amounts of data, but also meet the needs of fast query, while ensuring the accuracy and integrity of the data. In addition, through synchronization tasks, the scheme ensures the synchronization and consistency of transaction data, thereby improving the performance and reliability of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0035] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0036] Figure 2 A schematic diagram of data synchronization between the transaction query library and the TiDB library provided in Example 1 of the present application;

[0037] Figure 3 A flowchart of the second embodiment of the data processing method of this application is provided;

[0038] Figure 4 A schematic diagram of the data synchronization process provided in Example 2 of the present application;

[0039] Figure 5 A schematic diagram of a three-layer query architecture of a data processing method provided in Example 1 of the present application;

[0040] Figure 6 A schematic diagram of a two-layer query architecture of a data processing method provided in Example 1 of the present application;

[0041] Figure 7 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0042] Figure 8Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0043] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0045] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0046] The main solution of the embodiment of the present application is: receiving transaction query requests through a gateway; responding to and processing the transaction query requests and obtaining transaction data according to the type of transaction query requests and the transaction processing database; synchronizing the transaction data in each transaction processing database according to the preset synchronization tasks.

[0047] With the increasing vitality of economic activities and the deepening of digital transformation, the amount of transaction data has shown a rapid growth trend. In this context, the storage and query capabilities of transaction data have become key factors in system stability and customer satisfaction. In order to meet this challenge, it is necessary to build an efficient massive data storage and query system. At the same time, in the face of the ever-increasing and expanding customer query channels and scenarios, the system must not only achieve stable storage of transaction data, but also ensure that the data can support fast queries in multiple channels and scenarios.

[0048] In current database management practices, it is common to perform storage and query operations on the transaction business table of the ORACLE database. In order to maintain the performance of the business table, the final transaction data is usually migrated to a dedicated history table regularly to reduce the storage and performance pressure of the business table. However, as time goes by, the amount of data in the history table will gradually increase, eventually leading to a decrease in query efficiency. When the history table becomes a large table, traditional solutions can no longer meet the growing data volume and query optimization needs.

[0049] The present application provides a solution implementation scheme, which aims to meet the storage and query challenges of massive transaction data. In comparison with Oracle, this embodiment selects the high-performance distributed database TiDB as a storage solution for massive data. With its seamless horizontal expansion capabilities, TiDB can flexibly cope with the continuous growth of data volume. In the current macro context of Anxinchuang's technological transformation, TiDB's MySQL protocol compatibility greatly simplifies the migration and integration process of existing applications. In addition, TiDB also provides strong consistency data protection, high availability, and automatic failover functions, thereby ensuring data security and system stability, providing a solid data foundation for digital transformation.

[0050] In order to optimize query performance, this embodiment adopts intelligent time interval division and data routing technology to achieve hierarchical query.

[0051] In addition, the flexible switching between the three-layer and two-layer query architectures not only supports the three-layer query architecture, but also can flexibly switch to the two-layer architecture. On the one hand, during the switching process, the full amount of data will be synchronized to the transaction query library and TiDB, which can achieve high security and can be flexibly switched when there is an accident in a certain library. On the other hand, if there is no obvious distinction between medium and low-frequency queries in the future after the transaction data range is obviously extended (for example, there is no obvious boundary between querying medium-frequency data and low-frequency data, and the current customer query demand for financial management is basically the current month's transactions), switching to a two-layer architecture can save resources while eliminating the need for additional development.

[0052] In addition, the cleared data in the transaction business database is synchronized to the transaction query database and TiDB database, with a complete end-of-day synchronization mechanism and verification mechanism to ensure the data is complete and reliable.

[0053] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes a personal computer as an example to illustrate this embodiment and the following embodiments.

[0054] The professional terms involved in the embodiments of this application include:

[0055] Transaction business library: Mainly responsible for processing real-time transaction data, and undertakes the most core business logic and data writing operations. In order to reduce its pressure, non-medium and low-frequency query requests will be directed to other levels of databases. This library only undertakes relatively high-frequency query operations that are closely related to the current transaction.

[0056] Transaction query library: usually used to store historical transaction data, can quickly respond to query requests for historical transaction records, reduce direct access to the transaction business library, and improve the system's response speed and overall performance. Provide medium-frequency query.

[0057] TiDB library: TiDB is a distributed database with good horizontal scalability and high availability, suitable for processing large-scale data query and analysis. It provides low-frequency query or statistical query of non-real-time large data volumes in non-real-time channels.

[0058] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data processing method of the present application.

[0059] In this embodiment, the data processing method includes steps S10 to S30:

[0060] Step S10, receiving a transaction query request through a gateway;

[0061] It should be noted that this embodiment is applied to the query architecture, which includes a three-layer query architecture and a two-layer query architecture. Among them, the three-layer query architecture includes a query aggregation layer, a transaction query cluster, and a transaction processing database, and the transaction processing database includes a transaction business library, a transaction query library, and a TiDB library. The three-layer query optimizes data query efficiency and improves user experience. The two-layer query architecture includes a query aggregation layer, a transaction query cluster, a transaction business library, and a TiDB library or a transaction query library.

[0062] In addition, it should be noted that when it is detected that at least one of the transaction business library, transaction query library, and TiDB library is unavailable or has other failures, the conversion from three layers to two layers can be achieved by slicing the time into two parts. Secondly, as the business develops, the retention time of transaction data is infinitely extended. When analyzing its user behavior, there may not be a clear distinction between medium frequency and low frequency. It can be directly switched to a two-layer architecture, saving resources without investing in additional development.

[0063] Based on the above, the three-tier query architecture and the two-tier query architecture receive transaction query requests from various channels, such as mobile banking, online banking, etc., through the gateway.

[0064] Step S20, responding to and processing the transaction query request according to the type of the transaction query request and the transaction processing database, and acquiring transaction data;

[0065] It should be noted that the transaction processing database includes but is not limited to one or more of the transaction business library, transaction query library and TiDB library. Transaction query requests include but are not limited to high-frequency query requests, medium-frequency query requests and low-frequency query requests.

[0066] It is worth noting that before executing step S20, it is also necessary to identify the type of transaction query request. Therefore, before step S20, step S201 is also included:

[0067] Step S201, according to a preset segmentation strategy, the transaction query request is segmented into the high-frequency query request, the medium-frequency query request and the low-frequency query request.

[0068] It should be noted that this embodiment adopts a preset segmentation strategy as a condition for querying transaction data, wherein the time interval in the query condition is used as a key basis for selecting a data source. Specifically, first, the three-layer query architecture divides the transaction query request into high-frequency query requests, medium-frequency query requests, and low-frequency query requests according to the time interval, and obtains transaction data from different data sources respectively.

[0069] Among them, high-frequency query refers to transaction data in the recent period, such as the last week or month. Medium-frequency query refers to transaction data that has been settled but in the recent period, such as transaction data settled in the last year. Low-frequency query refers to historical transaction data, such as transaction data over one year.

[0070] Specifically, first, the three-layer query architecture determines whether the query request belongs to high-frequency, medium-frequency or low-frequency query based on the time interval in the transaction query request. For example, if the query time is a transaction within the last week, it will be marked as a high-frequency query request; if the query time is a transaction that has been settled within the last year, it will be marked as a medium-frequency query request; if the query time is a transaction that is more than one year old, it will be marked as a low-frequency query request.

[0071] Based on step S201, in a feasible embodiment, step S20 further includes steps S21 to S23:

[0072] Step S21, if the transaction query request is a high-frequency query request, calling the transaction service library, and summarizing and querying transaction data from the transaction service table and the history table in the transaction service library;

[0073] It should be noted that the transaction business library is mainly used to store the latest transaction data and recently settled transaction data. The transaction business library includes but is not limited to the transaction business table and history table.

[0074] Specifically, since the amount of data involved in high-frequency queries is relatively small, the system directly calls the transaction business table and history table in the transaction business library to directly summarize the query results. For example, when querying transaction data within the past week, the transaction data of the last few days is obtained from the transaction business table, and the transaction data of the previous few days is obtained from the history table, and then the combined results are returned to the user.

[0075] Step S22: if the transaction query request is a medium frequency query request, calling the transaction query library and obtaining the processed transaction data in the transaction query library;

[0076] It should be noted that the transaction query library is mainly used to store transaction data that has been settled but is relatively recent.

[0077] Specifically, since medium-frequency queries involve a large amount of data but a low query frequency, the transaction query library can optimize storage and query performance. For example, when querying transaction data that has been cleared within the past year, the corresponding transaction data can be directly obtained from the transaction query library.

[0078] Step S23: If the transaction query request is a low-frequency query request, the TiDB library is called and historical transaction data is obtained from the TiDB library.

[0079] It should be noted that the TiDB library is mainly used to store historical transaction data and supports large-scale data storage and high-concurrency queries.

[0080] Specifically, since low-frequency queries involve a large amount of data but a very low query frequency, the distributed nature of the TiDB library can effectively cope with the storage and query needs of large-scale data. For example, when querying transaction data for more than one year, the corresponding historical transaction data can be directly obtained from the TiDB library.

[0081] In addition, if at least one of the transaction business library, transaction query library, and TiDB library is unavailable or other failures occur, the three-layer query architecture switches to a two-layer query architecture.

[0082] After receiving a transaction query request, the two-layer query architecture uses the time interval in the query condition as the key basis for selecting the data source. Specifically, if the query request is a high-frequency query, the transaction business library is called to obtain the required data from the transaction business table. If the query request is a medium-low query, the TiDB library or the transaction query library is called.

[0083] Through the above steps, a three-layer query architecture is introduced, breaking the restriction that transactions and queries must always be confined to the same database. By distributing data according to time intervals to query the transaction business database, transaction query database, and TiDB database, the business processing pressure of the transaction business database is effectively reduced, allowing the transaction database to focus on transactions and ensuring the stability and reliability of customer transactions. At the same time, the efficiency and accuracy of data queries are also improved. In addition, the three-layer query architecture can be flexibly switched to a two-layer query architecture. On the one hand, it can flexibly switch to a high guarantee for transaction queries. On the other hand, when there is no obvious distinction between medium and low frequencies in the data, it can be flexibly switched to two layers, saving storage resources without investing in additional development and operation. Efficient use of resources and high guarantee of business are achieved.

[0084] Step S30, synchronizing the transaction data in each of the transaction processing databases according to the preset synchronization task.

[0085] It should be noted that the preset synchronization tasks used in this embodiment include end-of-day applications and scheduled tasks. Among them, the end-of-day application refers to the data synchronization or processing tasks performed at the end of the day (or before the start of the next working day). This type of task is mainly used to ensure that all transaction data sent on the day can be accurately recorded and updated to the system. Scheduled tasks refer to tasks that are automatically executed at pre-set time intervals. This type of task is very important for maintaining the consistency and up-to-dateness of data between different systems, especially in business scenarios that require frequent exchange of information.

[0086] It is worth noting that in order to avoid task suspension due to insufficient write program thread resources when multiple tasks are executed concurrently, executing step S30 not only ensures timely update of data, but also effectively reduces the query pressure of the main database and improves the overall performance of the system.

[0087] In a feasible implementation, step S30 may include steps A31 to A33:

[0088] Step A31, querying the transaction service database according to the end-of-day application to obtain transaction data in the transaction service database;

[0089] Step A32, integrating the transaction data in the transaction service library into the pre-set partition, and retrieving the transaction data in the pre-set partition and inserting it into the transaction query library; and / or

[0090] Step A33: query the transaction service library according to the scheduled task, obtain the transaction data in the transaction service library, and insert the transaction data in the transaction service library into the distributed database.

[0091] Please refer to Figure 2 , Figure 2Schematic diagram of data synchronization between the transaction query library and the TiDB library provided for this embodiment. Specifically, the end-of-day application is started at a preset time point, and the end-of-day application is connected to the transaction business library to perform query operations to obtain all transaction data for the day. All required transaction data is extracted from the transaction business library, such as subscriptions, redemptions, and order withdrawals initiated by customers; passive dividends, forced redemptions, and forced enhancements and reductions; value-added appointment purchases and appointment redemptions. The data extracted from the transaction business library is integrated into pre-set partitions. It should be noted that the partitions preset in this embodiment are divided according to the standard of the month. Subsequently, the integrated data is inserted into the corresponding table in the transaction query library to complete the synchronization of the transaction data in the transaction business library to the transaction query library.

[0092] In addition to synchronizing the transaction data in the transaction business library to the transaction query library, this embodiment also provides a method for synchronizing the transaction data in the transaction business library to the TiDB library. Specifically, according to a preset time interval, such as every hour, every day, etc., a scheduled task is started. The scheduled task calls the data synchronization cluster to connect to the transaction business library to query the latest transaction data. The latest transaction data queried is inserted into the TiDB library to complete the synchronization of the transaction data in the transaction business library to the TiDB library.

[0093] Through the above steps, the cleared data in the transaction business database is synchronized to the transaction query database and TiDB database, with a complete end-of-day synchronization mechanism and verification mechanism to ensure the data is complete and reliable.

[0094] Through the above-mentioned embodiment method, the transaction query request is specifically received through the gateway; through the type of transaction query request and the transaction processing database, the transaction query request is responded to and processed to obtain transaction data; according to the preset synchronization task, the transaction data in each of the transaction processing databases is synchronized. This solution enables multiple nodes to process query requests in parallel through a preset splitting strategy, greatly improving the concurrent processing capability of the system. Based on the transaction processing database, this solution can not only store massive amounts of data, but also meet the needs of fast query, while ensuring the accuracy and integrity of the data. In addition, through the synchronization task, the solution ensures the synchronization and consistency of transaction data, thereby improving the performance and reliability of the entire system.

[0095] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S30 also includes steps B31-B32:

[0096] Step B31, issuing synchronization tasks to each partition through a task distribution application, and submitting the synchronization tasks in each partition to an asynchronous thread pool;

[0097] Step B32, when the synchronous writing application monitors the synchronous task in the asynchronous thread pool, the transaction data in each partition is written to the transaction processing database.

[0098] Optionally, first, the system starts the task distribution application according to the preset time and prepares to issue the synchronization task. According to the characteristics of the transaction data, the transaction data is divided into multiple partitions. Each partition can represent a different data set, such as partition by date, partition by transaction type, etc.

[0099] The task distribution application creates a synchronization task for each partition and sends these tasks to the corresponding partition. Each task contains information such as partition identification, task priority, and task deadline. Then, the task distribution application submits the synchronization tasks in each partition to the asynchronous thread pool. The asynchronous thread pool can dynamically manage thread resources and dynamically adjust the number of threads according to the system load to ensure efficient execution of tasks. At the same time, the task distribution application records the status of each task, such as pending, processing, completed, etc., and provides an interface for external query of the synchronization status of each partition.

[0100] After the synchronous writing application is started, it continuously monitors the synchronous task queue in the asynchronous thread pool. When a new synchronous task is detected, the synchronous writing application takes it out of the queue and starts processing the task. The synchronous tasks are processed partition by partition according to the preset order, such as the natural order of the partition identifier, task priority, etc. For each partition, the synchronous writing application connects to the transaction business library to query and extract the transaction data in the partition. The extracted transaction data is then written to the transaction query library and the TiDB library to ensure data consistency and synchronization.

[0101] After the data is written, the synchronous writing application sends a notification to the task distribution application to report the task completion status.

[0102] After receiving the notification, the task distribution application updates the status of the task to "completed" and reflects this change in its internal records.

[0103] The specific data synchronization process is as follows: Figure 4 As shown, Figure 4 The following is a schematic diagram of the data synchronization process.

[0104] like Figure 4 It can be seen that in order to ensure sufficient resources when multiple tasks are executed concurrently and improve the efficiency and stability of data synchronization, this embodiment adopts the following strategies:

[0105] Specifically, first, data synchronization tasks for different tables are divided into different task groups to avoid simultaneous startup of partition synchronization tasks for the same table, which would result in insufficient thread resources. For different partitions of the same table, the execution interval (i.e., sleep duration) of synchronization tasks is adjusted to stagger the task startup time, effectively avoiding resource competition and ensuring that each task has sufficient thread resources for data writing.

[0106] Since the grayscale database has a small amount of data and full synchronization is easy to verify, the full synchronization strategy is adopted. For normal databases, the partition synchronization strategy is adopted to divide the data into multiple small batches for synchronization to reduce the amount of data synchronized at a time and improve synchronization efficiency.

[0107] After each partition synchronization is completed, small batch checks of the partitions and daily total checks are performed to ensure integrity.

[0108] Also, it should be noted that when synchronizing data to the TiDB library for the first time, this process may take a long time because all data needs to be migrated. In order to optimize this process, avoid duplicate development, and effectively control the scope of synchronization, the synchronization time period is set through the initial task configuration, and it is chosen to be performed at night during the off-peak period of business to reduce the impact on daily business. Daily, the system will perform incremental data synchronization, migrating only the data that has been added or changed since the last synchronization. This strategy is particularly important after the initial full synchronization. In order to further improve the efficiency of incremental synchronization, the task scheduling strategy has been optimized. By shortening the time interval for task execution, more frequent data update capture is achieved to ensure the real-time and accuracy of the data.

[0109] Through the above-mentioned implementation method, the incremental synchronization mechanism has reached a state of efficient operation, which can ensure that the daily incremental data is synchronized within 2 hours. This time window fully meets the real-time requirements of the business, while also ensuring the efficiency and stability of data processing. Through this optimization, we not only reduce unnecessary resource consumption, but also significantly improve the efficiency of data synchronization, providing a strong guarantee for business continuity and data accuracy. Through the data synchronization strategy, not only the timely update of data is guaranteed, but also the query pressure of the main database is effectively reduced, and the overall performance of the system is improved.

[0110] For example, to help understand the implementation process of the data processing method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 5 , Figure 5 A three-layer query architecture diagram of a data processing method is provided, specifically:

[0111] The outermost layer of the three-layer query architecture is the query aggregation layer, which receives transaction query requests from various channels through the gateway, such as mobile banking, online banking, etc. When a request arrives, the query aggregation layer forwards the request to the transaction query cluster, which is responsible for processing and responding.

[0112] The transaction query cluster is the core layer of the three-tier query architecture, responsible for processing and responding to query requests. It is connected to the query aggregation layer through API or HTTP / HTTPS protocol, and connected to the transaction processing database through the database connection pool. In the transaction query cluster, the time interval in the query condition is used as the key basis for data source selection. If it is a high-frequency query, the cluster will call the transaction business library to summarize the transaction data required for the query from the transaction business table and the history table. For medium-frequency queries, the cluster will turn to the transaction query library, which backs up the transaction data that has been processed. For low-frequency transaction data queries, the cluster will call the TiDB library, which is specifically used to store and process historical transaction data.

[0113] Currently, high-frequency, medium-frequency, and low-frequency slices are sliced ​​based on the time within one month, one month to one year, and beyond one year. This slice will be dynamically adjusted based on user behavior and channel display. For example, if the future financial management has 20 years or even longer transaction data, the corresponding frequency slice will be adjusted based on the time and channel buried point statistics of user behavior.

[0114] For example, to help understand the implementation process of the data processing method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 6 , Figure 6 A two-layer query architecture diagram of a data processing method is provided, specifically:

[0115] The outermost layer of the two-layer query architecture is the query aggregation layer, which receives transaction query requests from various channels through the gateway, such as mobile banking, online banking, etc. Similarly, the transaction query cluster is the core layer of the two-layer query architecture, responsible for processing and responding to query requests. It is connected to the query aggregation layer through API or HTTP / HTTPS protocol, and connected to the transaction processing database through the database connection pool. In the transaction query cluster, the time interval in the query condition is still used as the key basis for data source selection. Specifically, if the query is a high-frequency current transaction, the cluster will call the transaction business library to obtain the required data from the transaction business table. For stable medium and low-frequency transaction data queries, the cluster will call the TiDB library. This design is to further optimize data storage and query efficiency. In addition, a scheduled task is set. When the data query function of the TiDB library is stable, the task will regularly clean up the historical table data in the transaction business library to release storage space, avoid data redundancy, and ensure the efficient operation of the system.

[0116] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0117] This application also provides a data processing device, please refer to Figure 7 , the data processing device comprises:

[0118] A receiving module 10, configured to receive a transaction query request through a gateway;

[0119] The processing module 20 is used to respond to and process the transaction query request and obtain transaction data according to the type of the transaction query request and the transaction processing database;

[0120] The synchronization module 30 is used to synchronize the transaction data in each of the transaction processing databases according to a preset synchronization task.

[0121] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of how to ensure the stable storage and efficient query of massive transaction data. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0122] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0123] Reference below Figure 8 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Figure 8As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0125] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0126] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of how to ensure the stable storage and efficient query of massive transaction data. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0127] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0129] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0130] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0131] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0132] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data processing device, the data processing device: receives a transaction query request through a gateway; responds to and processes the transaction query request according to the type of query request A and B and the transaction processing database, and obtains transaction data; and synchronizes the transaction data in each of the transaction processing databases according to a preset synchronization task.

[0133] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0136] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of how to ensure stable storage and efficient query of massive transaction data. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be elaborated here.

[0137] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0138] The computer program product provided by this application can solve the technical problem of how to ensure stable storage and efficient query of massive transaction data. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.

[0139] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Receive transaction query requests through the gateway; Respond to and process the transaction query request according to the type of the transaction query request and the transaction processing database, and obtain transaction data; According to the preset synchronization task, the transaction data in each of the transaction processing databases are synchronized.

2. The method according to claim 1, characterized in that The transaction processing database includes one or more of a transaction service library, a transaction query library, and a distributed database. The step of responding to and processing the transaction query request according to the type of the transaction query request and the transaction processing database to obtain transaction data includes: If the transaction query request is a high-frequency query request, calling the transaction service library, and summarizing and querying transaction data from the transaction service table and the history table in the transaction service library; If the transaction query request is a medium frequency query request, calling the transaction query library and obtaining the processed transaction data in the transaction query library; If the transaction query request is a low-frequency query request, the distributed database is called and historical transaction data is obtained from the distributed database.

3. The method according to claim 2, characterized in that Before the step of responding to and processing the transaction query request and obtaining transaction data by using the transaction query request type and the transaction processing database, the step further includes: According to a preset segmentation strategy, the transaction query request is segmented into the high-frequency query request, the medium-frequency query request and the low-frequency query request.

4. The method according to any one of claims 2 to 3, characterized in that The preset segmentation strategy uses the time interval as the key basis for segmenting the transaction query request. After the step of dividing the transaction query request into the high-frequency query request, the medium-frequency query request and the low-frequency query request according to the preset segmentation strategy, the method further includes: According to user behavior and channel display, the slices of the high-frequency query request, the medium-frequency query request and the low-frequency query request are dynamically adjusted.

5. The method according to claim 4, characterized in that The transaction business library, transaction query library and distributed database constitute a three-layer query architecture, and at least two of the transaction business library, transaction query library and distributed database constitute a two-layer query architecture. When it is detected that at least one of the transaction business library, the transaction query library or the distributed database has a fault, the three-layer query architecture is switched to the two-layer query architecture.

6. The method according to claim 3, characterized in that The preset synchronization tasks include end-of-day applications and scheduled tasks. The steps of synchronizing the transaction data in each of the transaction processing databases according to the preset synchronization tasks include: querying the transaction service library according to the end-of-day application to obtain transaction data in the transaction service library; Integrate the transaction data in the transaction service library into a preset partition, and retrieve the transaction data in the preset partition and insert it into the transaction query library; and / or The transaction service library is queried according to the scheduled task to obtain the transaction data in the transaction service library, and the transaction data in the transaction service library is inserted into the distributed database.

7. The method according to claim 6, characterized in that The step of synchronizing the transaction data in each of the transaction processing databases according to the preset synchronization task also includes: Send synchronization tasks to each partition through the task distribution application, and submit the synchronization tasks in each partition to the asynchronous thread pool; When the synchronous writing application monitors the synchronous task in the asynchronous thread pool, the transaction data in each partition is written to the transaction processing database.

8. A data processing device, characterized in that: The device comprises: A receiving module, used for receiving a transaction query request through a gateway; A processing module, used to respond to and process the transaction query request according to the type of the transaction query request and the transaction processing database, and obtain transaction data; The synchronization module is used to synchronize the transaction data in each of the transaction processing databases according to a preset synchronization task.

9. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images