Batch processing-based business data extraction and distribution method

Abstract

The invention relates to a business data extraction and distribution method based on batch processing. The method comprises the following steps: S1, setting configuration of data extraction; s2, scene information is acquired, tasks are produced based on the scene information, the tasks are divided into a plurality of task groups corresponding to different service domains, the service domains correspond to downstream data models, and the tasks are distributed, scheduled and arranged; s3, after data cleaning is carried out on the source data, a spring batch processing framework is adopted to execute tasks, and the task execution comprises the specific steps that the cleaned data is divided into data blocks, and data reading, data processing and data writing are carried out on each data block to achieve data extraction; and S4, recording a starting moment of data extraction in the step S3, an ending moment of data extraction and a state corresponding to the task group, and broadcasting the recorded moments and states to a downstream data model. Compared with the prior art, the method has the advantages of improving data extraction flexibility and the like.

Description

Technical Field

[0001] The present invention relates to the field of business data extraction, and in particular to a business data extraction and distribution method based on batch processing. Background Art

[0002] In the financial system, after the upstream trading system and settlement system generate business data, the downstream business services need to convert the business data into the business data models required by each business system according to the domain. For scenarios with small-scale data volumes, model conversion can be achieved through interfaces. This method has many problems, such as numerous interfaces, complex parameters, and performance degradation when the data scale increases. In addition, you can also manually complete the source database query, model conversion, data import and other processes by writing shell scripts. This is costly and has a low degree of automation, and is not suitable for large-scale and high-frequency data extraction. For scenarios with large-scale data volumes, the industry usually uses ETL tools to extract, clean and convert data. This method is suitable for application scenarios such as data warehouses and data marts. It solves the scenario of large amounts of business data processing complex indicator data between homogeneous or heterogeneous databases. The implementation method is often inflexible and cannot provide customized data models on demand according to the needs of downstream business systems. Summary of the invention

[0003] The purpose of the present invention is to provide a batch-processing-based business data extraction and distribution method in order to improve the flexibility of business data extraction.

[0004] A batch processing-based business data extraction and distribution method, the method comprising the following steps:

[0005] S1. Set the configuration of data extraction;

[0006] S2. Obtain scenario information, generate tasks based on the scenario information, divide the tasks into multiple task groups corresponding to different business types, the business types correspond to downstream data models, and distribute, schedule and arrange the tasks;

[0007] S3. After cleaning the tables that need to be updated in the source data, the spring batch batch processing framework is used to execute the task. The tables that need to be updated in the source data include the table synchronization group classified by business type and other tables that are strongly related to the business. The specific steps of executing the task are: dividing the cleaned data into data blocks, reading data, processing data and writing data for each data block to realize data extraction;

[0008] S4. Record the start time of data extraction, the end time of data extraction and the status corresponding to the task group in S3, and broadcast the recorded time and status to the downstream data model of the corresponding business type.

[0009] Furthermore, in the process of extracting data by reading, processing and writing data for each data block, if an abnormality occurs in the data extraction of a data block, the data block is rolled back and the data is restored to the last backup state.

[0010] Furthermore, the data extraction supports two scenarios: incremental data extraction based on trading days and full data extraction. In the incremental data extraction scenario, data extraction is performed on the incremental data of the table that needs to be updated in the source data. In the full data extraction scenario, data extraction is performed on the full data of the table that needs to be updated in the source data.

[0011] Furthermore, data extraction is implemented based on a sql field, and the sql field is determined based on the configuration of S1.

[0012] Furthermore, the sql field is used to implement reading data and writing data.

[0013] Furthermore, the read data refers to data read from a source database, and the write data refers to data written to a target database.

[0014] Furthermore, the source database is one or more.

[0015] Furthermore, the source database is a homogeneous database or a heterogeneous database.

[0016] Furthermore, the sql field realizes processing data, and the data conversion logic corresponding to the processed data supports customization.

[0017] Furthermore, in the process of task distribution, scheduling and orchestration, multiple parallel tasks are managed through the thread pool, and the thread pool configuration is fine-tuned according to the actual scenario.

[0018] The purpose of the present invention can be achieved by the following technical solutions:

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The present invention can extract and distribute upstream data by domain to the data model required by the downstream business system through batch processing technology based on Springbatch. Both source database reading and target database writing are read in sql configuration mode. Through flexible extraction strategies, lightweight extraction and distribution of customized business data in different dimensions can be provided, which greatly enriches the richness of downstream data models. In addition, multiple data extraction tasks are managed through pooling technology. Multiple parallel data extraction tasks are managed by thread pools, and memory usage is controllable. The thread pool configuration can be fine-tuned according to actual scenarios to optimize task execution efficiency. The present invention supports incremental or full update scenarios based on trading days. It meets the needs of various business scenarios such as full data update, incremental update, historical archiving, etc. The data extraction step of the present invention supports the configuration of a single chunk size. When a single task has a processing exception, it will only affect the chunk-level data, and complete the rollback of the abnormal chunk data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The module flow chart corresponding to the method of the present invention. DETAILED DESCRIPTION

[0022] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0023] A batch-based business data extraction and distribution method, the corresponding module flow chart is as follows Figure 1 As shown, the method comprises the following steps:

[0024] S1. Set the configuration of data extraction;

[0025] S2. Obtain scenario information, generate tasks based on the scenario information, divide the tasks into multiple task groups corresponding to different business types, the business types correspond to downstream data models, and distribute, schedule and arrange the tasks;

[0026] S3. After cleaning the tables that need to be updated in the source data, the spring batch batch processing framework is used to execute the task. The tables that need to be updated in the source data include the table synchronization group classified by business type and other tables that are strongly related to the business. The specific steps of executing the task are: dividing the cleaned data into data blocks, reading data, processing data and writing data for each data block to realize data extraction;

[0027] S4. Record the start time of data extraction, the end time of data extraction and the status corresponding to the task group in S3, and broadcast the recorded time and status to the downstream data model of the corresponding business type.

[0028] In the present invention, Spring Batch is a lightweight batch processing framework that can support batch processing jobs of large amounts of data and can be applied to batch processing services of data.

[0029] Chunk is the smallest unit for processing data in Spring Batch. It usually contains components for reading, processing, and writing data. After the reading and processing are completed, the data is written.

[0030] The thread pool is a form of multi-threaded processing. During the processing, tasks to be executed can be added to the task processing queue, and threads can be created to execute the tasks.

[0031] The present invention includes a configuration management module, a task distribution module, a task execution module, and a message notification module. The task execution module is composed of data cleaning and data extraction. The data extraction processes data in sequence according to the execution units of the reader, processor, and writer. After all data processing chunks are processed, the message notification module is called to broadcast messages to notify the downstream that the application data is ready.

[0032] The configuration management module is used to store and read data extraction configurations, including the task triggering method, triggering time, task validity configuration, and key information such as the source database, target database, metadata query SQL, data cleaning SQL, field mapping, and target table write statements. Among them, the innovation of the above configuration is mainly reflected in the flexible data extraction configuration method. Since both the source database reading and the target database writing are read using the SQL configuration method, updates and maintenance are simple and flexible; customized data conversion logic is provided, providing database field mapping combined with custom implementation logic of SQL fields, and implementing field processing on demand; supports multiple data source configurations, and can support multiple configuration methods for homogeneous and heterogeneous databases to adapt to different data extraction application scenarios.

[0033] The task distribution module is used to distribute, schedule and execute tasks after the tasks are completed according to the scenario trigger. Multiple data extraction tasks are managed through pooling technology. Multiple parallel data extraction tasks are managed through thread pools, and the memory usage is controllable. The thread pool configuration can be fine-tuned according to the actual scenario to optimize the task execution efficiency.

[0034] The task execution module is the core module of data extraction. It adopts the spring batch batch processing framework, including two units: data cleaning and data extraction. After the metadata is cleaned by the customized data cleaning strategy, the data is poured into the data extraction unit, and data extraction is realized through the three steps of reader, processor, and writer. Specifically, data extraction supports two scenarios: incremental data extraction based on trading days and full data extraction, which meet the needs of incremental update and full data update respectively, and are suitable for multiple application scenarios such as real-time business data model conversion and historical data archiving. The data extraction process supports chunk-level data extraction configuration. The data extraction step supports the configuration of a single chunk size. When a single task has a processing exception, it will only affect the chunk-level data, and the abnormal chunk data will be rolled back, which greatly improves the task execution efficiency and isolates the mutual influence between task executions.

[0035] The message notification module is responsible for broadcasting the start and end of task extraction. When the task starts, since the data is not ready, the downstream application can suspend business processing after obtaining the task status to avoid dirty reading of data. After the task is completed, the downstream application learns the data ready status and can resume business execution to achieve upstream and downstream data status broadcast. The main innovation lies in the multi-granularity data extraction batch processing status broadcast. It provides global data extraction start and end status broadcast and merges into task group-level status broadcast solutions by business domain to notify business services of data instability.

[0036] The present invention provides a lightweight data extraction solution based on Spring Batch batch processing, which extracts data from different source databases into multiple target databases. The main innovations include: 1) Flexible data extraction method. Both source database reading and target database writing adopt sql configuration method to read, and update and maintenance are simple and flexible. 2) Customized data conversion logic. Provide database field mapping combined with customized implementation logic of sql field to implement field processing on demand. 3) Manage multiple data extraction tasks through pooling technology. Manage multiple parallel data extraction tasks through thread pool, and the memory usage is controllable. The thread pool configuration can be fine-tuned according to the actual scenario to optimize the task execution efficiency. 4) Support incremental or full update scenarios based on trading days. Meet the requirements of various business scenarios such as full data update, incremental update, historical archiving, etc. 5) Chunk-level data extraction configuration. The data extraction step supports the configuration of a single chunk size. When a single task has a processing exception, it will only affect the chunk-level data, and complete the rollback of the abnormal chunk data. 6) Multi-granularity data extraction batch processing status broadcast. Provides global data extraction start and end status broadcasts and merges into task group-level status broadcasts by business domain to notify business services of data instability periods.

[0037] The present invention uses batch processing technology based on Springbatch to extract and distribute upstream data by domain to the data model required by the downstream business system. Through flexible extraction strategies, it can provide lightweight extraction and distribution of customized business data in different dimensions, greatly enriching the richness of downstream data models.

[0038] In terms of data rollback, the present invention retains data backup before daily data synchronization and regularly cleans up backup data after a certain period of time.

[0039] In the process of distributing, scheduling and arranging tasks in the present invention, strongly related business tables are established as a table synchronization group on the business data.

[0040] When broadcasting messages, use the message middleware to send messages;

[0041] Before the daily data synchronization, a message will be sent to all downstream business services to start the data synchronization task. If the downstream services have business operations at this time, they will be prompted that data synchronization is in progress and write operations are not supported;

[0042] After a group of strongly related table synchronization tasks are synchronized, a message will be sent to inform the downstream service that a certain type of data synchronization task has been completed;

[0043] The downstream service listens to messages indicating that data synchronization is complete and decides on its own whether to enable certain business operations.

[0044] When a data synchronization request is received daily, all tables to be synchronized are detected. If it is found that the tables to be synchronized have other tables that are strongly related to the business, all the data in all the tables involved are updated according to the date.

[0045] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A batch-based business data extraction and distribution method, characterized in that: The method comprises the following steps: S1. Set the configuration of data extraction; S2. Obtain scenario information, generate tasks based on the scenario information, divide the tasks into multiple task groups corresponding to different business types, the business types correspond to downstream data models, and distribute, schedule and arrange the tasks; S3. After cleaning the tables that need to be updated in the source data, the spring batch batch processing framework is used to execute the task. The tables that need to be updated in the source data include the table synchronization group classified by business type and other tables that are strongly related to the business. The specific steps of executing the task are: dividing the cleaned data into data blocks, reading data, processing data and writing data for each data block to realize data extraction; S4. Record the start time of data extraction, the end time of data extraction and the status corresponding to the task group in S3, and broadcast the recorded time and status to the downstream data model of the corresponding business type.

2. A batch processing-based business data extraction and distribution method according to claim 1, characterized in that: In the process of extracting data by reading, processing and writing data for each data block, if an abnormality occurs in the data extraction of a data block, the data block is rolled back and the data is restored to the last backup state.

3. A batch processing-based business data extraction and distribution method according to claim 1, characterized in that: The data extraction supports two scenarios: incremental data extraction based on trading days and full data extraction. In the incremental data extraction scenario, data extraction is performed on the incremental data of the table that needs to be updated in the source data. In the full data extraction scenario, data extraction is performed on the full data of the table that needs to be updated in the source data.

4. A batch processing-based business data extraction and distribution method according to claim 3, characterized in that: Data extraction is implemented based on sql fields, which are determined based on the configuration of S1.

5. A batch processing-based business data extraction and distribution method according to claim 4, characterized in that: The sql field is used to read and write data.

6. A batch processing-based business data extraction and distribution method according to claim 5, characterized in that: The read data refers to data read from the source database, and the write data refers to data written to the target database.

7. A batch processing-based business data extraction and distribution method according to claim 6, characterized in that: The source database is one or more.

8. A batch processing-based business data extraction and distribution method according to claim 7, characterized in that: The source database is a homogeneous database or a heterogeneous database.

9. A batch processing-based business data extraction and distribution method according to claim 8, characterized in that: The sql field implements data processing, and the data conversion logic corresponding to the processed data supports customization.

10. A batch processing-based business data extraction and distribution method according to claim 1, characterized in that: In the process of task distribution, scheduling and orchestration, multiple parallel tasks are managed through the thread pool, and the thread pool configuration is fine-tuned according to the actual scenario.

Images