Data synchronization method and device for high concurrency data under big data application

Abstract

The application relates to the field of information technology, and discloses a data synchronization device for high-concurrency data under big data application. The device is composed of a data acquisition module, a to-be-downloaded data generation module, a data downloading module, a data processing module, a data result returning module, a data judging module, a data processing result warehousing module and a data correlation module. The device has the capability of synchronously returning multiple batches of high-concurrency data, and guarantees the real-time performance of data processing and the data filling rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology. BACKGROUND

[0002] At present, data traceability is becoming more and more common in actual software project development. And in various scenarios, there is a strong real-time and high concurrency requirement for such traceability. For example, in actual work, a certain domain name or a certain Internet protocol address is targeted, and now it is necessary to quickly query whether the domain name has been registered, and the database of the detailed information of the registered domain name (such as domain name owner, domain name registrar information), record information, survival information, or geographic attribution data of the Internet protocol address, specific business information of the record company, etc., to form the final unified result and provide it for subsequent analysis. Generally, a batch of data is about 3000, and it is required to return within 15 minutes. There will be multiple batches of data to be processed in the same time period.

[0003] The traditional way of data traceability mainly includes: through traditional technical means, such as multi-threaded mode, to define customized processes for each scene. This way mainly has the following problems: it can meet the high concurrency of a single batch, but it cannot meet the requirements of data multiple batch high concurrency and data synchronization. Multi-threaded mode can be high-concurrent for a single batch, and multiple batches must rely on queues, but after adding queues, data synchronization cannot be guaranteed, resulting in that the data return time and data filling rate cannot meet the requirements.

[0004] Existing technologies used

[0005] Big data cluster: here refers to HADOOP cluster, which is a basic framework that allows distributed processing of large data sets on computer clusters using a simple programming model, divided into three parts: HDFS distributed storage system, YARN distributed resource management system and MapReduce computing engine.

[0006] Big data computing engine: batch processing uses Spark, and streaming processing uses sparkstreaming; memory computing engine, using sparksql offline memory computing and sparkstreaming real-time computing.

[0007] Big data data warehouse: using hive, a data warehouse based on Hadoop HDFS, with the advantages of mass data storage, horizontal scalability, offline batch processing, solving the problems of traditional relational data warehouse that cannot support mass data storage and poor horizontal scalability.

[0008] Big data data warehouse internal table: internal table managed table, that is, hive managed table, the management of hive internal table includes both logical and syntactic and actual physical meaning, that is, when creating hive internal table, data will exist in the directory of the table, and when deleting internal table, physical data and files are also deleted.

[0009] Big data data warehouse table partition: in order to manage the table reasonably and improve the query efficiency, hive can organize the table into partitions, a partition is actually a directory under the table, a table can be partitioned in multiple dimensions, and the relationship between partitions is the relationship of directory tree.

[0010] Big data data warehouse partition table: when building a table, the data warehouse table with partition is a big data data warehouse partition table, which can assist query, reduce query range, speed up data retrieval speed and manage data according to certain specifications and conditions. According to the number of partition fields, it is divided into different levels of partition.

[0011] Data warehouse overall level division:

[0012] Original library: abbreviated as ODS layer, which collects, converges and retains the original business process data, which is basically consistent with the business system, only for simple integration and unstructured data structuring;

[0013] Resource library: also known as public dimension model layer, its main function is to complete data processing and integration, establish consistent dimensions, build reusable detailed fact tables for analysis and statistics, and summarize public granularity indicators, which are divided as follows:

[0014] Public dimension layer DIM: based on the concept of dimension modeling, establish consistent dimensions for the whole enterprise;

[0015] Detailed granularity fact layer DWD: based on the characteristics of each specific business process, build the most detailed granularity fact table, which can be combined with the data usage characteristics of the enterprise to make appropriate redundancy of some important dimension attribute fields of the detailed fact table, that is, wide table processing;

[0016] Public summary granularity fact layer DWS: based on the modeling drive of the theme object of analysis, based on the index demand of the upper layer application and product, build a public granularity summary index fact table, and use wide table method to physicalize the model;

[0017] Theme library: object-oriented modeling, integrating cross-domain object data, connecting the data of the same object, forming the theme of the object, facilitating deep analysis, mining and application.

[0018] Big data message queue: use Kafka, it is a distributed, partitioned, multi-copy, zookeeper coordinated distributed message system, its biggest feature is that it can process a large amount of data in real time to meet various demand scenarios:

[0019] Producer: the application program responsible for sending messages to the big data message queue;

[0020] Consumer: the client responsible for consuming data from the big data message queue cluster;

[0021] Data consumer group, the data consumer group can have multiple, each data consumer group consumes the same data; divide multiple consumer threads into a group, all members in the group consume the data of a topic together, and the members cannot consume repeatedly.

[0022] Distributed relational database: use TiDB, it is a distributed relational database independently designed and developed by PingCAP company, which is a fusion type distributed database product supporting online transaction processing and online analytical processing at the same time, and has important features such as horizontal expansion or contraction, financial level high availability, real-time HTAP, cloud native distributed database, compatibility with MySQL5.7 protocol and MySQL ecology.

[0023] Big data processing task: batch processing task, refers to the task of using big data batch processing computing engine to operate data, and batch processing input is the data collected in a period of time; streaming processing task, refers to the task of using big data batch processing computing engine to operate data, and streaming processing is to receive and process a series of continuous and changing data in real time. SUMMARY

[0024] In view of the shortcomings of the prior art, the data synchronization device for high-concurrency data under big data application provided by the application is composed of a data acquisition module, a to-be-issued data generation module, a data issuing module, a data processing module, a data result returning module, a data judgment module, a data processing result storage module and a data correlation module;

[0025] The data acquisition module receives the task number transmitted from the outside, filters the data to be processed this time from the distributed relational database according to the task number, the data to be processed this time is the data to be traced issued by the external system, is collected into the big data cluster environment, and is stored into the big data data warehouse original library layer, the data table is a partition table, and the partition field corresponds to the task number.

[0026] The to-be-issued data generation module receives a task number transmitted from outside, generates a batch number, takes the current time plus four random digits as the batch number, filters the original library layer data by the task number, and counts the current data volume as the batch total volume, and writes the batch total volume into a scheduling process control parameter table of a distributed relational database. The scheduling process control parameter table at least includes a task number, a batch number, a batch total volume, and a batch state. The batch state includes batch generation, initialization, issued, processed, and warehoused. The batch state of the newly generated batch data is set to batch generation;

[0027] The to-be-issued data generation module divides the original library layer data into fact tables and dimension tables by using a big data computing engine, and marks the batch number on the tables and stores the tables in a big data data warehouse. The result is a fact table as a first-level partition table and a dimension table as a first-level partition table, and the partition field is the batch number. The fact table is a detailed granularity fact layer, and the dimension table is a public dimension layer. After the processing is completed, the scheduling process control parameter table of the distributed relational database is updated, and the batch state is set to initialization.

[0028] The data issuance module receives the batch data transmitted from the to-be-issued data generation module, filters the detailed granularity fact layer in the data warehouse by the batch number as the partition field, then traverses the filtered data one by one, and splices the fields by using a special symbol. The special symbol is a computer special symbol selected as a splicing symbol. A producer is created, the detailed granularity fact layer data after the processing is traversed one by one, and is sent to a to-be-processed topic of a big data message queue. After the processing of the batch is completed, the scheduling process control parameter table is updated, and the batch state is set to issued.

[0029] The data processing module reads data of a to-be-processed topic of a big data message queue, performs logical processing, and writes the processing result data into a specific topic of the big data message queue; a plurality of logical processing sub-modules are set according to different business logics, when the business logic corresponding to the logical processing sub-module is domain name query, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; processing logic is performed, the data is parsed piece by piece, a specific field corresponding to the domain name is cut by a special symbol, data processing is performed, a processing result return value is encapsulated, and the processing result return value is spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the processing result data specific topic of the big data message queue; when the business logic corresponding to the logical processing sub-module is to call the record interface, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; processing logic is performed, the data is parsed piece by piece, a specific field corresponding to the record interface is cut by a special symbol, data processing is performed, a processing result return value is encapsulated, and the processing result return value is spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the processing result data specific topic of the big data message queue; when the business logic corresponding to the logical processing sub-module is to query whether the domain name has been registered, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; processing logic is performed, the data is parsed piece by piece, a specific field corresponding to the registered domain name is cut by a special symbol, data processing is performed, a processing result return value is encapsulated, and the processing result return value is spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the processing result data specific topic of the big data message queue; when the business logic corresponding to the logical processing sub-module is to query domain name detailed information, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; processing logic is performed, the data is parsed piece by piece, a specific field corresponding to the domain name detailed information is cut by a special symbol, data processing is performed, a processing result return value is encapsulated, and the processing result return value is spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the processing result data specific topic of the big data message queue;

[0030] The data result return module reads the processing result data specific topic of the big data message queue and writes into the big data distributed storage system;

[0031] The data judgment module reads the processing result data of the current batch of big data distributed storage system within the current time range, the current batch of data to be issued, compares according to the priority of the module, judges whether the current batch of data is processed, and under the premise that the current batch of data is processed, the next process is performed; under the premise that the current batch of data is not processed, it is judged whether the current judgment module processing time exceeds the pre-defined timeout, and if it exceeds the pre-defined timeout, the next process is performed; if it does not exceed the pre-defined timeout, it is put to sleep for a period of time, and then the next cycle is performed to read the current batch of data processing result and the current batch of data to be issued within the current time range; the current time range includes: the current time before 1 hour plus the current time current hour plus the current time after 1 hour;

[0032] The data processing result storage module reads the processing result data of the current batch of big data distributed storage system within the current time range, merges and removes duplicates, and stores it into the big data data warehouse-data processing result table.

[0033] The data correlation module reads the data processing result data in the current batch of data warehouse, the current batch of fact data table, and the current batch of dimension data table, takes the current batch of fact data table as the main table, left-associates the current batch of dimension data table and the current batch of processing result data, and forms the final result.

[0034] Beneficial effects

[0035] It has the ability of multiple batch high-concurrency data and data synchronous return, which guarantees the real-time performance of data processing and data filling rate. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 It is the system structure diagram of the application. DETAILED DESCRIPTION

[0037] Referring to Figure 1 The high-concurrency data synchronization device provided by the application under the big data application comprises a data collection module 1, a data to be issued generation module 2, a data issuing module 3, a data processing module 4, a data result returning module 5, a data judgment module 6, a data processing result storage module 7 and a data correlation module 8.

[0038] The data collection module 1 receives the task number transmitted from the outside, filters the data to be processed this time from the distributed relational database according to the task number, collects the data to be processed this time, that is, the data to be traced from the external system, into the big data cluster environment, and stores it into the big data data warehouse original library layer; the data table is a partition table, and the partition field corresponds to the task number.

[0039] The to-be-delivered data generation module 2 receives the task number delivered from the outside, generates a batch number, takes the current time plus four random digits as the batch number, filters the original library layer data by the task number, and counts the current data volume as the batch total volume, and writes into the scheduling process control parameter table of the distributed relational database. The scheduling process control parameter table at least contains the task number, the batch number, the batch total volume, the batch state, and the batch state contains: batch generation, initialization, has been delivered, has been processed, has been warehoused, and the batch state of the newly generated batch data is set to batch generation;

[0040] The to-be-delivered data generation module 2 uses a big data computing engine to divide the original library layer data into fact tables and dimension tables, and mark them with batch numbers, and then store them in a big data data warehouse. The result is a fact table as a first-level partition table and a dimension table as a first-level partition table, and the partition field is the batch number. The fact table is a detailed granularity fact layer, and the dimension table is a public dimension layer. After processing, the scheduling process control parameter table of the distributed relational database is updated, and the batch state is set to initialization.

[0041] The data delivery module 3 receives the batch data delivered from the to-be-delivered data generation module, filters the detailed granularity fact layer in the data warehouse by the batch number as the partition field, then traverses the filtered data one by one, and splices the fields with special symbols. The special symbol is a computer special symbol selected as a splicing symbol. A producer is created to traverse the processed detailed granularity fact layer data one by one, and send them to the to-be-processed topic of the big data message queue. After the completion of the batch processing, the scheduling process control parameter table is updated, and the batch state is set to has been delivered.

[0042] The data processing module 4 reads the data of the to-be-processed topic of the big data message queue, performs logical processing, and writes the processed result data to a specific topic of the big data message queue; a plurality of logical processing sub-modules are set according to different business logics, when the business logic corresponding to the logical processing sub-module is domain name query, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; the data is parsed and processed in a piece-by-piece manner, the specific field corresponding to the domain name is cut by a special symbol, the processing result return value is encapsulated and spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the specific topic of the big data message queue; when the business logic corresponding to the logical processing sub-module is to call the record interface, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; the data is parsed and processed in a piece-by-piece manner, the specific field corresponding to the record interface is cut by a special symbol, the processing result return value is encapsulated and spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the specific topic of the big data message queue; when the business logic corresponding to the logical processing sub-module is to query whether the domain name has been registered, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; the data is parsed and processed in a piece-by-piece manner, the specific field corresponding to the registered domain name is cut by a special symbol, the processing result return value is encapsulated and spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the specific topic of the big data message queue; when the business logic corresponding to the logical processing sub-module is to query the domain name detailed information, a consumer is created, and data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; the data is parsed and processed in a piece-by-piece manner, the specific field corresponding to the domain name detailed information is cut by a special symbol, the processing result return value is encapsulated and spliced by the special symbol; a producer is created, and the spliced processing result return value is sent to the specific topic of the big data message queue.

[0043] The data result return module 5 reads the specific topic of the processed result data of the big data message queue and writes to the big data distributed storage system.

[0044] The data judgment module 6 reads the processing result data of the current batch of big data distributed storage system within the current time range, the current batch of data to be issued, compares according to the priority of the module, judges whether the current batch of data is processed, and under the premise that the current batch of data is processed, the next process is carried out; under the premise that the current batch of data is not processed, it is judged whether the current judgment module processing time exceeds the pre-defined timeout, and if it exceeds the pre-defined timeout, the next process is carried out; if it does not exceed the pre-defined timeout, it is put to sleep for a period of time, and then the next cycle is carried out, and the current batch of data processing result and the current batch of data to be issued are compared again; the current time range includes: the current time before 1 hour plus the current time current hour plus the current time after 1 hour;

[0045] The data processing result storage module 7 reads the processing result data of the current batch of big data distributed storage system within the current time range, merges and removes duplicates, and stores it in the big data data warehouse-data processing result table;

[0046] The data correlation module 8 reads the data processing result data in the current batch of data warehouse, the current batch of fact data table, and the current batch of dimension data table, takes the current batch of fact data table as the main table, left-joins the current batch of dimension data table and the current batch of processing result data, and forms the final result.

Claims

1. A data synchronization device for high-concurrency data under big data application, characterized in that The data acquisition module receives a task number transmitted from outside, filters data to be processed this time from a distributed relational database according to the task number, the data to be processed this time is data to be traced from the external system, is collected into a big data cluster environment, and is stored into a big data data warehouse original library layer, a data table is a partition table, and a partition field corresponds to a task number; The to-be-transmitted data generation module receives a task number transmitted from outside, generates a batch number, takes the current time plus four random digits as the batch number, filters the original library layer data according to the task number, and counts the current data quantity as a batch total quantity, and writes into a scheduling process control parameter table of the distributed relational database, the scheduling process control parameter table at least includes a task number, a batch number, a batch total quantity, and a batch state, the batch state includes: batch generation, initialization, has been transmitted, has been processed, and has been stored, and the batch state of the newly generated batch data is set to batch generation; The to-be-transmitted data generation module uses a big data computing engine, divides the original library layer data into fact tables and dimension tables, all of which are marked with batch numbers, and are stored into a big data data warehouse, the result is a fact table as a first-level partition table and a dimension table as a first-level partition table, and the partition field is the batch number; the fact table is a detailed granularity fact layer, and the dimension table is a public dimension layer; after processing is completed, the scheduling process control parameter table of the distributed relational database is updated, and the batch state is set to initialization; The data transmission module receives batch data transmitted from the to-be-transmitted data generation module, filters the detailed granularity fact layer in the data warehouse according to the batch number as a partition field, then traverses the filtered data one by one, and splices fields by using a special symbol; a producer is created, the detailed granularity fact layer data after processing is traversed one by one, is sent to a to-be-processed topic of a big data message queue, after this batch processing is completed, the scheduling process control parameter table is updated, and the batch state is set to has been transmitted; The data processing module reads data of the to-be-processed topic of the big data message queue, writes into a processing result data specific topic of the big data message queue after logical processing, sets a plurality of logical processing submodules according to different business logics, when the business logic corresponding to the logical processing submodule is domain name query, a consumer is created, data is read from the to-be-processed topic of the big data message queue according to the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups, the data is processed one by one, a specific field corresponding to the domain name is cut by using a special symbol, a processing result return value is encapsulated, and the processing result return value is spliced by using the special symbol; ​ The processing result return value after splicing is sent to the processing result data specific topic of the big data message queue by creating a producer; when the business logic corresponding to the logic processing submodule is calling the filing interface, a consumer is created, and data is read from the to-be-processed topic of the big data message queue under the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; the processing logic parses the data one by one, cuts the specific fields corresponding to the filing interface with special symbols, performs data processing, encapsulates the processing result return value, and splices it with special symbols; The processing result return value after splicing is sent to the processing result data specific topic of the big data message queue by creating a producer; when the business logic corresponding to the logic processing submodule is calling the filing interface, a consumer is created, and data is read from the to-be-processed topic of the big data message queue under the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; the processing logic parses the data one by one, cuts the specific fields corresponding to the filing interface with special symbols, performs data processing, encapsulates the processing result return value, and splices it with special symbols; The processing result return value after splicing is sent to the processing result data specific topic of the big data message queue by creating a producer; when the business logic corresponding to the logic processing submodule is calling the filing interface, a consumer is created, and data is read from the to-be-processed topic of the big data message queue under the principle that the same business is in the same data consumer group and different businesses are in different data consumer groups; the processing logic parses the data one by one, cuts the specific fields corresponding to the filing interface with special symbols, performs data processing, encapsulates the processing result return value, and splices it with special symbols; The data result return module reads the processing result data specific topic of the big data message queue and writes it to the big data distributed storage system; The data judgment module reads the processing result data of the current batch of the big data distributed storage system within the current time range, compares the to-be-issued data of the current batch according to the priority of the module, judges whether the current batch of data is processed, and performs the next process under the premise that the current batch of data is processed; under the premise that the current batch of data is not processed, it is judged whether the current judgment module processing time exceeds the pre-defined timeout time, and if it exceeds the pre-defined timeout time, the next process is performed; if it does not exceed the pre-defined timeout time, it sleeps for a period of time and then performs the next cycle to read the current batch of data processing result and the to-be-issued data within the current time range for comparison; the current time range includes: the current time plus the current hour plus the next hour; The data processing result storage module reads the processing result data of the current batch of the big data distributed storage system within the current time range, performs merging and deduplication, and stores it in the big data data warehouse-data processing result table. The data association module reads the data processing result data in the batch data warehouse, the batch fact data table and the batch dimension data table, takes the batch fact data table as the main table, left associates the batch dimension data table and the batch processing result data, and forms the final result.

Images