A non-invasive data extraction processing method and system

Abstract

The application provides a non-invasive data extraction processing method and system, belonging to the technical field of data processing, comprising: extracting incremental data of multiple business systems by using an incremental subscription consumption component; obtaining stock historical data of the multiple business systems through JDBC; performing real-time processing on the incremental data and the stock historical data to obtain processed data; storing the processed data, and synchronizing the stored processed data to the multiple business systems for real-time calling by the multiple business systems. The application realizes real-time calculation data storage, historical data backtracking processing and massive data fast query by using non-invasive sensing to obtain incremental and total data of business systems and to process data in real time, and can perform data comprehensive management without modifying the business systems, and has high real-time performance and high efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a non-invasive data extraction processing method and system. BACKGROUND

[0002] At present, in the information technology, a micro-service concept is mostly used to split a complex business system into multiple systems through a business boundary, and each system has a respective database, and the systems interact with each other through network communication to finally complete the full-process logic processing of the business.

[0003] In the above data architecture, a traditional join table method is usually used for data processing, but the join table method cannot be applied to this data architecture, especially cannot realize real-time data calculation processing in the case of massive data.

[0004] Therefore, a new data processing method needs to be proposed to meet the data processing requirements of multiple business systems and high real-time performance. SUMMARY

[0005] The present application provides a non-invasive data extraction processing method and system to solve the defects that the prior art cannot realize high real-time performance and non-perception for the massive real-time data processing requirements of multiple business systems.

[0006] In a first aspect, the present application provides a non-invasive data extraction processing method, comprising:

[0007] extracting incremental data of multiple business systems by using an incremental subscription consumption component;

[0008] obtaining stock historical data of the multiple business systems through JDBC;

[0009] real-time processing the incremental data and the stock historical data to obtain processed data;

[0010] storing the processed data, and synchronizing the stored processed data to the multiple business systems for real-time calling by the multiple business systems.

[0011] According to the non-invasive data extraction processing method provided by the present application, the incremental data of multiple business systems is extracted by using an incremental subscription consumption component, comprising:

[0012] Canal synchronization listening is adopted to obtain binlog data of databases in the multiple business systems;

[0013] The binlog data is sent to a message middleware kafka.

[0014] The application provides a non-invasive data extraction processing method, which comprises the following steps of: acquiring inventory historical data of a plurality of business systems through JDBC, including:

[0015] configuring a business system database based on a preset configuration mode, determining DML and a table corresponding primary key ID of a table name to be synchronized in the database;

[0016] acquiring inventory historical data in a database table sorted through JDBC according to the DML and the table corresponding primary key ID;

[0017] sending the inventory historical data to a message middleware kafka.

[0018] The application provides a non-invasive data extraction processing method, which comprises the following steps of: configuring a business system database based on a preset configuration mode, including:

[0019] configuring a database account, a password, a table and a primary key field to be synchronized by adding, deleting, searching and modifying a page of the business system database.

[0020] The application provides a non-invasive data extraction processing method, which comprises the following steps of: processing increment data and inventory historical data in real time to obtain processed data, including:

[0021] storing the increment data and the inventory historical data into Elasticsearch through kafka message data, creating an index table according to a table name in the business system database to obtain the processed data;

[0022] The index table is used for data verification and query.

[0023] The application provides a non-invasive data extraction processing method, which comprises the following steps of: processing increment data and inventory historical data in real time to obtain processed data, further comprising:

[0024] storing the increment data and the inventory historical data according to a preset business dimension through kafka message data;

[0025] performing data aggregation in any dimension in the preset business dimension, storing other data in Elasticsearch in a json format, and outputting a preset wide table of any dimension;

[0026] aggregating data in any dimension in a plurality of business systems according to business logic, and storing the data in any business field of the preset wide table of any dimension in an array form to obtain the processed data.

[0027] According to the non-invasive data extraction processing method provided by the application, the processed data is stored, the stored processed data is synchronized to a plurality of business systems, and the plurality of business systems are called in real time, comprising:

[0028] The storage database is determined as Elasticsearch;

[0029] The aggregation calculation result of each field data in the processed data of the plurality of business systems is obtained, and the stored original data is called by using the deduplication identifier.

[0030] According to the non-invasive data extraction processing method provided by the application, the stored original data is called by using the deduplication identifier, comprising:

[0031] The primary key ID of the called consumption data is stored in Redis, and a preset validity period is determined;

[0032] The primary key ID in the Redis is compared with the current primary key ID when the data is called within the preset validity period, if it is determined that they are inconsistent, the calling is continued, otherwise the calling is stopped.

[0033] In the second aspect, the application further provides a non-invasive data extraction processing system, comprising:

[0034] The incremental extraction module is used for extracting incremental data of a plurality of business systems by using an incremental subscription consumption component;

[0035] The inventory extraction module is used for obtaining inventory historical data of a plurality of business systems through JDBC;

[0036] The processing module is used for processing the incremental data and the inventory historical data in real time to obtain processed data;

[0037] The data calling module is used for storing the processed data, synchronizing the stored processed data to a plurality of business systems, and calling the plurality of business systems in real time.

[0038] In the third aspect, the application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the non-invasive data extraction processing method as described above.

[0039] In the fourth aspect, the application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the non-invasive data extraction processing method as described above.

[0040] The application provides a non-invasive data extraction processing method and system, which realizes real-time data storage, historical data backtracking processing and massive data fast query by using non-invasive sensing to obtain incremental and full data of business systems and processing the data in real time, and can realize data comprehensive management without modifying the business systems, and has high real-time performance and high efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0042] Figure 1 is one of the flowcharts of the non-invasive data extraction processing method provided by the present application;

[0043] Figure 2 is another flowchart of the non-invasive data extraction processing method provided by the present application;

[0044] Figure 3 is a real-time data storage schematic diagram provided by the present application;

[0045] Figure 4 is a structural schematic diagram of the non-invasive data extraction processing system provided by the present application;

[0046] Figure 5 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions in the present application will be described clearly and completely in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the present application.

[0048] Figure 1 is one of the flowcharts of the non-invasive data extraction processing method provided by the present application, as shown in Figure 1 , comprising:

[0049] Step 100: extracting incremental data of multiple business systems by using an incremental subscription consumption component;

[0050] Step 200: obtaining stock historical data of multiple business systems by using JDBC;

[0051] Step 300: real-time processing of the incremental data and the inventory historical data to obtain processed data;

[0052] Step 400: storing the processed data and synchronizing the stored processed data to a plurality of business systems for real-time calling by the plurality of business systems.

[0053] The embodiment of the application is directed to data processing of a plurality of business systems, first extracts incremental data of a business system through a non-invasive business system, extracts inventory historical data of the business system, then performs real-time processing of the extracted incremental data and inventory historical data, and finally stores the processed data and provides the processed data for real-time calling by the business system.

[0054] Specifically, as shown in the figure, Figure 2 The databases of the business systems are independently set, such as database A, database B, database C, database D, and the like, and are generally MYSQL databases. On one hand, incremental subscription consumption components Canal are used to extract log data, such as binlog data, in the above plurality of databases, and the binlog data is sent to message middleware kafka to complete data subscription of the incremental data. On the other hand, Java Database Connectivity (JDBC) is used to traverse table data of each database to extract inventory historical data, and the inventory historical data is also sent through message middleware kafka. After the data processing is completed, the processed data is searched in a data analysis engine Elasticsearch to realize data storage, and real-time data can be queried through the storage database.

[0055] The application realizes real-time calculation of data storage, historical data backtracking processing, and support for massive data fast query by using non-invasive sensing to obtain incremental and full data of business systems and real-time processing of the data, and data comprehensive management can be performed without modifying the business systems, and the application has high real-time performance and high efficiency.

[0056] On the basis of the above embodiment, the incremental subscription consumption components are used to extract incremental data of a plurality of business systems, including:

[0057] Canal is used to synchronously listen to and obtain binlog data of databases in the plurality of business systems;

[0058] The binlog data is sent to message middleware kafka.

[0059] Specifically, the embodiment of the present application synchronizes binlog data of each business system database through an incremental subscription consumption component Canal, and sends the binlog data to a message middleware kafka.

[0060] Here, Canal is an incremental subscription consumption component of the Alibaba MySQL database binlog, and the supported businesses include: database mirroring, database real-time backup, multi-level indexing, business cache refreshing, and important business messages such as price changes.

[0061] The binlog data is binary log, which contains "events" describing database changes, such as table creation operations or changes to table data. MYSQL has two layers of structure, the first layer is the server layer, which contains connectors, query caches, parsers, optimizers and executors, and the second layer is the storage engine layer, such as: InnoDB, MyISAM, Memory and multiple storage engines. The binlog data is generated in the server layer of MYSQL.

[0062] The message middleware kafka is an open source stream processing platform developed by the Apache Software Foundation, which is written in Scala and Java. Kafka is a high-throughput distributed publish / subscribe message system that can handle all action stream data of consumers in a website.

[0063] On the basis of the above embodiment, the inventory historical data of the plurality of business systems is obtained through JDBC, including:

[0064] The business system database is configured based on a preset configuration method, and the data manipulation language (DML) and the table corresponding primary key ID of the table name to be synchronized in the database are determined;

[0065] According to the DML and the table corresponding primary key ID, the inventory historical data in the database table sorted is obtained through JDBC;

[0066] The inventory historical data is sent to the message middleware kafka.

[0067] Among them, the business system database is configured based on a preset configuration method, including:

[0068] The database account, password, table and primary key field to be synchronized are configured by adding, deleting, finding and modifying the page of the business system database.

[0069] Specifically, the embodiment of the present application configures the account password of the business system database in need of synchronization by a pre-set configuration mode, here being read-only permission, defines the data manipulation language (DML) of the table name in need of synchronization and the primary key ID corresponding to the table, and reads the historical inventory data by sorting by ID through the JDBC operation table and sending the historical inventory data to the message middleware kafka.

[0070] The above configuration mode includes the database table configuration of the database account, password, table and primary key field in need of synchronization, and supports the increase, deletion, search and modification of the configuration data through the management page.

[0071] DML is used for database operation, and is a programming statement for operating objects and data in the database. It is a subset of a database-specific programming language, and is centered on INSERT, UPDATE and DELETE instructions, which respectively represent insertion, update and deletion. It is an instruction that must be used for developing data-centered application programs, and additionally includes the SELECT instruction.

[0072] On the basis of the above embodiment, the incremental data and the inventory historical data are processed in real time to obtain processed data, including:

[0073] The incremental data and the inventory historical data are stored in Elasticsearch through kafka message data, an index table is created according to the table name in the business system database, and the processed data is obtained;

[0074] The index table is used for data verification and query.

[0075] In addition, the incremental data and the inventory historical data are processed in real time to obtain processed data, and the processing further includes:

[0076] The incremental data and the inventory historical data are stored according to a preset business dimension through kafka message data;

[0077] Data is aggregated in any dimension of the preset business dimension, other data is stored in Elasticsearch in json format, and a preset wide table of any dimension is output;

[0078] Data of any dimension in multiple business systems is aggregated according to business logic, and is stored in any business field of the preset wide table of any dimension in array form, to obtain the processed data.

[0079] Specifically, the embodiment of the present application is directed to the real-time processing of the obtained incremental data and stock historical data, and the real-time data processing is divided into two modes:

[0080] The first mode is to store the original data in the search and data analysis engine Elasticsearch by consuming the kafka message data, and to create an index by the table name in the database. The original table supports data verification and general query operation of the business.

[0081] Elasticsearch is a distributed, highly scalable and real-time search and data analysis engine. It can easily make a large amount of data searchable, analyzable and explorable, and fully utilize the horizontal scalability of Elasticsearch to make data more valuable in the production environment. The implementation principle of Elasticsearch mainly includes the following steps: first, the user submits data to the Elasticsearch database, then the word segmentation controller segments the corresponding sentence, stores the weight and segmentation result in the data, and ranks the result according to the weight when the user searches the data, and returns the result to the user.

[0082] The second mode is to store the data aggregated by business dimension by consuming the kafka message data, such as aggregating a data by customer number, and storing other fields on the data in Elasticsearch in json format to form a wide table with customer number as the dimension or a certain business dimension, for example Figure 3 The real-time computing data storage method is shown in the schematic diagram, different business fields A, B and C store data in Elasticsearch in json format, and when consuming the kafka message, the data of the customer number or a certain business dimension (a number that can associate multiple data, such as customer number, ID number, unified serial number and order number) in multiple business systems is aggregated according to the business logic and stored in the form of array in a certain business field of the wide table.

[0083] On the basis of the above embodiment, the processed data is stored, and the stored processed data is synchronized to multiple business systems for real-time calling by the multiple business systems, including:

[0084] The storage database is determined to be Elasticsearch;

[0085] The aggregation calculation result of each field data in the processed data of the multiple business systems is obtained, and the original data is called by using the deduplication identifier.

[0086] The original data is called by using the deduplication identifier, including:

[0087] The primary key ID of the called consumption data is stored in a remote dictionary service Redis, and a preset validity period is determined;

[0088] The primary key ID in the Redis is compared with a current primary key ID when calling data within the preset validity period, if it is determined that the primary key ID is inconsistent, the calling is continued, otherwise the calling is stopped.

[0089] Specifically, the embodiment of the application stores the processed data and provides the data for real-time calling of a business system, and Elasticsearch is selected as a storage database.

[0090] When storing data, if the data in each field is an aggregated calculation result, the aggregated calculation result is stored and the original data is saved, and the original data needs to be distinguished by a de-duplication identifier to avoid inaccurate data caused by repeated consumption.

[0091] It should be noted that the de-duplication identifier is stored in the redis by the data that has been consumed, and an effective period needs to be set, and whether the data has been consumed is queried from the redis each time consumption, so that repeated consumption in a short time can be avoided. For repeated consumption caused by multiple backtracking of data, whether the data has been consumed is judged according to the data business state, so that repeated consumption of calling data is avoided.

[0092] For example, if a business system needs to query the number of behaviors of a certain customer, only the data of the behavior field in the customer dimension wide table needs to be queried through the customer number, and the query result can be obtained by simple aggregation calculation after obtaining the data.

[0093] The non-intrusive data extraction processing system provided by the application will be described below. The non-intrusive data extraction processing system described below can be correspondingly referred to the non-intrusive data extraction processing method described above.

[0094] Figure 4 The non-intrusive data extraction processing system provided by the embodiment of the application is a structural schematic diagram, as shown in Figure 4 The non-intrusive data extraction processing system provided by the embodiment of the application is a structural schematic diagram, as shown in

[0095] The incremental extraction module 41 is used for extracting incremental data of a plurality of business systems by using an incremental subscription consumption component; the inventory extraction module 42 is used for obtaining inventory historical data of the plurality of business systems through JDBC; the processing module 43 is used for processing the incremental data and the inventory historical data in real time to obtain processed data; and the data calling module 44 is used for storing the processed data, synchronizing the stored processed data to the plurality of business systems, and calling the processed data in real time by the plurality of business systems.

[0096] Figure 5 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 5 The electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute a non-invasive data extraction processing method, which includes extracting incremental data of a plurality of business systems using an incremental subscription consumption component; obtaining stock historical data of the plurality of business systems through JDBC; performing real-time processing on the incremental data and the stock historical data to obtain processed data; storing the processed data; and synchronizing the stored processed data to the plurality of business systems for real-time calling by the plurality of business systems.

[0097] In addition, the logical instruction in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0098] On the other hand, the present application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the non-invasive data extraction processing method provided by the above-mentioned methods, which includes extracting incremental data of a plurality of business systems using an incremental subscription consumption component; obtaining stock historical data of the plurality of business systems through JDBC; performing real-time processing on the incremental data and the stock historical data to obtain processed data; storing the processed data; and synchronizing the stored processed data to the plurality of business systems for real-time calling by the plurality of business systems.

[0099] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the non-invasive data extraction processing method provided by the above method, the method comprising: extracting incremental data of a plurality of business systems by using an incremental subscription consumption component; obtaining stock historical data of the plurality of business systems through JDBC; performing real-time processing on the incremental data and the stock historical data to obtain processed data; storing the processed data, and synchronizing the stored processed data to the plurality of business systems for real-time calling by the plurality of business systems.

[0100] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0101] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in terms of contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0102] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A non-invasive data extraction processing method, characterized by, The method comprises the following steps: extracting incremental data of a plurality of business systems by using an incremental subscription consumption component; obtaining stock historical data of the plurality of business systems through a Java Database Connectivity (JDBC); processing the incremental data and the stock historical data in real time to obtain processed data; storing the processed data and synchronizing the stored processed data to the plurality of business systems for real-time calling by the plurality of business systems.

2. The non-invasive data extraction processing method of claim 1, wherein, The method for extracting incremental data of a plurality of business systems by using an incremental subscription consumption component comprises the following steps: synchronously listening to binary log (binlog) data of a database in the plurality of business systems by using an incremental subscription consumption component Canal; sending the binlog data to a message middleware distributed publishing and subscribing message system Kafka.

3. The non-invasive data extraction processing method of claim 1, wherein, The method for obtaining stock historical data of a plurality of business systems through a JDBC comprises the following steps: configuring a business system database based on a preset configuration method, determining a data manipulation language (DML) of a table name to be synchronized and a table corresponding primary key ID in the database; obtaining stock historical data in a database table sorted by the DML and the table corresponding primary key ID through a JDBC; sending the stock historical data to a message middleware Kafka.

4. The non-invasive data extraction processing method of claim 3, wherein, The method for configuring a business system database based on a preset configuration method comprises the following steps: configuring a database account, a password, a table and a primary key field to be synchronized by adding, deleting, searching and modifying a page of the business system database.

5. The non-invasive data extraction processing method of claim 1, wherein, The method for processing the incremental data and the stock historical data in real time to obtain processed data comprises the following steps: storing the incremental data and the stock historical data in a search and data analysis engine (Elasticsearch) through Kafka message data, creating an index table according to a table name in a business system database to obtain the processed data; wherein, the index table is used for data verification and query.

6. The non-invasive data extraction processing method of claim 5, wherein, The method for processing the incremental data and the stock historical data in real time to obtain processed data further comprises the following steps: storing the incremental data and the stock historical data according to a preset business dimension through Kafka message data; performing data aggregation in any dimension of the preset business dimension, storing other fields except the any dimension in the Kafka message data in the Elasticsearch in a json format, outputting a preset wide table of the any dimension; aggregating data of any dimension in the plurality of business systems according to business logic and storing the data in an array form in any business field of the preset wide table of the any dimension to obtain the processed data.

7. The non-invasive data extraction process of claim 1, wherein, The method for storing the processed data and synchronizing the stored processed data to the plurality of business systems for real-time calling by the plurality of business systems comprises the following steps: determining that a storage database is Elasticsearch; obtaining an aggregation calculation result of each field data in the processed data of the plurality of business systems and calling stored original data by using a deduplication identifier.

8. The non-invasive data extraction processing method of claim 7, wherein, The method for calling stored original data by using a deduplication identifier comprises the following steps: storing a primary key ID of consumed data in a remote dictionary service (Redis) and determining a preset validity period. The primary key ID in the Redis is compared with the current primary key ID when the data in the preset effective period is called, if it is determined that the primary key ID is inconsistent with the current primary key ID, the calling is continued, otherwise the calling is stopped.

9. A non-invasive data extraction processing system, characterized by, Comprise: An incremental extraction module configured to extract incremental data of a plurality of business systems by using an incremental subscription consumption component; A stock extraction module configured to acquire stock historical data of the plurality of business systems through JDBC; A processing module configured to process the incremental data and the stock historical data in real time to obtain processed data; A data calling module configured to store the processed data, and synchronize the stored processed data to the plurality of business systems for real-time calling by the plurality of business systems.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the non-invasive data extraction processing method of any one of claims 1 to 8.

Images