Data processing method and device based on Canal

By acquiring and processing Canal messages and using target listeners to perform data statistics, the problem of low reliability of data statistics methods in the prior art is solved, and efficient and reliable data statistics and redundant data reduction are achieved.

CN111913853BActive Publication Date: 2024-12-06CHINA XIAOYUN (BEIJING) INTERNET OF THINGS TECH RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010820448.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-14
Publication Date
2024-12-06
Estimated Expiration
2040-08-14

AI Technical Summary

Technical Problem

The methods used for data statistics in the prior art are relatively low in reliability, especially when migrating data from MySQL database tables to Hadoop platform for statistics, resulting in redundant data storage and high database load problems.

Method used

By obtaining the Canal message generated by the incremental log Canal component, the target listener is determined based on its characteristic information, and the listener is used to process the message, obtain and process the target data, and send it to a predetermined statistical platform to achieve the reliability of data statistics to improve the reliability of data statistics. .

Benefits of technology

It improves the reliability and efficiency of data statistics, reduces redundant data storage and database load, is suitable for customized statistical needs for different businesses, and reduces the pressure on the Hadoop platform.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111913853B_ABST
    Figure CN111913853B_ABST
Patent Text Reader

Abstract

The present invention discloses a data processing method and device based on Canal. The method comprises: obtaining a Canal message generated by an incremental log Canal component; determining a target listener based on characteristic information of the Canal message; and processing the Canal message using the target listener to obtain target data. The present invention solves the technical problem that the data statistics method used for data statistics in the related art has low reliability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a Canal-based data processing method and device. Background Art

[0002] In the era of big data statistics, in most cases, it is necessary to count the total number and change amount of a business record. Usually, people will migrate all the contents in the MySQL database table to Hadoop and then perform statistics. This increases the statistical work of Hadoop, causing the original data file to contain a large amount of data that is irrelevant to the statistical business.

[0003] In response to the above problems, some existing technologies use a massive platform as the source database, a Hadoop big data platform as the target database, and an automated incremental data import method developed in combination with Java to achieve the purpose of importing incremental data from the source database into the target database. Although this solution requires the addition of an intermediate layer to process the data before importing it into the Hadoop platform, it still generally imports all incremental data into the Hadoop platform and performs some basic legal data judgments, rather than generating incremental data directly related to the statistical business, which will result in the storage of content unrelated to statistics into Hadoop. In addition, in a small number of situations where the Hadoop platform does not need to be used for statistics, this solution cannot be used.

[0004] On the other hand, there will also be a Canal-based cache update method including: S1. After the database changes the data field, an update diary is generated, and the update diary includes the changed value of the data field; S2. Use Canal to read the database to obtain the update diary; S3. Canal parses the update diary to obtain the changed value, and sends the changed value to the message queue; S4. The message queue receives and sends the changed value to the message consumer, the message consumer receives and sends the changed value to the cache, and the cache updates the changed value. Although the solution mentions that Canal can monitor data changes in database tables and can store the changed data in the Redis cache, it does not involve statistics-related business, but only refreshes the cache to keep the data in the Redis cache consistent with the database. In addition, this solution will have the defect of excessive load on the database due to continuous queries to the database.

[0005] With respect to the problem of low reliability of the data statistics method used for data statistics in the above-mentioned related technologies, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiment of the present invention provides a Canal-based data processing method and device to at least solve the technical problem of low reliability of a data statistics method used for performing data statistics in the related art.

[0007] According to one aspect of an embodiment of the present invention, a Canal-based data processing method is provided, including: obtaining a Canal message generated by an incremental log Canal component; determining a target listener based on characteristic information of the Canal message; and processing the Canal message using the target listener to obtain target data.

[0008] Optionally, obtaining the Canal message generated by the incremental log Canal component includes: determining a data monitoring relationship between the Canal component and a data storage medium; using the Canal component to perform data monitoring on the data storage medium to obtain data change information of the data storage medium; and sending the data change message to a Kafka message queue to obtain the Canal message.

[0009] Optionally, before determining the target listener based on the characteristic information of the Canal message, the Canal-based data processing method further includes: obtaining a service processing request message; and allocating a listener to the service corresponding to the service processing request message based on the service attributes of the service processing request message.

[0010] Optionally, after allocating a listener to the business corresponding to the business processing request message based on the business attributes of the business processing request message, the Canal-based data processing method also includes: generating a data processing strategy according to the business corresponding to each listener, wherein the data processing strategy is multiple; and storing the data processing strategy in the corresponding listener.

[0011] Optionally, the Canal message is processed using the target listener to obtain the target data, including: determining the target business corresponding to the data storage medium monitored by the Canal component; obtaining the business requirements of the target business; selecting a target data processing strategy from multiple data processing strategies stored in the target listener based on the business requirements; and processing the Canal message using the target data processing strategy to obtain the target data.

[0012] Optionally, after the target listener is used to process the Canal message to obtain target data, the Canal-based data processing method further includes: sending the target data to a predetermined statistical platform.

[0013] According to another aspect of an embodiment of the present invention, a Canal-based data processing device is also provided, including: an acquisition unit, used to acquire the Canal message generated by the incremental log Canal component; a determination unit, used to determine the target listener based on the characteristic information of the Canal message; and a processing unit, used to process the Canal message using the target listener to obtain target data.

[0014] Optionally, the acquisition unit includes: a first determination module, used to determine the data monitoring relationship between the Canal component and the data storage medium; a monitoring module, used to use the Canal component to monitor the data storage medium to obtain data change information of the data storage medium; and a sending module, used to send the data change message to the Kafka message queue to obtain the Canal message.

[0015] Optionally, the Canal-based data processing device also includes: the acquisition unit, used to acquire the business processing request message before determining the target listener based on the characteristic information of the Canal message; and the allocation unit, used to allocate a listener to the business corresponding to the business processing request message based on the business attributes of the business processing request message.

[0016] Optionally, the Canal-based data processing device also includes: a generation unit, which is used to generate a data processing strategy according to the business corresponding to each listener after allocating a listener to the business corresponding to the business processing request message based on the business attributes of the business processing request message, wherein there are multiple data processing strategies; and a storage unit, which is used to store the data processing strategy in the corresponding listener.

[0017] Optionally, the processing unit includes: a second determination module, used to determine the target business corresponding to the data storage medium monitored by the Canal component; an acquisition module, used to obtain the business requirements of the target business; a selection module, used to select a target data processing strategy from multiple data processing strategies stored in the target listener based on the business requirements; and a processing module, used to process the Canal message using the target data processing strategy to obtain the target data.

[0018] Optionally, the Canal-based data processing device further includes: a sending unit, configured to send the target data to a predetermined statistical platform after the target listener is used to process the Canal message to obtain the target data.

[0019] According to another aspect of an embodiment of the present invention, a Canal-based data processing system is provided. The Canal-based data processing system uses any one of the Canal-based data processing methods described above.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the computer storage medium is located is controlled to execute any one of the Canal-based data processing methods described above.

[0021] According to another aspect of an embodiment of the present invention, a processor is provided, wherein the processor is used to run a computer program, wherein the computer program executes any one of the above-mentioned Canal-based data processing methods when running.

[0022] In the embodiment of the present invention, the Canal message generated by the incremental log Canal component is obtained; the target listener is determined based on the characteristic information of the Canal message; the Canal message is processed by the target listener to obtain the target data. The Canal-based data processing method provided by the embodiment of the present invention realizes the purpose of using the Canal component to monitor the storage medium storing business data so as to determine the target listener to process the Canal message after obtaining the Canal message to obtain the required data, thereby achieving the technical effect of improving the reliability of data statistics of the tire, and further solving the technical problem of low reliability of the data statistics method used for data statistics in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 is a flow chart of a Canal-based data processing method according to an embodiment of the present invention;

[0025] Figure 2 is a flow chart of an optional Canal-based data processing method according to an embodiment of the present invention;

[0026] Figure 3 is a flowchart of processing Canal messages internally by a listener according to an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of a Canal-based data processing method according to an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a Canal-based data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Example 1

[0032] According to an embodiment of the present invention, a method embodiment of a Canal-based data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 1 is a flow chart of a data processing method based on Canal according to an embodiment of the present invention. Figure 1 As shown, the Canal-based data processing method includes the following steps:

[0034] Step S102, obtaining the Canal message generated by the incremental log Canal component.

[0035] Optionally, Canal is an incremental log component based on binary logs. Its main purpose is to provide incremental data subscription and consumption based on incremental log parsing of MySQL database.

[0036] In an optional embodiment, obtaining the Canal message generated by the incremental log Canal component includes: determining the data monitoring relationship between the Canal component and the data storage medium; using the Canal component to monitor the data storage medium to obtain data change information of the data storage medium; sending the data change message to the Kafka message queue to obtain the Canal message.

[0037] For example, for a certain business, if you want to count its data changes, you can build a data monitoring relationship between the Canal component and the data storage medium corresponding to the business, where the data storage medium here can be a MySQL database; then, use the Canal component to monitor the data storage medium to obtain changes in the data stored in the data storage medium, and store the data change status in the Kafka message queue to obtain the Canal message.

[0038] Step S104: determining a target listener based on the characteristic information of the Canal message.

[0039] Optionally, the characteristic information here may be the name of the data storage medium and the table name.

[0040] It should be noted that in order to obtain data change status information more accurately and to count business data, the legitimacy of the Canal message can be judged after it is obtained, that is, whether the Canal message is the message that needs to be processed currently, that is, before processing the data, the data can be filtered to improve data processing efficiency, and when the Canal message is judged to be an illegal message, it is discarded. If it is a message that needs to be processed currently, the Canal message is encapsulated into a spring event and broadcasted.

[0041] Step S106: Use the target listener to process the Canal message to obtain target data.

[0042] As can be seen from the above, in an embodiment of the present invention, the Canal message generated by the incremental log Canal component is obtained; the target listener is determined based on the characteristic information of the Canal message; the Canal message is processed by the target listener to obtain the target data, thereby realizing the use of the Canal component to monitor the storage medium storing business data so as to determine the target listener to process the Canal message after obtaining the Canal message to obtain the required data, thereby achieving the technical effect of improving the reliability of tire data statistics.

[0043] Therefore, the Canal-based data processing method provided by the embodiment of the present invention solves the technical problem of low reliability of the data statistics method used for data statistics in the related art.

[0044] Since the customized statistical service platform in the embodiment of the present invention can be targeted at different businesses, that is, it can handle data changes of different businesses; therefore, in the embodiment of the present invention, for different business data, it is also necessary to pre-specify a listener for it.

[0045] Therefore, in an optional embodiment, before determining the target listener based on the characteristic information of the Canal message, the Canal-based data processing method may also include: obtaining a business processing request message; and assigning a listener to the business corresponding to the business processing request message based on the business attributes of the business processing request message.

[0046] It should be noted that the business monitored by each listener may have different types of data, or data with different attributes may need to be counted based on business requirements. Therefore, after allocating a listener to the business corresponding to the business processing request message based on the business attributes of the business processing request message, the Canal-based data processing method also includes: generating a data processing strategy according to the business corresponding to each listener, wherein there are multiple data processing strategies; and storing the data processing strategy in the corresponding listener.

[0047] That is, in the embodiment of the present invention, different listeners may be specified for different services; and different data processing strategies may be formulated for a single service to perform data processing in a targeted manner.

[0048] Figure 2 is a flow chart of an optional Canal-based data processing method according to an embodiment of the present invention, such as Figure 2 As shown, assuming that there are incremental data of three tables A, B and C that need to be processed, the processing of these three tables corresponds to the three listeners A, B and C; the corresponding data processing method is as follows: after obtaining the Canal message, the legality of the Canal message is judged. If the judgment result is illegal, the Canal message is directly discarded; otherwise, the corresponding listener is selected by broadcasting the Canal message, that is, which listener is used to process it; after determining the listener, the corresponding listener can be used for processing.

[0049] In the above embodiment, when the data storage medium is a database, when the Canal message enters this customized statistical service, the service will process the message based on Spring's event listening mechanism to improve the concurrency capability of the service. First, the service will obtain the database name and table name in the Canal message to determine whether the current message is a message that needs to be processed, that is, whether the current message is legal. If it is illegal, it will be discarded. If the current message is a message that needs to be processed, the message will be encapsulated into a Spring event and broadcast. Different table names will correspond to different listeners. When the listener receives the broadcast message, it will judge according to its corresponding table name to confirm whether the current Canal message is a message that needs to be processed by this listener. Suppose we have incremental data in tables A, B, and C that need to be processed. The processing of these three tables corresponds to the three listeners A, B, and C. The concurrent processing flow is as follows Figure 2 shown.

[0050] In an optional embodiment, the Canal message is processed using a target listener to obtain target data, including: determining a target business corresponding to a data storage medium monitored by the Canal component; obtaining business requirements of the target business; selecting a target data processing strategy from a plurality of data processing strategies stored in the target listener based on the business requirements; and processing the Canal message using the target data processing strategy to obtain target data.

[0051] It can be seen from the above embodiments that multiple data processing strategies are stored in the listener. After receiving the Canal message, one or part of the data processing strategies that meet the business needs can be selected from the multiple data processing strategies according to business needs, and the Canal message can be processed to obtain the target data.

[0052] Since the same database table usually corresponds to multiple statistical requirements, the program needs to use the strategy pattern factory pattern to obtain all the corresponding strategies that will be used, and process them one by one to ensure that the business is not missed and has good scalability. Assume that the Canal message generated by table A is currently obtained, and there are now strategies generated by three statistical requirements, A, B, and C, corresponding to table A. The processing flow of listener A is to pass the current Canal message to the factory of the statistical strategy. The factory determines which strategies among all strategies need to process the current Canal message based on specific parameters. When listener A obtains all statistical strategies, it will call the processing method of the statistical strategy one by one to process the Canal message. Different statistical strategies can count the incremental information contained in the Canal message according to the statistical requirements, and the information not included in the Canal message can also be obtained through additional network requests, so that the target data obtained is closer to user needs.

[0053] Figure 34 is a flow chart of processing Canal messages internally by a listener according to an embodiment of the present invention, such as Figure 3 As shown, when the Canal message to be processed enters the listener, the listener will obtain the corresponding statistical strategy list, declare the variable i=1, and obtain the length L of the statistical strategy list; determine whether i is less than or equal to L; if so, obtain the i-th data processing strategy and process it, otherwise, i is increased by 1 until i is greater than L, and the end is completed.

[0054] In an optional embodiment, after the Canal message is processed by the target listener to obtain the target data, the Canal-based data processing method further includes: sending the target data to a predetermined statistical platform.

[0055] In an embodiment of the present invention, within each statistical strategy (i.e., data processing strategy), the incremental information will be processed to generate statistical information records, and the statistical service will be stored in the Hadoop platform by appending writes for use in subsequent statistical tasks. Assuming that information such as user names and provinces are stored in Table A, and the current statistical requirement is to count the incremental number of users in each province on the day, then the strategy will determine whether the current Canal message belongs to the type of addition, deletion, or update, obtain the province name according to the corresponding situation, and record the incremental situation. If it is a new addition, such as adding a user in Hebei Province, "Hebei: 1" will be recorded; if it is a deletion operation, such as deleting a user in Beijing, "Beijing: -1" will be recorded; if it is an update operation, such as a user originally belonging to Hebei, he changes the province information to Beijing, it will record "Hebei: -1 Beijing: 1" and the corresponding generated statistical file should be as shown in Table 1 below:

[0056] Table 1

[0057]

[0058]

[0059] It should be noted that the names of the original statistical files corresponding to each statistical strategy are different, which will not cause file reading and writing problems. If the data to be processed is small and only the total number is required, the incremental data statistics can also be directly written to Redis in the statistical strategy, and the total number can be directly obtained by accumulating using the Redis INCR command. In this way, the storage destination of the incremental data can be customized, expanding the application scenarios.

[0060] Figure 4 is a schematic diagram of a data processing method based on Canal according to an embodiment of the present invention, such as Figure 4As shown, the canal component can generate a canal message after obtaining the monitored data and send the canal message to the Kafka queue. The customized statistical service platform can obtain the Canal message in the Kafka message queue by subscribing to the Kafka message queue, process the Canal message, obtain the target data, and send the target data to Hadoop or other statistical platforms.

[0061] The Canal-based data processing method provided by the embodiment of the present invention solves the demand for customized generation of original files required for statistics, improves the speed of subsequent statistics, reduces redundant data, saves disk overhead of original files, and uses customized statistical data processing services to monitor the Kafka message queue to generate non-redundant statistical original files directly related to big data statistics based on existing statistical needs.

[0062] In addition, the Canal-based data processing method provided by the embodiment of the present invention satisfies many statistical scenarios and has good universality; it has good scalability, and when new statistical needs are added, only the corresponding statistical strategies need to be added, which greatly reduces the maintenance cost of the project; it also reduces the statistical pressure on Hadoop, and the statistics are based on fewer redundant fields in the original files, and are no longer limited to Hadoop statistical results. It can be decided according to actual conditions to store the generated incremental statistical data in other storage databases for statistics.

[0063] Example 2

[0064] According to another aspect of the embodiment of the present invention, a data processing device based on Canal is further provided. Figure 5 is a schematic diagram of a Canal-based data processing device according to an embodiment of the present invention. Figure 5 As shown, the data processing device based on Canal may include: an acquisition unit 51, a determination unit 53 and a processing unit 55. The data processing device based on Canal is described in detail below.

[0065] The acquisition unit 51 is used to acquire the Canal message generated by the incremental log Canal component.

[0066] The determining unit 53 is configured to determine a target listener based on the characteristic information of the Canal message.

[0067] The processing unit 55 is used to process the Canal message using the target listener to obtain target data.

[0068] It should be noted that the acquisition unit 51, the determination unit 53 and the processing unit 55 correspond to steps S102 to S106 in Embodiment 1, and the examples and application scenarios implemented by the above units and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above units, as part of the device, can be executed in a computer system such as a set of computer executable instructions.

[0069] As can be seen from the above, in the above embodiments of the present application, the acquisition unit can be used to acquire the Canal message generated by the incremental log Canal component; then the determination unit can be used to determine the target listener based on the characteristic information of the Canal message; and then the processing unit can be used to process the Canal message using the target listener to obtain the target data. Through the Canal-based data processing device provided by the embodiment of the present invention, the storage medium storing business data is monitored by the Canal component to determine the target listener to process the Canal message after acquiring the Canal message to obtain the desired data, thereby achieving the technical effect of improving the reliability of data statistics of the tire, and thus solving the technical problem of low reliability of the data statistics method used for data statistics in the related technology.

[0070] In an optional embodiment, the acquisition unit includes: a first determination module, used to determine the data monitoring relationship between the Canal component and the data storage medium; a monitoring module, used to use the Canal component to monitor the data storage medium to obtain data change information of the data storage medium; and a sending module, used to send the data change message to the Kafka message queue to obtain the Canal message.

[0071] In an optional embodiment, the Canal-based data processing device also includes: an acquisition unit, used to acquire a business processing request message before determining a target listener based on characteristic information of the Canal message; and an allocation unit, used to allocate a listener to a business corresponding to the business processing request message based on business attributes of the business processing request message.

[0072] In an optional embodiment, the Canal-based data processing device also includes: a generation unit, which is used to generate a data processing strategy according to the business corresponding to each listener after allocating a listener to the business corresponding to the business processing request message based on the business attributes of the business processing request message, wherein there are multiple data processing strategies; and a storage unit, which is used to store the data processing strategy in the corresponding listener.

[0073] In an optional embodiment, the processing unit includes: a second determination module, used to determine the target business corresponding to the data storage medium monitored by the Canal component; an acquisition module, used to obtain the business requirements of the target business; a selection module, used to select a target data processing strategy from multiple data processing strategies stored in the target listener based on the business requirements; and a processing module, used to process the Canal message using the target data processing strategy to obtain the target data.

[0074] In an optional embodiment, the Canal-based data processing device further includes: a sending unit, configured to send the target data to a predetermined statistical platform after processing the Canal message using the target listener to obtain the target data.

[0075] Example 3

[0076] According to another aspect of an embodiment of the present invention, a Canal-based data processing system is provided. The Canal-based data processing system uses any one of the Canal-based data processing methods described above.

[0077] Example 4

[0078] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is executed by a processor, the device where the computer storage medium is located is controlled to execute any one of the above-mentioned Canal-based data processing methods.

[0079] Example 5

[0080] According to another aspect of an embodiment of the present invention, a processor is provided, and the processor is used to run a computer program, wherein the computer program executes any one of the above-mentioned Canal-based data processing methods when running.

[0081] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0082] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0085] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0087] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method based on Canal, characterized in that: include: Obtaining a Canal message generated by an incremental log Canal component, wherein obtaining the Canal message generated by the incremental log Canal component includes: determining a data monitoring relationship between the Canal component and a data storage medium; using the Canal component to perform data monitoring on the data storage medium to obtain a data change message of the data storage medium; sending the data change message to a Kafka message queue to obtain the Canal message; Determine a target listener based on the characteristic information of the Canal message; The target listener is used to process the Canal message to obtain target data, wherein the process of using the target listener to process the Canal message is as follows: after the Canal message to be processed enters the target listener, the target listener obtains the corresponding statistical strategy list, declares a variable i=1, and obtains the length L of the statistical strategy list; determines whether i is less than or equal to L; if so, obtains the i-th data processing strategy and processes it, otherwise, i is increased by 1 until i is greater than L, and ends; Among them, the target listener is used to process the Canal message to obtain the target data, including: determining the target business corresponding to the data storage medium monitored by the Canal component; obtaining the business requirements of the target business; selecting a target data processing strategy from multiple data processing strategies stored in the target listener based on the business requirements; and using the target data processing strategy to process the Canal message to obtain the target data.

2. The method according to claim 1, characterized in that: Before determining the target listener based on the characteristic information of the Canal message, the method further includes: Get business processing request message; Based on the service attribute of the service processing request message, a listener is allocated to the service corresponding to the service processing request message.

3. The method according to claim 2, characterized in that After allocating a listener to the service corresponding to the service processing request message based on the service attribute of the service processing request message, the method further includes: Generate a data processing strategy according to the business corresponding to each listener, wherein there are multiple data processing strategies; The data processing strategy is stored in the corresponding listener.

4. The method according to any one of claims 1 to 3, characterized in that After the target listener is used to process the Canal message to obtain the target data, the method further includes: The target data is sent to a predetermined statistical platform.

5. A data processing device based on Canal, characterized in that: include: An acquisition unit is used to acquire a Canal message generated by an incremental log Canal component, wherein the acquisition unit comprises: determining a data monitoring relationship between the Canal component and a data storage medium; using the Canal component to perform data monitoring on the data storage medium to obtain a data change message of the data storage medium; sending the data change message to a Kafka message queue to obtain the Canal message; A determining unit, configured to determine a target listener based on characteristic information of the Canal message; A processing unit is used to process the Canal message using the target listener to obtain target data, wherein the processing process of the processing unit is as follows: after the Canal message to be processed enters the target listener, the target listener obtains the corresponding statistical strategy list, declares a variable i=1, and obtains the length L of the statistical strategy list; determines whether i is less than or equal to L; if so, obtains the i-th data processing strategy and processes it, otherwise, i is increased by 1 until i is greater than L, and ends; Among them, the processing unit includes: a second determination module, used to determine the target business corresponding to the data storage medium monitored by the Canal component; an acquisition module, used to obtain the business requirements of the target business; a selection module, used to select a target data processing strategy from multiple data processing strategies stored in the target listener based on the business requirements; a processing module, used to use the target data processing strategy to process the Canal message to obtain the target data.

6. A data processing system based on Canal, characterized in that: The Canal-based data processing system uses the Canal-based data processing method described in any one of claims 1 to 4 above.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the Canal-based data processing method according to any one of claims 1 to 4.

8. A processor, characterized in that: The processor is used to run a computer program, wherein the computer program executes the Canal-based data processing method according to any one of claims 1 to 4 when running.