Supercharge Your Innovation With Domain-Expert AI Agents!

Data processing method and device based on Canal

A technology of data processing and data processing strategy, applied in the computer field, can solve the problem of low reliability of data statistics, and achieve the effect of improving the reliability of data statistics and solving the problem of low reliability.

Pending Publication Date: 2020-11-10
中消云(北京)物联网科技研究院有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The embodiment of the present invention provides a Canal-based data processing method and device to at least solve the technical problem of low reliability of the data statistics method used for data statistics in the related art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device based on Canal
  • Data processing method and device based on Canal
  • Data processing method and device based on Canal

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0032]According to an embodiment of the present invention, a method embodiment of a Canal-based data processing method is provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions , and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

[0033] figure 1 is a flowchart of a Canal-based data processing method according to an embodiment of the present invention, such as figure 1 As shown, the Canal-based data processing method includes the following steps:

[0034] Step S102, acquiring the Canal message generated by the incremental log Canal component.

[0035] Optionally, Canal is an incremental log component based on binary logs. Its main purpose is to parse incremental logs based on the MySQL database and provide incremental data ...

Embodiment 2

[0064] According to another aspect of the embodiment of the present invention, a Canal-based data processing device is also provided, Figure 5 is a schematic diagram of a Canal-based data processing device according to an embodiment of the present invention, such as Figure 5 As shown, the Canal-based data processing device may include: an acquisition unit 51 , a determination unit 53 and a processing unit 55 . The Canal-based data processing device will be described in detail below.

[0065] The obtaining unit 51 is configured to obtain the Canal message generated by the incremental log Canal component.

[0066] A determining unit 53, configured to determine a target listener based on the feature information of the Canal message.

[0067] The processing unit 55 is configured to use the target listener to process the Canal message to obtain target data.

[0068] It should be noted here that the above acquisition unit 51, determination unit 53 and processing unit 55 corresp...

Embodiment 3

[0076] According to another aspect of the embodiments of the present invention, a Canal-based data processing system is also provided, and the Canal-based data processing system uses any one of the above-mentioned Canal-based data processing methods.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing method and device based on Canal. The method comprises the steps of obtaining a Canal message generated by an incremental log Canal component; determining a target monitor based on the feature information of the Canal message; and processing the Canal message by using the target monitor to obtain target data. According to the method and the device, the technical problem of relatively low reliability of a data statistics mode for data statistics in related technologies is solved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a Canal-based data processing method and device. Background technique [0002] In the era of big data statistics, in most cases, it is necessary to count the total number and changes of a certain business record. Usually, people will migrate the entire content of the MySQL database table to Hadoop, and then perform statistics. This aggravates Hadoop's statistical work, making the original data file contain a large amount of data that is not related to the statistical business. [0003] In view of the above problems, some of the existing technologies use a massive platform as the source database, a Hadoop big data platform as the target database, and combine the automatic incremental data import method developed by Java to realize the incremental data import in the source database into the The purpose of the target database. Although this solution adopts the need to a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F11/30G06F11/34
CPCG06F11/3065G06F11/3452
Inventor 吕恒
Owner 中消云(北京)物联网科技研究院有限公司
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com