Log reading method based on log analysis synchronization and synchronization system

A log and readable technology, applied in the direction of file system, file system function, structured data retrieval, etc., can solve problems such as data synchronization errors

Pending Publication Date: 2020-10-30
WUHAN DAMENG DATABASE
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

After testing, this phenomenon has a high probability of occurring in the running environment of the virtual machine. Once it occurs, it will cause data synchronization errors. What is more serious is that if the database fails to restart at this time, the database will not be able to use this damaged REDO log for recovery

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log reading method based on log analysis synchronization and synchronization system
  • Log reading method based on log analysis synchronization and synchronization system
  • Log reading method based on log analysis synchronization and synchronization system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0052] At present, although the data lock technology can only allow one process to access a certain file, the data lock is applicable to the same type of database or the database of the same manufacturer. The write operations on one side and the read operations on the destination database side originate from databases of two different manufacturers, so the above data lock cannot be used, and the method in this embodiment is applicable to databases of different sources or types.

[0053] In order to avoid read-write conflicts between the data synchronization process and the database process, it is necessary to prevent these two processes from simultaneously accessing data at the same offset in the same file at the same time. In order to achieve this goal, it is necessary for the data synchronization process to mutually exclude it and the database by judging whether the current log to be read and the log to be written to the database are in the same position of the same log file ...

Embodiment 2

[0088] In combination with embodiment 1, this embodiment provides another log reading method, combining figure 2 , the log reading method includes the following steps:

[0089] Step 201: Obtain the log sequence number LSN1 of the log record to be read, and obtain the reading position of the log file through the log sequence number LSN1.

[0090] Step 202: Obtain the current largest log sequence number LSN2 of the source database, and obtain the write location of the log file through the log sequence number LSN2.

[0091] Wherein, the specific implementation process of step 201 and step 202 is the same as the foregoing embodiment 1, see embodiment 1 for details, and will not be repeated here.

[0092] Step 203: Determine whether the log file number ID1 corresponding to the log sequence number LSN1 is equal to the log file number ID2 corresponding to the log sequence number LSN2.

[0093] Although in actual application scenarios, in order to reduce the synchronization delay, ...

Embodiment 3

[0101] Based on the foregoing embodiment 1, here, taking DM6 as an example, the implementation process of the foregoing embodiment 1 is illustrated:

[0102] Build a data synchronization system on database A and database B, and set the maximum number of pages allowed to be read by the data synchronization process to Z pages; set the time interval for polling the database log LSN to 1 second, and synchronize the source data The system creates an auxiliary table T (C INT) on database A to push logs.

[0103] The database interaction process of the source log synchronization system is as follows:

[0104] (1) The database interaction thread obtains the largest log LSN of the current database from the source database every 1 second, and the obtained SQL is select get_last_lsn() from dual; the log sequence number LSN2 is obtained.

[0105] (2) The database interaction thread then executes the statement: BEGIN UPDATE T SET C=1; COMMIT; END; to ensure that the log operations whose l...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a log reading method based on log analysis synchronization and a synchronization system, and the method comprises the steps of obtaining a log sequence number LSN1 of a to-be-read log record, and obtaining the reading position of a log file through the log sequence number LSN1; obtaining the current maximum log sequence number LSN2 of the source end database, and obtainingthe writing position of the log file through the log sequence number LSN2; obtaining a log page number difference value of the log serial number LSN1 and the log serial number LSN2 in the log file based on the reading position of the log file and the writing position of the log file; and performing strategic log reading and synchronization according to the log page number difference. According tothe invention, by calculating the distance between the log sequence number of the log record to be read and the current maximum log sequence number in the database system, the number of log pages readwhen the log file is read can be dynamically adjusted, strategic log reading and synchronization are carried out, and log reading and writing conflicts are prevented.

Description

technical field [0001] The invention belongs to the technical field of data synchronization, and more particularly relates to a log reading method and a synchronization system based on log parsing and synchronization. Background technique [0002] In the solution based on log parsing and synchronization, a database log capture process will be deployed on the source database to continuously scan and read the online logs of the database, so as to capture the running logs of the database for data synchronization at the first time. This architecture will cause the log file of the database to be opened by two or more processes at the same time. The database opens the online log file and writes the log, while the data synchronization process opens the online log file to read the log. If the database write and data synchronization Read operations concurrently on the same file offset of the same online log file may cause read-write conflicts. The new data that is about to be writte...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/17G06F16/178G06F16/23G06F16/27
CPCG06F16/1734G06F16/178G06F16/23G06F16/27
Inventor 孙峰付铨彭青松刘启春
Owner WUHAN DAMENG DATABASE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap