Binlog event displacement correction implementation method, device and equipment
By comparing the end displacement between adjacent events and correcting the end displacement of the current event, the numerical overflow problem when the binlog log file size exceeds 4GB in the MySQL database is solved, ensuring normal data synchronization.
Patent Information
- Application Number
- CN202411903954.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-15
AI Technical Summary
In MySQL database, when the binlog log file size exceeds 4GB, the event end displacement value will overflow, resulting in logging errors, affecting the normal progress of data synchronization.
By comparing the end displacements between adjacent events, we judge whether a numerical overflow problem occurs, and correct the end displacement of the current event based on the end displacement of the previous event and the message length of the current event.
The end displacement value of large transaction data events in the binlog log is effectively corrected, which avoids the problem of numerical overflow, ensures the accurate calculation of the event start displacement, and ensures the normal synchronization of MySQL database data.
Smart Images
Figure CN119829679A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device and equipment for implementing binlog event displacement correction. Background Art
[0002] The binlog log records all write operations that change the MySQL (a relational database management system) database, including all operations that change the database data, table structure, index, etc. By parsing the binlog log, the write operations that occurred in the source MySQL database are repeated in sequence on the destination, which is a common way to synchronize data with the MySQL source. Binlog is the binary log file of the MySQL database.
[0003] In binlog logs, write operations exist in the form of events. Events consist of a message header and a message body. The message header is a byte array with a length of 19. The unsignedInt (an unsigned integer type) value stored in the four bytes from 10 to 13 is the message body length of the current event, and the unsignedInt value stored in the four bytes from 14 to 17 is the end displacement of the current event in the binlog log file. According to the end displacement of the current event and the message body length of the current event, the starting displacement of the current event in the binlog log can be calculated. In the data synchronization process, the starting displacement of the event plays an irreplaceable role as the global unique identifier of the event.
[0004] In MySQL, all operations in a transaction are recorded in the same binlog file and will not be recorded across files. When a transaction is large, the binlog file that records the transaction will also be large. Furthermore, when the size of the binlog file exceeds 4GB, the end displacement value of some events recorded in the binlog file will be greater than the maximum unsignedInt value that can be stored in four bytes (i.e. 4294967295), that is, a numerical overflow problem occurs, resulting in an error in recording the end displacement of the event. Therefore, how to correct the end displacement value of large transaction data events in the binlog log to avoid the error in calculating the start displacement of the event due to the error in recording the end displacement of the event, which affects the normal progress of MySQL database data synchronization, has become a problem that needs to be solved urgently. Summary of the invention
[0005] The present application provides a binlog event displacement correction implementation method, device and equipment, which can effectively correct the end displacement value of large transaction data events in binlog log data and solve the problem of value overflow.
[0006] In a first aspect, an embodiment of the present application provides a method for implementing binlog event displacement correction, and the method for implementing binlog event displacement correction includes:
[0007] Obtain binlog log data, and parse the binlog log data to obtain the end displacement and message body length of a single event;
[0008] Based on the comparison of the end displacements between adjacent events, the judgment of numerical overflow problem is realized to correct the end displacement of the current event by combining the end displacement of the previous event and the message body length of the current event.
[0009] In combination with the first aspect, in one implementation, the comparison of the end displacements between adjacent events is used to implement the judgment of the numerical overflow problem so as to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event, specifically including:
[0010] Determine the type of the current event:
[0011] If it is a format description type, no processing is done;
[0012] If it is a file switching type, the corresponding value of the end displacement of the previous event is reset to zero;
[0013] If it is a non-format description type and a file switching type, based on the comparison of the end displacement between the previous event and the current event, the numerical overflow problem is judged to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event.
[0014] In conjunction with the first aspect, in one implementation, for events of non-format description type and file switching type:
[0015] If the end displacement of the current event is greater than the end displacement of the previous event, it is determined that the current event does not have a numerical overflow problem, and the end displacement of the previous event is refreshed;
[0016] If the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event. The end displacement of the current event is corrected based on the end displacement of the previous event and the message body length of the current event, and the end displacement of the previous event is refreshed.
[0017] In combination with the first aspect, in one implementation, the correction of the end displacement of the current event is specifically as follows:
[0018] Add the end offset of the previous event to the message body length of the current event, and the resulting value is used as the end offset of the current event.
[0019] In combination with the first aspect, in one implementation, refreshing the displacement of the last event is specifically as follows:
[0020] Replaces the end offset of the previous event with the end offset of the current event.
[0021] In a second aspect, an embodiment of the present application provides a BINLOG event displacement correction implementation device, the BINLOG event displacement correction implementation device comprising:
[0022] An acquisition unit, which is used to acquire binlog log data and parse the binlog log data to obtain the end displacement and message body length of a single event;
[0023] The execution unit is used to judge the numerical overflow problem based on the comparison of the end displacements between adjacent events so as to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event.
[0024] In conjunction with the second aspect, in one implementation, the acquisition unit specifically includes:
[0025] The binlog data stream listener is used to establish a connection with the MySQL database to monitor the binlog log data;
[0026] The event parser is used to parse the binlog log data monitored by the binlog data stream listener to obtain a single event, and obtain the end displacement and message body length of the single event.
[0027] In conjunction with the second aspect, in one implementation,
[0028] The execution unit specifically includes an event displacement correction module, an event buffer queue, and a replay operation encapsulation and execution module;
[0029] The event displacement correction module is used to implement the judgment of the numerical overflow problem based on the comparison of the end displacements between adjacent events according to the parsing result of the event parser, so as to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event, and write the event into the event cache queue;
[0030] The replay operation encapsulation and execution module is used to take out events from the event cache queue in sequence, encapsulate the events into replay operations adapted to the target data warehouse, and execute them in the target data warehouse to achieve data synchronization from the MySQL database to the target data warehouse.
[0031] In conjunction with the second aspect, in one implementation, the event displacement correction module is specifically configured to:
[0032] Determine the type of the current event:
[0033] If it is a format description type, the current event is written into the event buffer queue;
[0034] If it is a file switching type, the corresponding value of the end displacement of the previous event is reset to zero, and the current event is written into the event cache queue;
[0035] If it is a non-format description type and a file switching type, then:
[0036] - When the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event, and the end displacement of the current event is corrected in combination with the end displacement of the previous event and the message body length of the current event, and the end displacement of the previous event is refreshed, and the corrected current event is written into the event cache queue;
[0037] -When the end displacement of the current event is greater than the end displacement of the previous event, it is determined that the current event does not have a numerical overflow problem, then the end displacement of the previous event is refreshed, and the current event is written into the event cache queue.
[0038] In a third aspect, an embodiment of the present application provides a BINLOG event displacement correction implementation device, the BINLOG event displacement correction implementation device comprising a processor, a memory, and a BINLOG event displacement correction implementation program stored in the memory and executable by the processor, wherein when the BINLOG event displacement correction implementation program is executed by the processor, the steps of the above-mentioned BINLOG event displacement correction implementation method are implemented.
[0039] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:
[0040] By comparing the end displacements of adjacent events in the same binlog log data, if the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event. Then, the end displacement of the current event is corrected in combination with the end displacement of the previous event and the message body length of the current event, so as to realize the correction of the end displacement value of the large transaction data event in the binlog log data, avoid the situation where the event end displacement value is greater than the maximum unsigned Int value that can be stored in four bytes, ensure the accuracy of the displacement record, and realize the accurate calculation of the event start displacement according to the accurately recorded event end displacement, thereby ensuring the normal synchronization of MySQL database data. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a flowchart of the method for implementing BINLOG event displacement correction in this application;
[0042] Figure 2 This is a functional module diagram of the BINLOG event displacement correction implementation device of this application;
[0043] Figure 3 This is a schematic diagram of the specific architecture of the device for implementing BINLOG event displacement correction in this application;
[0044] Figure 4 It is a flowchart of the specific implementation process of the event displacement correction module;
[0045] Figure 5 Schematic diagram of the hardware structure of the device for implementing BINLOG event displacement correction in this application. DETAILED DESCRIPTION
[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0048] In a first aspect, an embodiment of the present application provides a method for implementing BINLOG event displacement correction.
[0049] In one embodiment, referring to Figure 1 , Figure 1 This is a flowchart of the method for implementing the BINLOG event displacement correction in this application. Figure 1 As shown, the BINLOG event displacement correction implementation method includes:
[0050] S1: Obtain binlog log data, and parse the binlog log data to obtain the end displacement and message body length of a single event;
[0051] S2: Based on the comparison of the end displacements between adjacent events, the numerical overflow problem is judged to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event. In actual applications, when the end displacement of the current event is less than the end displacement of the previous event, it indicates that the current event has a numerical overflow problem and the end displacement of the current event needs to be corrected.
[0052] Furthermore, in one embodiment, based on the comparison of the end displacements between adjacent events, the judgment of the numerical overflow problem is implemented to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event, specifically including:
[0053] Determine the type of the current event:
[0054] If it is a format description type, no processing is done;
[0055] If it is a file switching type, the corresponding value of the end displacement of the previous event is reset to zero;
[0056] If it is a non-format description type and a file switching type, based on the comparison of the end displacement between the previous event and the current event, the numerical overflow problem is judged to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event.
[0057] Specifically, before comparing the end displacements between adjacent events and determining whether a numerical overflow problem occurs in the current event, it is necessary to determine the type of the current event. If the current event is a format description event that is resent during automatic reconnection, the value of the end displacement of this type of event is always 0. Although it satisfies the condition that "the end displacement of the current event is less than the end displacement of the previous event", it does not constitute a numerical overflow problem, so the event is not processed. If the current event is a file switching event, which means that the binlog log data has been switched, it is necessary to reset the corresponding value of the end displacement of the previous event to zero, and then start a new round of numerical overflow problem determination and correction process. If the current event is of other types, based on the comparison of the end displacements between the previous event and the current event, the numerical overflow problem determination is implemented to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event.
[0058] Furthermore, for events of non-format description type and file switching type:
[0059] If the end displacement of the current event is greater than the end displacement of the previous event, it is determined that the current event does not have a numerical overflow problem, and the end displacement of the previous event is refreshed;
[0060] If the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event. The end displacement of the current event is corrected based on the end displacement of the previous event and the message body length of the current event, and the end displacement of the previous event is refreshed.
[0061] Specifically, for other types of events, if the end displacement value of the current event is smaller than the end displacement value of the previous event, it indicates that a numerical overflow problem has occurred. At this time, the end displacement of the current event is corrected in combination with the end displacement of the previous event and the message body length of the current event to achieve the correction of the end displacement of the current event and refresh the end displacement of the previous event; if the end displacement value of the current event is larger than the end displacement value of the previous event, it indicates that no numerical overflow problem has occurred, and the end displacement of the previous event is refreshed.
[0062] Furthermore, in one embodiment, the correction of the end displacement of the current event is specifically as follows: the end displacement of the previous event is added to the message body length of the current event, and the resulting value is used as the end displacement of the current event. That is, when the end displacement of the current event needs to be corrected, the value of the end displacement of the previous event is added to the message body length of the current event, and the resulting value is used as the end displacement of the current event.
[0063] Furthermore, in one embodiment, the refresh of the end displacement of the previous event is specifically as follows: the end displacement of the previous event is replaced with the end displacement of the current event. That is, when the end displacement of the previous event needs to be refreshed, the end displacement of the previous event is replaced with the end displacement of the current event to achieve the refresh of the end displacement of the previous event.
[0064] The binlog event displacement correction implementation method of the embodiment of the present application compares the end displacements between adjacent events in the same binlog log data. If the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event. The end displacement of the current event is corrected in combination with the end displacement of the previous event and the message body length of the current event, thereby effectively correcting the end displacement value of the large transaction data event in the binlog log data and ensuring the accuracy of the displacement record.
[0065] In a second aspect, an embodiment of the present application also provides a device for implementing BINLOG event displacement correction.
[0066] In one embodiment, referring to Figure 2 , Figure 2 This is a functional module diagram of the BINLOG event displacement correction implementation device of this application. Figure 2 As shown, the BINLOG event displacement correction implementation device includes: an acquisition unit and an execution unit.
[0067] The acquisition unit is used to acquire binlog log data and parse the binlog log data to obtain the end displacement and message body length of a single event; the execution unit is used to judge the numerical overflow problem based on the comparison of the end displacements between adjacent events to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event.
[0068] See also Figure 3 As shown, it is a specific architecture diagram of the BINLOG event displacement correction implementation device of the present application. The acquisition unit specifically includes a binlog data stream listener and an event parser. The binlog data stream listener is used to establish a connection with a MySQL database to realize monitoring of binlog log data. Specifically, the binlog data stream listener can establish a connection with the MySQL database through a Socket (socket). After identity authentication, a binlog log data input stream of a specified binlog log data and a starting displacement is opened to realize continuous and real-time monitoring of binlog log data.
[0069] The event parser is used to parse the binlog log data monitored by the binlog data stream listener to obtain a single event, and obtain the end displacement and message body length of the single event. Specifically, the event parser parses the binlog log data into a single event according to the file format of the binlog log, obtains the end displacement and message body length of the single event, and sends the parsing result to the event displacement correction module.
[0070] The execution unit specifically includes an event displacement correction module, an event cache queue, and a replay operation encapsulation and execution module; the event displacement correction module is used to judge the numerical overflow problem based on the parsing result of the event parser and the comparison of the end displacements between adjacent events, so as to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event, and write the event into the event cache queue; the replay operation encapsulation and execution module is used to take out events from the event cache queue in sequence, encapsulate the events into replay operations adapted to the target data warehouse, so as to execute them in the target data warehouse, and realize data synchronization from the MySQL database to the target data warehouse.
[0071] The event displacement correction module is specifically used to: determine the type of the current event: if it is a format description type, write the current event into the event cache queue; if it is a file switching type, reset the corresponding value of the end displacement of the previous event to zero, and write the current event into the event cache queue; if it is a non-format description type and a file switching type, then:
[0072] When the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event, and the end displacement of the current event is corrected in combination with the end displacement of the previous event and the message body length of the current event, and the end displacement of the previous event is refreshed, and the corrected current event is written into the event cache queue;
[0073] When the end displacement of the current event is greater than the end displacement of the previous event, it is determined that no numerical overflow problem occurs in the current event, and the end displacement of the previous event is refreshed, and the current event is written into the event cache queue.
[0074] Specifically, for events of the format description type, the event has no actual effect in the data synchronization process, and the value of its end displacement is always 0, so the event displacement correction module ignores this type of event; for events of the file switching type, it indicates that the current event is at the end of the binlog log data and is about to switch to the next binlog log data, so the corresponding value of the end displacement of the previous event is reset to zero at this time, and a new round of event displacement inspection and correction begins; for other types of events, if the end displacement value of the current event is smaller than the end displacement value of the previous event, it indicates that a numerical overflow problem has occurred. At this time, the end displacement of the current event is corrected in combination with the end displacement of the previous event and the message body length of the current event, so as to realize the correction of the end displacement of the current event and refresh the end displacement of the previous event; if the end displacement value of the current event is larger than the end displacement value of the previous event, it indicates that no numerical overflow problem has occurred, and the end displacement of the previous event is refreshed at this time.
[0075] The following combination Figure 4 The specific implementation process of the event displacement correction module is explained.
[0076] A: Get the event and go to B;
[0077] B: Determine the type of the current event. If it is a file switching type, go to C; if it is a format description type, go to G; if it is other types, go to D;
[0078] C: reset the corresponding value of the end displacement of the previous event to zero and go to G;
[0079] D: Determine the end displacement between the current event and the previous event. If the end displacement of the current event is greater than that of the previous event, go to E. If the end displacement of the current event is less than that of the previous event, go to F.
[0080] E: Replace the end displacement of the previous event with the end displacement of the current event, and go to G;
[0081] F: Add the end displacement of the previous event to the message body length of the current event, and use the resulting value as the end displacement of the current event. Replace the end displacement of the previous event with the end displacement of the current event, and go to G;
[0082] G: Write the current event into the event cache queue, and then go to A to judge and process the next event.
[0083] The application effect of the BINLOG event displacement correction implementation device of the present application is described below with reference to an example.
[0084] In the same transaction, 12 rows of data are inserted into the table (ID int, V longblob), where all V column values are files of about 500MB in size, as described in the following table. Table 1 shows the event attributes before correction, and Table 2 shows the event attributes after correction.
[0085] Table 1
[0086]
[0087]
[0088] Table 2
[0089]
[0090]
[0091] It can be seen from Table 1 above that, starting from the 22nd event, the value of the event end displacement exceeds the maximum unsigned Int value (4294967295) that can be stored in four bytes, and a numerical overflow problem occurs, resulting in abnormal event displacement. It can be seen from Table 2 that after correction by the correction method described in the present application, the event displacement and end displacement are restored to normal.
[0092] In a third aspect, an embodiment of the present application provides a BINLOG event displacement correction implementation device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.
[0093] Reference Figure 5 , Figure 5 Schematic diagram of the hardware structure of the BINLOG event displacement correction implementation device involved in the embodiment of the present application. In the embodiment of the present application, the BINLOG event displacement correction implementation device may include a processor, a memory, a communication interface and a communication bus.
[0094] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.
[0095] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to interconnect the devices inside the BINLOG event displacement correction implementation device, and the interface used to interconnect the BINLOG event displacement correction implementation device with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.
[0096] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0097] The processor may be a general-purpose processor, and the general-purpose processor may call the BINLOG event displacement correction implementation program stored in the memory, and execute the BINLOG event displacement correction implementation method provided by the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the BINLOG event displacement correction implementation program is called may refer to the various embodiments of the BINLOG event displacement correction implementation method of the present application, which will not be repeated here.
[0098] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0099] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.
[0100] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.
[0101] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.
[0102] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.
[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.
[0104] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for implementing binlog event displacement correction, characterized in that: The binlog event displacement correction implementation method includes: Obtain binlog log data, and parse the binlog log data to obtain the end displacement and message body length of a single event; Based on the comparison of the end displacements between adjacent events, the judgment of numerical overflow problem is realized to correct the end displacement of the current event by combining the end displacement of the previous event and the message body length of the current event.
2. A binlog event displacement correction implementation method as claimed in claim 1, characterized in that: The comparison of the end displacements between adjacent events is used to determine the numerical overflow problem so as to correct the end displacement of the current event by combining the end displacement of the previous event and the message body length of the current event, and specifically includes: Determine the type of the current event: If it is a format description type, no processing is done; If it is a file switching type, the corresponding value of the end displacement of the previous event is reset to zero; If it is a non-format description type and a file switching type, based on the comparison of the end displacement between the previous event and the current event, the numerical overflow problem is judged to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event.
3. A binlog event displacement correction implementation method as claimed in claim 2, characterized in that: For non-format description type and file switching type events: If the end displacement of the current event is greater than the end displacement of the previous event, it is determined that the current event does not have a numerical overflow problem, and the end displacement of the previous event is refreshed; If the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event. The end displacement of the current event is corrected based on the end displacement of the previous event and the message body length of the current event, and the end displacement of the previous event is refreshed.
4. A binlog event displacement correction implementation method as claimed in claim 3, characterized in that: The correction for the end displacement of the current event is as follows: Add the end offset of the previous event to the message body length of the current event, and the resulting value is used as the end offset of the current event.
5. A binlog event displacement correction implementation method as claimed in claim 3, characterized in that: For the refresh of the displacement at the end of the previous event, specifically: Replaces the end offset of the previous event with the end offset of the current event.
6. A BINLOG event displacement correction implementation device, characterized in that: The BINLOG event displacement correction implementation device comprises: An acquisition unit, which is used to acquire binlog log data and parse the binlog log data to obtain the end displacement and message body length of a single event; The execution unit is used to judge the numerical overflow problem based on the comparison of the end displacements between adjacent events so as to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event.
7. A BINLOG event displacement correction implementation device as claimed in claim 6, characterized in that: The acquisition unit specifically includes: The binlog data stream listener is used to establish a connection with the MySQL database to monitor the binlog log data; The event parser is used to parse the binlog log data monitored by the binlog data stream listener to obtain a single event, and obtain the end displacement and message body length of the single event.
8. The device for implementing BINLOG event displacement correction as claimed in claim 7, characterized in that: The execution unit specifically includes an event displacement correction module, an event buffer queue, and a replay operation encapsulation and execution module; The event displacement correction module is used to implement the judgment of the numerical overflow problem based on the comparison of the end displacements between adjacent events according to the parsing result of the event parser, so as to correct the end displacement of the current event in combination with the end displacement of the previous event and the message body length of the current event, and write the event into the event cache queue; The replay operation encapsulation and execution module is used to take out events from the event cache queue in sequence, encapsulate the events into replay operations adapted to the target data warehouse, and execute them in the target data warehouse to achieve data synchronization from the MySQL database to the target data warehouse.
9. The BINLOG event displacement correction implementation device according to claim 8, characterized in that: The event displacement correction module is specifically used to: Determine the type of the current event: If it is a format description type, the current event is written into the event buffer queue; If it is a file switching type, the corresponding value of the end displacement of the previous event is reset to zero, and the current event is written into the event cache queue; If it is a non-format description type and a file switching type, then: - When the end displacement of the current event is less than the end displacement of the previous event, it is determined that a numerical overflow problem occurs in the current event, and the end displacement of the current event is corrected in combination with the end displacement of the previous event and the message body length of the current event, and the end displacement of the previous event is refreshed, and the corrected current event is written into the event cache queue; -When the end displacement of the current event is greater than the end displacement of the previous event, it is determined that the current event does not have a numerical overflow problem, then the end displacement of the previous event is refreshed, and the current event is written into the event cache queue.
10. A BINLOG event displacement correction implementation device, characterized in that: The BINLOG event displacement correction implementation device includes a processor, a memory, and a BINLOG event displacement correction implementation program stored in the memory and executable by the processor, wherein when the BINLOG event displacement correction implementation program is executed by the processor, the steps of the BINLOG event displacement correction implementation method as described in any one of claims 1 to 5 are implemented.