Method and apparatus for binary log file parsing

Abstract

The application discloses a binary log file parsing method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: acquiring a binary log file; determining a position point of the binary log file to be parsed corresponding to a specified time according to the specified time; and parsing the binary log file according to the position point. The implementation solves the technical defect that the prior art cannot quickly acquire a binary log file according to a specified time, thereby achieving the technical effects of quickly locating a binary log file to be parsed according to a specified time, reducing the client from receiving useless data, thereby reducing the consumption of the CPU of the client, and reducing the delay of binary log file parsing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and device for parsing binary log file. BACKGROUND

[0002] There are more and more scenarios of data synchronization through binary log file (binlog file). The advantages of using binary log file to synchronize data are strong real-time performance, high data accuracy and simple use. In the prior art, binary log synchronization tools are often used to parse binary log.

[0003] In the process of implementing the present application, the inventors have found that the prior art has at least the following problems:

[0004] 1. The prior art is slow in addressing incremental logs;

[0005] 2. The prior art cannot achieve high-performance synchronization of historical data;

[0006] 3. Data filtering cannot be performed according to business requirements. SUMMARY

[0007] Therefore, the embodiments of the present application provide a method and device for parsing binary log file, which can quickly locate the binary log file to be parsed according to a specified time, reduce the consumption of CPU of the client, reduce the delay of parsing binary log file, and achieve the technical effects.

[0008] To achieve the above-mentioned purpose, according to an aspect of the embodiments of the present application, a method for parsing binary log file is provided, comprising:

[0009] obtaining a binary log file;

[0010] determining a position point of the binary log file to be parsed corresponding to a specified time according to the specified time;

[0011] parsing the binary log file according to the position point.

[0012] Optionally, determining the position point of the binary log file to be parsed corresponding to the specified time according to the specified time comprises:

[0013] judging whether the position point corresponding to the specified time exists in the client;

[0014] if not, searching for the position point corresponding to the specified time in the server by using bisection method.

[0015] Optionally, obtaining the binary log file comprises:

[0016] Determine the position and offset of the start time and end time of the binary log file respectively;

[0017] Determine the number of shards according to the size of the storage space occupied by the sub-file in the binary log file, the position of the start time and end time;

[0018] According to the association between each shard and the corresponding sub-file and the start position and end position of the binary log file, the binary log file is obtained.

[0019] Optionally, before parsing the binary log file according to the position point, it comprises:

[0020] Filter the data table in the binary log file.

[0021] Optionally, filtering the table in the binary log file comprises:

[0022] According to the black list and the white list, the data table in the binary log file is filtered;

[0023] Wherein, the black list is a set of data tables that are not allowed to be obtained, and the white list is a set of data tables that are allowed to be obtained.

[0024] Optionally, parsing the binary log file according to the position point comprises:

[0025] Determine the event type of the data table in the binary log file; the event type comprises: insertion, update, deletion;

[0026] When the event type is insertion, the data to be inserted is determined as the parsing result;

[0027] When the event type is update, the data before update and the data after update are determined as the parsing result;

[0028] When the event type is deletion, the primary key value data to be deleted is determined as the parsing result.

[0029] Optionally, the parsed binary log file is sent to the message queue, comprising:

[0030] Start at least one thread pool;

[0031] According to the thread pool, the parsed binary log is sent to the message queue.

[0032] Optionally, after parsing the binary log file according to the position point, it comprises:

[0033] The parsed binary log file is sent to the message queue;

[0034] Send the position point and the offset of the binary log file corresponding to the position point to the client.

[0035] Optionally, after sending the parsed binary log file to the message queue, the method further comprises:

[0036] Obtain a return result of the synchronization state of the client and the server.

[0037] According to the return result, determine a preset threshold condition.

[0038] If the preset threshold condition is met, determine that there is a delay in the parsing of the binary log file.

[0039] The preset threshold condition comprises: determining whether the time difference between the client and the server is greater than a preset threshold of the time difference; determining whether the last binary log file and the offset thereof in the client and the server are different; and determining whether the time difference between the last submission time of the server and the last stored cutoff time of the client is greater than a preset threshold of the cutoff time difference.

[0040] According to still another aspect of the embodiment of the application, a device for parsing a binary log file is provided, comprising:

[0041] A file obtaining module is configured to obtain a binary log file.

[0042] A position point determining module is configured to determine, according to a specified time, a position point of a binary log file to be parsed corresponding to the specified time.

[0043] A parsing module is configured to parse the binary log file according to the position point.

[0044] Optionally, determining, according to a specified time, a position point of a binary log file to be parsed corresponding to the specified time comprises:

[0045] Determining whether the position point corresponding to the specified time exists in the client.

[0046] If not, searching for the position point corresponding to the specified time in the server by using a dichotomy method.

[0047] Optionally, obtaining a binary log file comprises:

[0048] Respectively determining the position and offset of the start time and the end time of the binary log file.

[0049] According to the size of the storage space occupied by the sub-file in the binary log file, the position of the start time and the end time, determining the number of fragments.

[0050] According to the association between each shard and the corresponding sub-file and the start position and the end position of the binary log file, the binary log file is acquired.

[0051] Optionally, before the binary log file is parsed according to the position point, the method comprises:

[0052] The data table in the binary log file is filtered.

[0053] Optionally, the filtering of the table in the binary log file comprises:

[0054] The data table in the binary log file is filtered according to the black list and the white list.

[0055] The black list is a set of data tables that are not allowed to be acquired, and the white list is a set of data tables that are allowed to be acquired.

[0056] Optionally, the binary log file is parsed according to the position point, and the method comprises:

[0057] The event type of the data table in the binary log file is determined; the event type comprises insertion, update and deletion.

[0058] When the event type is insertion, the data to be inserted is determined as the parsing result.

[0059] When the event type is update, the data before update and the data after update are determined as the parsing result.

[0060] When the event type is deletion, the primary key value data to be deleted is determined as the parsing result.

[0061] Optionally, the parsed binary log file is sent to the message queue, and the method comprises:

[0062] At least one thread pool is started.

[0063] The parsed binary log is sent to the message queue according to the thread pool.

[0064] Optionally, after the binary log file is parsed according to the position point, the method comprises:

[0065] The parsed binary log file is sent to the message queue.

[0066] The position point and the offset of the position point corresponding to the binary log file are sent to the client.

[0067] Optionally, the method further comprises: after the parsed binary log file is sent to the message queue by the delay monitoring module.

[0068] Obtaining a return result of a client synchronizing a state with a server;

[0069] According to the return result, judging a preset threshold condition;

[0070] If the preset threshold condition is met, determining that there is a delay in the binary log file parsing;

[0071] The preset threshold condition includes: judging whether the time difference between the client and the server is greater than a preset threshold of the time difference; judging whether the last binary log file and its offset in the client and the server are different; and judging whether the time difference between the last submission time of the server and the last stored cutoff time of the client is greater than a preset threshold of the cutoff time difference.

[0072] According to another aspect of the embodiment of the present application, an electronic device for binary log file parsing is provided, comprising:

[0073] One or more processors;

[0074] A storage device for storing one or more programs,

[0075] When the one or more programs are executed by the one or more processors, the one or more processors implement the binary log file parsing method provided by the present application.

[0076] According to still another aspect of the embodiment of the present application, a computer readable medium having a computer program stored thereon is provided, and the program is executed by a processor to implement the binary log file parsing method provided by the present application.

[0077] One embodiment of the above-mentioned application has the following advantages or beneficial effects:

[0078] The present application can determine the position point corresponding to the specified time through the binary log file, and then parse the binary log file according to the position point, and store the parsed binary log file in the message queue for the client to obtain, thereby solving the technical defect that the prior art cannot quickly obtain the binary log file according to the specified time, and achieving the technical effects of quickly positioning the binary log file to be parsed according to the specified time, reducing the client to receive useless data, thereby reducing the consumption of the CPU of the client, and reducing the delay in the binary log file parsing.

[0079] The further effects of the above-mentioned non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0080] The accompanying drawings serve to better understand the present application and do not constitute an improper limitation on the present application. Among them:

[0081] Figure 1 is a schematic diagram of the main flow of the method for parsing binary log files according to an embodiment of the present application;

[0082] Figure 2 is a schematic diagram of the steps of acquiring binary log files by sharding and multi-processes according to an embodiment of the present application;

[0083] Figure 3 is a schematic diagram of the flow of sending binary log files to a message queue by multi-thread according to an embodiment of the present application;

[0084] Figure 4 is a schematic diagram of the overall framework of the method for parsing binary log files according to an embodiment of the present application;

[0085] Figure 5 is a schematic diagram of the main modules of the device for parsing binary log files according to an embodiment of the present application;

[0086] Figure 6 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0087] Figure 7 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0088] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, which should be considered in their context only. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0089] Figure 1 is a schematic diagram of the main flow of the method for parsing binary log files according to an embodiment of the present application, as shown in Figure 1

[0090] S101, acquiring binary log files;

[0091] S102, determining a position point of a binary log file to be parsed corresponding to a specified time according to the specified time;

[0092] S103, parsing the binary log file according to the position point.

[0093] The number of binary log files can be one or more; and there is at least one data table in each binary log file. ​

[0094] The position point refers to a position where a binary log file (binlog file) is stored in a server, which is in the form of a string of numbers. The client needs to obtain the position point in the server first to obtain the binary log file stored in the server.

[0095] The prior art more often uses a current position point or a specified position point to obtain a binary log file (binlog file). In some cases, the client needs to obtain the corresponding binary log file according to a specified time. However, the prior art cannot obtain and parse the binary log file according to the specified time.

[0096] The technical means of the present application can determine the position point corresponding to the specified time through the binary log file, and then parse the binary log file according to the position point, and store the parsed binary log file in the message queue for the client to obtain. The technical means solve the technical defect that the prior art cannot quickly obtain the binary log file according to the specified time, and further achieve the technical effects of quickly locating the binary log file to be parsed according to the specified time, reducing the client to receive useless data, and further reducing the consumption of the CPU of the client, and reducing the slow parsing of the binary log file.

[0097] After step S103, the following can also be included:

[0098] The parsed binary log file is sent to the message queue.

[0099] The position point and the offset of the binary log file corresponding to the position point are sent to the client.

[0100] The sending operation can be completed by using an asynchronous other thread. Other data such as the time point corresponding to the position point can also be stored in the database of the client.

[0101] For example, after the message queue obtains the parsed binary log file, the binary log file name, the offset, and the corresponding time are stored in the database Redis of the client at a speed of two minutes each time. If a failure occurs at 1 minute and 30 seconds, the log data that needs to be obtained later can be re-obtained according to the position point that has been stored in the Redis. The form stored in the database of the client is shown in the following table:

[0102]

[0103] According to the specified time acquisition of the binary log file to be parsed needs to determine the corresponding location point of the specified time in the server. When the client suddenly crashes during the process of parsing the binary log file, the first time the binary log file is parsed, or other cases where the location point of the specified time needs to be obtained, if the location point is already stored in the database of the client, the corresponding binary log file in the server is directly obtained using the location point; if the location point is not stored in the database of the client, the bisection method is used to find the corresponding location point of the specified time in the server, and then the binary log file is obtained according to the location point.

[0104] Specifically, after determining the location point, the location point is used, and the location point and the offset are sent to the server as parameters, so that the server returns the binary log file at the corresponding location point.

[0105] Optionally, before obtaining the binary log file, the method comprises:

[0106] According to the input parameters that the client can carry, such as the address, port, username, password and the like of the server database, the client is pseudo-constructed as a slave database of the server, and a long connection is established with the server (master) by sending a "0x15 COM_REGISTER_SLAVE" instruction.

[0107] After establishing the long connection, the client can send an instruction (such as "0x12 COM_BINLOG_DUMP") for obtaining the binary log file. When the server receives the instruction, the binary log file is returned.

[0108] Specifically, the input parameters can include the following input parameters:

[0109] Input parameters Value status 0x12 server-id Unique number, not repeated between clients binlog-filename mysql-bin.000241 binlog-pos Offset

[0110] The returned header in the binary log file can include the following returned parameters:

[0111] Return parameters Explanation timestamp Time of binlog execution event type Event event type server-id Slave id event-size Event length log pos Position flags

[0112] When the client needs to obtain a binary log file of a long time period, there may be a technical defect that the binary log file is too large, so that the speed of obtaining the binary log file is slow and the time required for synchronization is long. Therefore, the binlog file can be obtained in a way of sharding and multi-process, so as to achieve the beneficial effects of greatly improving the parsing speed and shortening the parsing time.

[0113] Figure 2is a schematic diagram of steps of acquiring a binary log file by a sharded multi-process according to an embodiment of the present application. The steps of acquiring a binary log file by a sharded multi-process include:

[0114] The start time and end time of the binary log file are determined respectively, and the position and offset of the binary log file are determined. If the binary log file and the corresponding offset are not found, it indicates that the binary log file in the time period has been deleted.

[0115] Since the storage space occupied by each sub-file of the binary log file is the same, the number of shards can be determined according to the size of the storage space occupied by the sub-file, the start time and the end time of the position. The number of shards can be set to be not less than the number of sub-files, thereby improving the acquisition efficiency.

[0116] According to the association between each shard and the corresponding sub-file and the start position and end position of the binary log file, the binary log file is acquired.

[0117] Before acquiring the binary log file, the start position and end position of each shard corresponding to the sub-file are determined, and the association between each shard and the corresponding sub-file is established.

[0118] Optionally, after acquiring the current shard returns the binlog file and the offset, the end position of the client can be compared. If the binlog file corresponding to the current shard coincides with the binlog file position of the end position of the client, the connection can be disconnected, indicating that the acquisition of the binary log file in the time period has been completed.

[0119] The above steps can be used to acquire the binary log file for each client of the server, thereby achieving the technical effect of high performance, multi-process and high concurrency in processing a large amount of historical data.

[0120] After the client initiates a binlog file request to the server database, the server returns the binlog file on the entire instance of the database to the client. If a server database has several hundred data tables (tables), but the client only needs to receive part of the data tables, the prior art needs to synchronize all the data tables to the client, which brings a lot of data pressure to the client, and subsequent parsing of the binlog file needs to be unpacked, which consumes more CPU and has low efficiency of receiving. Therefore, before parsing the binary log file according to the position point, it includes filtering the data tables in the binary log file.

[0121] Filtering the tables in the binary log file can include:

[0122] Filtering data tables in the binary log file according to the black list and the white list;

[0123] The black list is a set of data tables that are not allowed to be acquired, and the white list is a set of data tables that are allowed to be acquired. Before filtering, the black list and the white list for filtering data tables in the binary log file are configured. According to comparison between the header of the parsed binary log file and the black list and the white list, the entire binary log file does not need to be parsed, thereby achieving the technical effects of quickly filtering data tables and reducing CPU usage.

[0124] Optionally, the binary log file is parsed according to the position point, which can also be regarded as filtering the binary log file. Through the parsing operation, statements such as database definition language (DDL) and database control language (DCL) can be filtered out, reducing the client from receiving useless data and further reducing the consumption of the CPU of the client. Specifically, the binary log file is parsed, including:

[0125] The event type of the data table in the binary log file returned by the server is determined (optionally, ROWS_EVENT, operation content event). The event type includes insert (corresponding to WRITE_ROWS_EVENT in ROWS_EVENT), update (corresponding to UPDATE_ROWS_EVENT in ROWS_EVENT), and delete (corresponding to DELETE_ROWS_EVENT in ROWS_EVENT).

[0126] Specifically, when the event type is insert, the data to be inserted is determined as the parsing result.

[0127] When the event type is update, the data before update and the data after update are determined as the parsing result.

[0128] When the event type is delete, the primary key value data to be deleted is determined as the parsing result.

[0129] Before executing the ROWS_EVENT event, the database management system (in this embodiment, MySQL is taken as an example) can execute Format_desc, TABLE_MAP_EVENT, Query, Table_map, Xid and other events.

[0130] The Format_desc is used to indicate the version of MySQL and the version of the binlog file.

[0131] The TABLE_MAP_EVENT indicates the internal ID of the table and the definition of the table.

[0132] Query indicates the start of an event;

[0133] Xid indicates a commit transaction;

[0134] Table_map is used to describe the internal ID and structure definition of a table. By the above event, the correspondence of table_id (unique identification of a data table) and table_name (data table name) can be set and stored in the memory.

[0135] The byte structure of the binlog file is shown in the following table:

[0136]

[0137] The steps of parsing the binary log file according to the present application are described in detail below. Specifically:

[0138] First, the binlog file is obtained, and the operation type of the binlog file is determined according to the event type. If it is an insertion, update or deletion, it is indicated that the above data structure is met.

[0139] Then, the table_id can be obtained from the private time header of the binlog file. After obtaining the table_id, the black list and the white list pre-configured by the management console are matched, and the useless data table can be filtered out without deep parsing of the data table.

[0140] Finally, the binary log file is parsed according to the operation type of the binlog file.

[0141] The following is a message sample generated according to the above steps of parsing the binary log file:

[0142] BEGIN / / start of event

[0143] #180903 2:43:23 server id 10 end_log_pos 265 CRC32 0xfdd6e14d

[0144] / / client information

[0145] Table_map:`sequence_db1`.`sequence_0` mapped to number 81 / / table name

[0146] #at 265

[0147] #180903 2:43:23 server id 10 end_log_pos 314 CRC32 0xb7cdf0eb / / client information

[0148] Write_rows:table id 81 flags:STMT_END_F / / write data to table id=81

[0149] ###INSERT INTO `sequence_db1`.`sequence_0`

[0150] ###SET

[0151] ###@1='1111'

[0152] ###@2=2222

[0153] ###@3=33

[0154] #180903 2:43:23 server id 10 end_log_pos 345 CRC32 0x82c6b999 Xid=72

[0155] COMMIT / *!* / ;。

[0156] By filtering and parsing the binary log file, most of the unnecessary data has been filtered out, but if the parsed binary log file is sent to the message queue through a single thread, the speed is still slow.

[0157] Therefore, the parsed binary log file is optionally sent to the message queue, comprising:

[0158] Starting at least one thread pool;

[0159] According to the thread pool, the parsed binary log is sent to the message queue (MQ).

[0160] Figure 3 is a flowchart of sending a binary log file to a message queue using multiple threads according to an embodiment of the application. As shown, the step of sending the parsed binary log file to the message queue is explained in detail with a specific embodiment: Figure 3

[0161] S301, first store the received binlog file in the LinkedBlockingDeque queue (Queue) in the memory. In order to protect the heap memory from being occupied, the size of the queue can be set; ​

[0162] S302, start n thread pools, and batch send the parsed binary log files to a message queue, where n can be approximately equal to the number of threads of a CPU;

[0163] S303, after the message queue sends successfully, write the cached files and offsets to redis. In batch processing, if there is a data sending failure, it is determined that the data of the thread pool fails. Therefore, there can be duplicate data in the data of the message queue.

[0164] S304, if there is data sending failure in the message queue, the failed data can be deleted, and then the corresponding parsed binary log file is resent.

[0165] Optionally, after sending the parsed binary log file to the message queue, it further includes: delaying monitoring the synchronization of the parsed binary log file to the message queue, and specifically includes the following steps:

[0166] obtaining a return result of a synchronization state of a client and a server;

[0167] According to the return result, the following three situations are determined:

[0168] 1. According to the return result, it is determined whether the time difference (Seconds_Behind_Master) between the client and the server is greater than a preset threshold of the time difference; wherein, when Seconds_Behind_Master returns a number, it indicates the number of seconds between the client and the server; null indicates an unknown number, which can indicate that there is a problem between the client and the server. If the time difference is greater than the threshold, it is determined that there is a delay.

[0169] 2. According to the return result (show binary logs), it is determined whether the last binary log file and its offset in the client and the server are the same;

[0170] 3. According to the return result, it is determined whether the time difference between the last submission time of the server and the last stored cutoff time of the client is greater than a preset threshold of the cutoff time difference;

[0171] If the time difference is greater than its preset threshold, the last binary log file and its offset in the client and the server are not the same, and the submission time and the cutoff time difference are greater than its threshold (that is, the three situations are satisfied at the same time), it is determined that the binary log file parsing has a delay.

[0172] Figure 4 is a schematic diagram of the overall framework of the binary log file parsing method according to an embodiment of the application.

[0173] AsFigure 4 The client in the dispatch group initiates a request for the binary log file to the server according to respective needs. The needs include a specified time corresponding to the binary log file to be parsed.

[0174] According to the specified time, a position point corresponding to the specified time is determined, and then the binary log file is parsed. Then, the parsed binary log file is synchronized to the message queue in a concurrent manner. Finally, the position point and the offset are stored in the database of the client.

[0175] The delay monitoring module can be used to monitor whether there is a delay in the parsing process, and if there is a delay, a warning can be sent to the client.

[0176] Figure 5 is a schematic diagram of the main modules of the binary log file parsing device according to an embodiment of the application. As shown in Figure 5 The binary log file parsing device 200 is provided, which comprises:

[0177] The file acquisition module 501 is configured to acquire the binary log file.

[0178] The position point determination module 502 is configured to determine, according to the specified time, a position point of the binary log file to be parsed corresponding to the specified time.

[0179] The parsing module 503 is configured to parse the binary log file according to the position point.

[0180] Optionally, according to the specified time, the position point of the binary log file to be parsed corresponding to the specified time is determined, which comprises:

[0181] It is judged whether the position point corresponding to the specified time exists in the client.

[0182] If not, the binary search is used to find the position point corresponding to the specified time in the server.

[0183] Optionally, the binary log file is acquired, which comprises:

[0184] The positions and offsets of the start time and the end time of the binary log file are determined respectively.

[0185] According to the size of the storage space occupied by the sub-file in the binary log file, the positions of the start time and the end time, the number of fragments is determined.

[0186] According to the association between each fragment and the corresponding sub-file and the start position and the end position of the binary log file, the binary log file is acquired.

[0187] Optionally, before resolving the binary log file according to the position point, comprising:

[0188] Filtering the data table in the binary log file.

[0189] Optionally, filtering the table in the binary log file, comprising:

[0190] Filtering the data table in the binary log file according to the black list and the white list;

[0191] Wherein, the black list is a set of data tables that are not allowed to be obtained, and the white list is a set of data tables that are allowed to be obtained.

[0192] Optionally, resolving the binary log file according to the position point, comprising:

[0193] Determining the event type of the data table in the binary log file; the event type comprises: insertion, update, deletion;

[0194] When the event type is insertion, the data to be inserted is determined as the resolution result;

[0195] When the event type is update, the data before update and the data after update are determined as the resolution result;

[0196] When the event type is deletion, the primary key value data to be deleted is determined as the resolution result.

[0197] Optionally, sending the resolved binary log file to the message queue, comprising:

[0198] Starting at least one thread pool;

[0199] According to the thread pool, the resolved binary log is sent to the message queue.

[0200] Optionally, after resolving the binary log file according to the position point, comprising:

[0201] Sending the resolved binary log file to the message queue;

[0202] Sending the position point and the offset of the position point corresponding to the binary log file to the client.

[0203] Optionally, further comprising: a delay monitoring module, configured to send the resolved binary log file to the message queue after;

[0204] Obtaining the return result of the synchronization state of the client and the server;

[0205] According to the return result, judging the preset threshold condition;

[0206] If all preset threshold conditions are met, it is determined that there is a delay in parsing the binary log file;

[0207] The preset threshold conditions include: determining whether the time difference between the client and the server is greater than a preset threshold for the time difference; determining whether the last binary log file and its offset are different in the client and the server; and determining whether the difference between the server's last commit time and the client's last storage deadline time is greater than a preset deadline time difference threshold.

[0208] Figure 6 An exemplary system architecture 600 is shown that can be applied to the binary log file parsing method or binary log file parsing apparatus of the present invention.

[0209] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0210] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0211] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0212] Server 605 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 601, 602, and 603 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal devices.

[0213] It should be noted that the binary log file parsing method provided in this embodiment of the invention is generally executed by server 605, and correspondingly, the binary log file parsing device is generally set in server 605.

[0214] It should be understood that Figure 6The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0215] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0216] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0217] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0218] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing module (CPU) 701, it performs the functions defined above in the system of this invention.

[0219] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0220] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0221] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described modules can also be arranged in a processor, for example, a processor can be described as including a sending module, an obtaining module, a determining module and a first processing module. In some cases, the names of these modules do not constitute a limitation on the modules themselves, for example, the sending module can also be described as "a module that sends an image obtaining request to a connected server".

[0222] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0223] obtaining a binary log file;

[0224] determining a position point of the binary log file to be parsed corresponding to a specified time according to the specified time;

[0225] parsing the binary log file according to the position point.

[0226] According to the technical solutions of the embodiments of the present application, the following beneficial effects can be achieved:

[0227] The present application determines the position point corresponding to the specified time through the binary log file, then parses the binary log file according to the position point, and stores the parsed binary log file in the message queue for the client to obtain, thereby solving the technical defect that the prior art cannot quickly obtain the binary log file according to the specified time, and achieving the technical effects of quickly locating the binary log file to be parsed according to the specified time, reducing the client to receive useless data, thereby reducing the consumption of the CPU of the client, and reducing the delay of binary log file parsing.

[0228] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for parsing binary log files, characterized in that, include: Determine the position and offset of the start and end times of the binary log file, respectively; The number of fragments is determined based on the storage space occupied by the subfiles in the binary log file and the positions of the start and end times; Obtain the binary log file based on the association between each shard and its corresponding sub-file, and the start and end positions of the binary log file; Based on a specified time, determine the location point of the binary log file to be parsed corresponding to the specified time, where the location point refers to the location where the binary log file is stored on the server; The binary log file is parsed based on the location points.

2. The method according to claim 1, characterized in that, Determine the location point of the binary log file to be parsed corresponding to the specified time, including: determining whether the location point corresponding to the specified time exists in the client; If it does not exist, then a binary search method is used to find the location point corresponding to the specified time in the server.

3. The method according to claim 1, characterized in that, Before parsing the binary log file based on the said location point, the process includes: The data tables in the binary log file are filtered.

4. The method according to claim 3, characterized in that, Filtering the tables in the binary log file includes: The data tables in the binary log file are filtered based on the blacklist and whitelist; The blacklist is a set of data tables that are not allowed to be accessed, and the whitelist is a set of data tables that are allowed to be accessed.

5. The method according to claim 4, characterized in that, Parsing the binary log file based on the location points includes: Determine the event types of the data tables in the binary log file; the event types include: insert, update, and delete; When the event type is "insert", the data to be inserted is determined as the parsing result; When the event type is update, the data before and after the update are determined as the parsing result; When the event type is deletion, the primary key value data to be deleted is determined as the parsing result.

6. The method according to claim 5, characterized in that, Send the parsed binary log file to the message queue, including: Start at least one thread pool; Based on the thread pool, the parsed binary logs are sent to the message queue.

7. The method according to claim 6, characterized in that, After parsing the binary log file based on the stated location point, the process includes: Send the parsed binary log file to the message queue; The location point and the offset of the corresponding binary log file are sent to the client.

8. The method according to claim 6, characterized in that, After sending the parsed binary log file to the message queue, the following is also included: Get the return result of the client and server synchronization status; Based on the returned results, determine the preset threshold conditions; If all preset threshold conditions are met, it is determined that there is a delay in parsing the binary log file; The preset threshold conditions include: determining whether the time difference between the client and the server is greater than a preset threshold for the time difference; determining whether the last binary log file and its offset are different in the client and the server; and determining whether the difference between the server's last commit time and the client's last storage deadline time is greater than a preset deadline time difference threshold.

9. An apparatus for parsing binary log files, characterized in that, include: The file acquisition module is used to determine the position and offset of the start and end times of the binary log file, respectively. The number of shards is determined based on the storage space occupied by the sub-files in the binary log file, the start time, and the end time. The binary log file is then obtained based on the association between each shard and its corresponding sub-file, and the start and end positions of the binary log file. The location point determination module is used to determine the location point of the binary log file to be parsed corresponding to a specified time, where the location point refers to the location where the binary log file is stored on the server. The parsing module is used to parse the binary log file based on the location point.

10. An electronic device for parsing binary log files, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images