A data processing method, device, apparatus, and computer-readable storage medium

Abstract

The application discloses a data processing method, device and equipment and a computer readable storage medium, and is applied to the field of distributed computing, and comprises the following steps: obtaining change information of a binary log record; the binary log records all change events on a database in a binary format; based on the change information, corresponding numbering and partition keys are generated for the change events; the numbering is used for representing the change event order; the partition key is used for judging whether it is the same business entity; the change events are grouped according to the partition key, and the data of the latest generated change event in each group is sent to a downstream end according to the numbering. The application guarantees that the data consumption of the downstream is the ordered change record of the business entity, avoids the problem of inconsistent data caused by disorder, and guarantees the order and consistency of the data.

Description

Technical Field

[0001] This invention relates to the field of distributed computing, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] When processing binlog (binary log) events, the data out-of-order behavior caused by multiple parallel processing is essentially due to the parallel processing disrupting the original transaction time order in the binlog. When introducing multiple parallel processing, the system splits binlog events into different processing units for parallel execution to improve efficiency. However, this splitting may cause events that originally had sequential dependencies to be assigned to different units with varying processing speeds. Ultimately, this results in the order of data received or persisted downstream being inconsistent with the original transaction order in the binlog, leading to data logic errors.

[0003] Therefore, how to ensure the parallelism of binlog event processing to improve efficiency while strictly maintaining the original execution order of transactions and avoiding logical errors caused by data disorder is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data processing method, apparatus, device and computer-readable storage medium, which solves the problem of logical errors caused by data disorder in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention provides a data processing method, comprising:

[0006] Obtain change information recorded in the binary log; the binary log records all change events to the database in binary format;

[0007] Based on the change information, a corresponding number and partition key are generated for the change event; the number is used to represent the order of the change event; the partition key is used to determine whether they belong to the same business entity.

[0008] The change events are grouped according to the partition key, and the data of the latest generated change event in each group is sent to the downstream end according to the number.

[0009] Optionally, based on the change information, a corresponding number is generated for the change event, including:

[0010] The event location is obtained from the change information recorded in the binary log; the event location is the offset of the change event in the binary log.

[0011] Obtain the filename of the binary log, and calculate the version number corresponding to the change event based on the filename and the event location, and use it as the number.

[0012] Optionally, based on the filename and the event location, the version number corresponding to the change event is calculated, including:

[0013] Perform a hash calculation on the filename to obtain a hash value;

[0014] The hash value is converted, and the converted hash value is then used to perform a bitwise operation with the event position to obtain the version number.

[0015] Optionally, based on the change information, a corresponding partition key is generated for the change event, including:

[0016] Based on the change information, the corresponding target table is determined from the database, and the table name of the target table is obtained;

[0017] The primary key field names and corresponding field values ​​are determined by querying the preset primary key mapping table;

[0018] The partition key corresponding to the change event is generated based on the field value and the table name.

[0019] Optionally, generating a partition key corresponding to the change event based on the field value and the table name includes:

[0020] The field value is reversed to obtain the reversed field value;

[0021] The partition key is obtained by concatenating the reversed field value with the table name using an underscore.

[0022] Optionally, the data of the latest generated change event within each group is sent to the downstream end according to the number, including:

[0023] Get the current partition key and current number of the data to be processed;

[0024] The current number is compared with the number of the last processed data corresponding to the current partition key to determine whether the current data to be processed should be sent to the downstream end.

[0025] Optionally, after obtaining the current partition key and current number of the data to be processed, the following steps are also included:

[0026] Create a local cache, which is used to store the latest number corresponding to each of the partition keys;

[0027] Retrieve the number of the last processed partition key from the local cache;

[0028] Accordingly, after the current data to be processed is sent to the downstream end, the number of the current partition key in the local cache is updated to the current number.

[0029] The present invention also provides a data processing apparatus, comprising:

[0030] The change information acquisition module is used to acquire change information recorded in the binary log; the binary log records all change events to the database in binary format.

[0031] The generation module is used to generate a corresponding number and partition key for the change event based on the change information; the number is used to represent the order of the change events; the partition key is used to determine whether they belong to the same business entity.

[0032] The grouping and distribution module is used to group the change events according to the partition key, and send the data of the latest generated change event in each group to the downstream end according to the number.

[0033] The present invention also provides a data processing device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor for executing the computer program to implement the steps of the data processing method described above.

[0036] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the data processing method described above.

[0037] As can be seen, this invention obtains change information from binary log records; the binary log records all change events to the database in binary format; based on the change information, corresponding numbers and partition keys are generated for each change event; the number is used to characterize the order of change events; the partition key is used to determine whether they belong to the same business entity; change events are grouped according to the partition key, and the data of the latest generated change event in each group is sent to the downstream end according to the number. The beneficial effects of this invention are: generating a unique number and partition key for each change event recorded in the binary log, grouping change events of the same business entity into a group according to the partition key, and sending the updated change event data in each group to the downstream end according to the number, ultimately ensuring that the downstream data consumption is based on the orderly change records of business entities, avoiding the problem of data inconsistency caused by out-of-order processing, and thus ensuring the order and consistency of data.

[0038] In addition, the present invention also provides a data processing apparatus, device, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] First, let me explain some of the terms used in this application:

[0045] Binary Log (BIL): A read-only, binary-format log file generated by the MySQL database engine (such as the most commonly used InnoDB). It records all changes to the database structure and data (DDL and DML statements such as INSERT, UPDATE, DELETE, ALTER TABLE, etc.), but does not record queries that do not modify data (such as SELECT, SHOW). The existence of Binary Log is mainly for three key purposes: (1) Master-slave replication, which is the most core use. In a master-slave architecture, one MySQL database acts as the master, and other databases act as slaves. The master sends its generated Binary Log to the slaves. The slaves replay the operations recorded in the Binary Log, thereby ensuring that the data in the slaves eventually remains consistent with the master. This achieves: data backup, read-write separation, load balancing, high availability, etc. (2) Point-in-Time Recovery: If a full backup (such as a database backup from midnight yesterday) and all Binlog files from that point in time are available, the database can be restored to any point in time between midnight yesterday and now. Recovery process: First, restore the full backup, then replay the Binlogs in sequence until the desired point in time or position is reached. This is crucial for recovery after accidental data deletion. (3) Data Consumption & Synchronization: Many external systems (such as big data platforms, search engines, caching systems, etc.) can obtain database changes in real time by listening to and parsing Binlogs. These tools (such as Canal, Maxwell, Debezium) disguise themselves as a MySQL slave, pull Binlogs from the master database, parse them, and send them to a message queue (such as Kafka) for downstream services to consume. This is usually called CDC (Change Data Capture).

[0046] Binlog position (event position): Its functions include: (1) It can accurately locate data changes. Each Binlog event (such as INSERT / UPDATE) has a unique position in the log file. (2) Master-slave replication synchronization base point: The slave database knows where to continue pulling data next by recording the master database's Binlog position. (3) Data recovery and backtracking: When recovering the database, the replay operation can be specified to start from a certain Binlog position. (4) CDC consumption progress management: Tools such as Flink CDC (Flink Change Data Capture) and Canal (database change capture tool) achieve exactly-Once (consistency semantics in streaming data processing, which means that each piece of data will only be processed accurately once in the processing process, without duplication or loss, ensuring the accuracy of data processing results) processing by recording the Binlog position, that is, to continue consumption from the breakpoint after the fault is recovered.

[0047] Please refer to Figure 1 , Figure 1 A flowchart illustrating a data processing method provided in an embodiment of the present invention. The method may include:

[0048] S101: Retrieve change information recorded in the binary log.

[0049] The execution entity in this embodiment is the upstream end relative to the downstream end, which can be a terminal. Specifically, it can be applied to data consumption. The binary log in this embodiment refers to the MySQL binlog. The binary log is a special file automatically generated by MySQL. It records all modification operations that occur in the database in binary (non-plaintext) format; that is, the binary log records all change events to the database in binary format. For example, various change events such as INSERT, UPDATE, and DELETE, as well as the time of these operations, the tables and fields involved, and the values ​​before and after the specific changes. Obtaining the change information recorded in the binary log involves reading and parsing these binary-formatted log contents using specific tools or programs to extract the specific change details mentioned above, providing a basis for subsequent data consumption, data synchronization, data backup, auditing, and other business scenarios.

[0050] S102: Based on the change information, generate corresponding numbers and partition keys for the change events; the numbers are used to represent the order of the change events; the partition keys are used to determine whether they belong to the same business entity.

[0051] This step assigns a unique number to each database change event, clearly indicating the chronological order of the events; it can be understood as a time sequence number for the change event. For example, an event with a larger number occurs after an event with a smaller number. This step also assigns a partition key to each database change event, which is the partition key used to partition the data stream (to determine if it belongs to the same business entity). In this way, a unique "partition key" is generated for each database change event, ensuring that changes to the same business entity (such as the same order or the same user) are grouped together, facilitating subsequent control of the order by group.

[0052] Furthermore, the above-mentioned generation of corresponding numbers for change events based on change information can specifically include:

[0053] Step 11: Obtain the event location from the change information recorded in the binary log; the event location is the offset of the change event in the binary log.

[0054] Specifically, change information in the MySQL binlog can be obtained using Flink CDC (a streaming data integration tool). The binlog position (i.e., event position) in the change information is essentially a precise and unique log coordinate used to locate a specific event in the Binlog stream. The structure of the Binlog position includes: BinlogFilename (binary log file name). Format: mysql-bin.000001 (the prefix mysql-bin is defined by the configuration item log_bin, and the suffix is ​​a 6-digit incrementing number). The generation rule is: a new log file is created every time MySQL restarts or when the current Binlog file exceeds max_binlog_size (default 1GB). The numeric suffix monotonically increases (e.g., .000001 → .000002). Binlog Offset (binary log offset, i.e., event position): The byte offset of the change event in the Binlog file (calculated from the beginning of the file), that is, the event position is the offset of the change event in the binary log. Each change event occupies a continuous block of bytes, and its starting position is the offset of that change event. Offsets are strictly incremented sequentially by event order (e.g., the first change event has an offset of 4, the second change event has an offset of 120). The full location identifier is mysql-bin.000001:154, meaning that change events are read starting from byte 154 in the binary log file mysql-bin.000001.

[0055] Step 12: Obtain the filename of the binary log and calculate the version number corresponding to the change event based on the filename and event location.

[0056] In this embodiment, the number can be in the form of a version number. First, the filename storing the change event (e.g., mysql-bin.000003, used to distinguish different Binlog files) is extracted from the MySQL binary log. Then, the specific position of the change event within the corresponding log file (i.e., the offset, such as 154, used to locate the specific event within the file) is obtained. Next, these two pieces of information are calculated using preset rules (e.g., concatenating the numerical part of the filename with the event position value, converting it to a long integer, etc.) to finally obtain a unique version number. The core function of this version number is to represent the generation order of the change events. A larger filename number indicates a newer log file, and a larger position value within the same file indicates a later change event. The version number calculated by combining these two factors allows downstream processes to directly determine the order of different change events by comparing the version numbers, avoiding data processing disorder.

[0057] Furthermore, the above calculation of the version number corresponding to the change event based on the file name and event location can specifically include the following steps: performing a hash calculation on the file name to obtain a hash value; converting the hash value; and performing a bitwise operation between the converted hash value and the event location to obtain the version number.

[0058] Specifically, in this embodiment, the binlog filename and event position are converted into a unique long integer to identify the generation order of the change event (e.g., the larger the number, the newer the change event). For example, the binlog filename (e.g., mysql-bin.000001) is hashed to obtain the fileHash (hash value); the hash value is converted to a positive number (by performing a bitwise AND with Long.MAX_VALUE (maximum value of the long integer), ensuring that the highest bit is 0); the converted hash value is left-shifted by 32 bits (as the high-order bits, ensuring the priority of the file sequence number), and then bitwise ORed with the event position (e.g., 1024, taking the lower 32 bits) to finally obtain a 64-bit version number. If the hash value converted to a positive number is shifted left to 00000001 00000000 (high byte), the event position is 00000000 00001000 (low byte), and the merged version number is 00000001 00001000, ensuring the logical order of "file sequence number first, position within the same file second".

[0059] Furthermore, the above-mentioned generation of a corresponding partition key for each change event based on change information may specifically include the following steps: determining the corresponding target table from the database based on the change information and obtaining the table name of the target table; determining the field name of the primary key and the corresponding field value by querying a preset primary key mapping table; and generating the partition key corresponding to the change event based on the field value and the table name.

[0060] Specifically, the primary key type in this embodiment can include composite primary keys or single primary keys. Based on relevant information about database change events (such as table association data recorded in the Binlog), the target table corresponding to the change is determined, and the specific name of this table is extracted (e.g., the order table (used to store order information), the user table (used to store user information); the table name is the basis for distinguishing different business data). By querying the pre-configured primary key mapping table (which records the association between each table and its corresponding primary key), the primary key field name of the target table (e.g., the primary key of the order table is order_id) and the specific field value corresponding to that primary key in the current change event (e.g., 123) is found. Finally, the obtained primary key field value is combined with the target table name to generate a unique partition key for this change event. This ensures that the partition key uniquely corresponds to a specific record in a specific table, enabling precise grouping of change events for the same business entity (such as the same order or the same user).

[0061] Furthermore, to address the issue of data clustering, the aforementioned method of generating partition keys corresponding to change events based on field values ​​and table names can specifically include the following steps: reversing the field values ​​to obtain the reversed field values; concatenating the reversed field values ​​with the table name using underscores to obtain the partition key.

[0062] Specifically, if the original primary key value (e.g., 123) is used directly, records with similar primary key values ​​may be grouped together (e.g., 123, 124, and 125 in the same group), leading to an excessively large amount of data in one group and affecting processing efficiency. Therefore, this embodiment reverses the primary key value (e.g., 123→321, 124→421), which disrupts the continuity of the original primary key, distributing primary key values ​​of different ranges into different groups, achieving a uniform data distribution, and avoiding overload of a single group. Finally, the reversed primary key value (e.g., 321) is concatenated with an underscore; this concatenation can be understood as marking the reversed value for easier subsequent combination.

[0063] For example, obtain the table name (e.g., the `order` table) from the input data (i.e., the data in the change event); check if the table has a primary key (obtained from the `primaryKeyMap`, e.g., the primary key field name is `order_id`); if a primary key is configured (most tables have primary keys), reverse the primary key field values ​​and concatenate them with underscores; finally, concatenate the table name to form the complete partition key (e.g., `321_order`, representing the change event `order_id=123` in the `order` table). The record in the `order` table with `order_id=123` will generate the key `321_order`; subsequent changes to the same order (e.g., updates, deletions) will generate the same key and be grouped into the same group. (That is, ensuring that tables with the same primary key ID are grouped together.)

[0064] S103: Group the change events according to the partition key, and send the data of the latest generated change event in each group to the downstream end according to the number.

[0065] Specifically, based on the previously generated partition key (e.g., 321_order, corresponding to the business entity with order_id=123 in the order table), all change events are categorized and grouped. Essentially, this involves grouping all change events belonging to the same business entity (such as the same order or the same user) into the same group (e.g., multiple modifications to an order's creation, payment, or shipment), avoiding confusion between changes from different entities. Using previously generated numbers representing the sequence of change events, the latest change event (i.e., the most recent change to that business entity) is selected within each group. Finally, only the latest change event data within this group is sent to the downstream end, ensuring that the downstream always receives the latest data status for each business entity. This reduces invalid data transmission and prevents state confusion caused by processing old data downstream.

[0066] Furthermore, the above-mentioned method of sending the latest generated change event data in each group to the downstream end according to the number may specifically include the following steps: obtaining the current partition key and current number of the data to be processed; comparing the current number with the number of the last processed data corresponding to the current partition key to determine whether to send the data to be processed to the downstream end.

[0067] Specifically, the consumption order of binlog events is controlled by group. Based on the partition key and number, it ensures that binlog events within the same group are processed in the order they were generated (only events newer than the last one are processed, filtering out out-of-order events of older versions). First, the partition key (used to identify the business entity to which it belongs, such as the same user or the same order) and number (used to indicate the order in which change events occurred) are extracted from the current data to be processed. Then, the number of the last processed change event corresponding to that partition key is found (i.e., the sequence number of the most recently processed event for the same business entity), and the two are compared. For example, if the current number is larger (representing a newer event), the current data to be processed is sent downstream; if the current number is smaller (representing an older event), it is not sent. This ensures that the downstream only receives the latest change data for each business entity, avoiding old data overwriting new states.

[0068] Furthermore, after obtaining the current partition key and current number of the data to be processed, the above may further include: creating a local cache to store the latest number corresponding to each partition key; querying the number of the last processed data corresponding to the current partition key from the local cache; and correspondingly, after sending the data to be processed to the downstream end, updating the number of the current partition key in the local cache to the current number.

[0069] For example, taking the version number as an example, the actual execution process when processing and using data is as follows: (1) Initialize cache: Create a local cache (versionCache) in the open (component or service initialization method) method to store the last processed version number corresponding to each partition key. The maximum cache capacity is 20,000 entries, and it will automatically expire if it is not updated for 60 seconds. (2) Process each data entry (processElement method): Get the partition key and version number of the data to be processed from the input; query the version number of the last processed data corresponding to the partition key from the local cache (if not found, the default is the minimum value); if the current version number > the previous version number (indicating an updated change event), update the version number of the partition key in the local cache to the current version number; output the data to be processed to the downstream end (normal consumption); if the current version number ≤ the previous version number (indicating an out-of-order old event), only print the warning log and do not output the data (filter out-of-order events).

[0070] The data processing method provided in this invention involves acquiring change information from binary log records; generating corresponding numbers and partition keys for change events based on the change information; using the numbers to characterize the order of change events; using the partition keys to determine whether they belong to the same business entity; grouping change events according to the partition keys; and sending the data of the latest generated change event in each group to the downstream end according to the number. This invention ensures that downstream data consumption is based on ordered change records of business entities, avoiding data inconsistencies caused by out-of-order processing, and thus effectively guaranteeing data order and consistency. This invention generates a unique number and partition key for each change event in the binary log record, groups change events of the same business entity together according to the partition key, and sends the updated change event data in each group to the downstream end according to the number. Ultimately, this ensures that downstream data consumption is based on ordered change records of business entities, avoiding data inconsistencies caused by out-of-order processing, and thus effectively guaranteeing data order and consistency.

[0071] The specific benefits of this embodiment are as follows: A unique version number is generated for each change event to reflect the generation order; a partition key is generated using the table name + primary key field value to group the same business entities together; and the data in each group of change events is checked for order: only data with a version number newer than the last processed version number in that group is processed, filtering out-of-order data of older versions. Ultimately, this ensures that downstream consumption is of change records ordered by business entity. The core function is processing MySQL binlog data, ensuring that database changes with the same primary key are consumed in order, avoiding data inconsistencies caused by out-of-order processing. This processing logic can be applied to data synchronization scenarios (such as binlog synchronization to a data warehouse), avoiding data inconsistencies such as updates occurring before insertions or duplicate updates caused by parallel consumption and network latency. Based on actual application scenarios, the order and consistency of data are well guaranteed, ensuring the accuracy of the final calculation results.

[0072] The data processing apparatus provided in the embodiments of the present invention will be described below. The data processing apparatus described below and the data processing method described above can be referred to in correspondence.

[0073] Please refer to the details. Figure 2 , Figure 2 A schematic diagram of a data processing device provided in an embodiment of the present invention may include:

[0074] The change information acquisition module 100 is used to acquire change information recorded in the binary log; the binary log records all change events to the database in binary format.

[0075] The generation module 200 is used to generate a corresponding number and partition key for the change event based on the change information; the number is used to represent the order of the change events; the partition key is used to determine whether they belong to the same business entity.

[0076] The grouping and distribution module 300 is used to group the change events according to the partition key, and send the data of the latest generated change event in each group to the downstream end according to the number.

[0077] Based on the above embodiments, the generation module 200 may include:

[0078] An event location acquisition unit is used to acquire the event location from the change information recorded in the binary log; the event location is the offset of the change event in the binary log;

[0079] The numbering calculation unit is used to obtain the file name of the binary log, and calculate the version number corresponding to the change event based on the file name and the event position, and use it as the number.

[0080] Based on the above embodiments, the numbering calculation unit may include:

[0081] A hash calculation subunit is used to perform a hash calculation on the filename to obtain a hash value;

[0082] The version number calculation subunit is used to convert the hash value and perform a bitwise operation between the converted hash value and the event position to obtain the version number.

[0083] Based on the above embodiments, the generation module 200 may include:

[0084] The table name acquisition unit is used to determine the corresponding target table from the database based on the change information, and to acquire the table name of the target table;

[0085] The field value retrieval unit is used to determine the field name and corresponding field value of the primary key by querying a preset primary key mapping table;

[0086] The partition key generation unit is used to generate the partition key corresponding to the change event based on the field value and the table name.

[0087] Based on the above embodiments, the partition key generation unit may include:

[0088] The inversion subunit is used to invert the field value to obtain the inverted field value;

[0089] The concatenation subunit is used to concatenate the reversed field value with the table name using an underscore to obtain the partition key.

[0090] Based on the above embodiments, the grouping and distribution module 300 may include:

[0091] The acquisition unit is used to acquire the current partition key and current number of the data to be processed.

[0092] The comparison unit is used to compare the current number with the number of the last processed data corresponding to the current partition key, and determine whether to send the current data to be processed to the downstream end.

[0093] Based on the above embodiments, the data processing apparatus may further include:

[0094] The local cache creation module is used to create a local cache after obtaining the current partition key and the current number of the data to be processed. The local cache is used to store the latest number corresponding to each partition key.

[0095] The number acquisition module is used to query the number of the last processed partition key corresponding to the current partition key from the local cache.

[0096] The number update module is used to update the number of the current partition key in the local cache to the current number after the current data to be processed is sent to the downstream end.

[0097] It should be noted that the order of the modules and units in the above data processing device can be changed without affecting the logic.

[0098] The data processing apparatus provided in this embodiment of the invention includes a change information acquisition module 100, used to acquire change information recorded in a binary log; the binary log records all change events to the database in binary format; a generation module 200, used to generate corresponding numbers and partition keys for the change events based on the change information; the number is used to characterize the order of the change events; the partition key is used to determine whether they belong to the same business entity; and a grouping and distribution module 300, used to group the change events according to the partition key, and send the data of the latest generated change event in each group to the downstream end according to the number. This apparatus generates a unique number and partition key for each change event recorded in the binary log, groups change events of the same business entity into a group according to the partition key, and distributes the data of the updated change events in each group to the downstream end according to the number. Ultimately, this ensures that downstream data consumption is based on the ordered change records of business entities, avoiding data inconsistencies caused by out-of-order processing, and effectively guaranteeing the order and consistency of the data.

[0099] The data processing device provided in the embodiments of the present invention will be described below. The data processing device described below and the data processing method described above can be referred to each other.

[0100] Please refer to Figure 3 , Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention may include:

[0101] Memory 10 is used to store computer programs;

[0102] Processor 20 is used to execute computer programs to implement the data processing method described above.

[0103] The memory 10, processor 20, and communication interface 31 all communicate with each other through the communication bus 32.

[0104] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:

[0105] Retrieve change information recorded in the binary log; the binary log records all change events to the database in binary format;

[0106] Based on the change information, a corresponding number and partition key are generated for each change event; the number is used to represent the sequence of change events.

[0107] The partition key is used to determine whether they belong to the same business entity; change events are grouped according to the partition key, and the data of the latest generated change event in each group is sent to the downstream end according to the number.

[0108] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0109] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0110] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0111] Communication interface 31 can be an interface for the communication module, used to connect with other devices or systems.

[0112] Of course, it should be noted that, Figure 3 The structure shown does not constitute a limitation on the data processing device in the embodiments of the present invention. In practical applications, the data processing device may include more than Figure 3 More or fewer components as shown, or combinations of certain components.

[0113] The following describes the computer-readable storage medium provided in the embodiments of the present invention. The computer-readable storage medium described below can be referred to in correspondence with the data processing method described above.

[0114] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data processing method.

[0115] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0117] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0118] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0119] The above provides a detailed description of a data processing method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, include: Retrieve change information recorded in the binary log; The binary log records all change events to the database in binary format; Based on the change information, a corresponding number and partition key are generated for the change event; The number is used to represent the sequence of change events; the partition key is used to determine whether they belong to the same business entity. The change events are grouped according to the partition key, and the data of the latest generated change event in each group is sent to the downstream end according to the number. Based on the change information, a corresponding number is generated for the change event, including: The event location is obtained from the change information recorded in the binary log; the event location is the offset of the change event in the binary log. Obtain the filename of the binary log, and calculate the version number corresponding to the change event based on the filename and the event location, and use it as the number.

2. The data processing method according to claim 1, characterized in that, Based on the filename and the event location, the version number corresponding to the change event is calculated, including: Perform a hash calculation on the filename to obtain a hash value; The hash value is converted, and the converted hash value is then used to perform a bitwise operation with the event position to obtain the version number.

3. The data processing method according to claim 1, characterized in that, Based on the change information, a corresponding partition key is generated for the change event, including: Based on the change information, the corresponding target table is determined from the database, and the table name of the target table is obtained; The primary key field names and corresponding field values ​​are determined by querying the preset primary key mapping table; The partition key corresponding to the change event is generated based on the field value and the table name.

4. The data processing method according to claim 3, characterized in that, Generate the partition key corresponding to the change event based on the field value and the table name, including: The field value is reversed to obtain the reversed field value; The partition key is obtained by concatenating the reversed field value with the table name using an underscore.

5. The data processing method according to any one of claims 1 to 4, characterized in that, Based on the stated number, the data of the latest generated change event within each group is sent to the downstream end, including: Get the current partition key and current number of the data to be processed; The current number is compared with the number of the last processed data corresponding to the current partition key to determine whether the current data to be processed should be sent to the downstream end.

6. The data processing method according to claim 5, characterized in that, After obtaining the current partition key and current number of the data to be processed, the following steps are also included: Create a local cache, which is used to store the latest number corresponding to each of the partition keys; Retrieve the number of the last processed partition key from the local cache; Accordingly, after the current data to be processed is sent to the downstream end, the number of the current partition key in the local cache is updated to the current number.

7. A data processing apparatus, characterized in that, include: The change information acquisition module is used to acquire change information recorded in the binary log. The binary log records all change events to the database in binary format; The generation module is used to generate a corresponding number and partition key for the change event based on the change information; The number is used to indicate the sequence of change events; The partition key is used to determine whether they belong to the same business entity; The grouping and distribution module is used to group the change events according to the partition key, and send the data of the latest generated change event in each group to the downstream end according to the number. The generation module includes: An event location acquisition unit is used to acquire the event location from the change information recorded in the binary log; the event location is the offset of the change event in the binary log; The numbering calculation unit is used to obtain the file name of the binary log, and calculate the version number corresponding to the change event based on the file name and the event position, and use it as the number.

8. A data processing device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 6.

Images