Data processing method and device, electronic equipment and storage medium
By splitting and merging the event sequence of the source database in the ETL process and deleting redundant operation events, the problems of high data latency and high resource consumption in the ETL process are solved, and real-time updates and efficient data writing of the target database are achieved.
Patent Information
- Application Number
- CN202510795935.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-13
- Publication Date
- 2025-09-23
AI Technical Summary
In the ETL process, existing technologies have problems with high data latency and high resource consumption, resulting in the target database being unable to be updated in real time and low efficiency.
Through data change capture, the original event sequence of the source database is obtained, divided into multiple event subsequences, and a merge processing flow is performed to delete redundant operation events. The target event sequence is generated and synchronized to the target database. The redundant operation events are deleted during the merge processing flow, and only important operation events are retained to achieve real-time synchronization and reduce resource consumption.
It achieves real-time data updates of the target database, reduces data delays, and reduces resource consumption and improves data writing efficiency by merging processing flows.
Smart Images
Figure CN120687515A_ABST
Abstract
Description
Technical Field
[0001] The present invention mainly relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] In the ETL (extraction, transformation, loading) process, data is usually extracted from the source database periodically, cleansed and transformed, and then loaded into the target database. Although this can reduce the real-time impact on the source database, since data extraction is performed periodically, it cannot update data in the target database in real time, resulting in high latency. In addition, the amount of data loaded each time is too large, resulting in low efficiency. Summary of the Invention
[0003] The purpose of the present invention is to provide a data processing method, device, electronic device and storage medium, which can synchronize the pending operation events of the target database in real time during the ETL process, reduce data delay, reduce resource consumption by merging processing processes, and improve data writing efficiency.
[0004] In a first aspect, the present application provides a data processing method, comprising:
[0005] Acquire an original event sequence of a source database through data change capture, and divide the original event sequence into multiple event subsequences, each of which includes multiple operation events generated in the source database, and each operation event has corresponding attribute information, including an operation type and a key value of the operation event;
[0006] A merging process is performed on each event subsequence, and multiple event subsequences after the merging process are combined to obtain a target event sequence. The target event sequence is synchronized to a target database so that the target database updates data according to the target event sequence. The merging process includes:
[0007] Determine, in the operation subsequence, a plurality of associated sequences representing mappings to the same key value, and perform the following steps for each associated sequence:
[0008] Determine the first operation event and the last operation event in the association sequence according to the event arrangement order of the association sequence;
[0009] According to the operation type of the first operation event and the operation type of the last operation, redundant operation events are deleted in the associated sequence to update the associated sequence.
[0010] In some embodiments, deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes:
[0011] Determining whether the number of operation events in the associated sequence is less than or equal to one;
[0012] If so, do not process the associated sequence;
[0013] Otherwise, redundant operation events are deleted from the associated sequence according to the operation type of the first operation event and the operation type of the last operation event.
[0014] In some embodiments, deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes:
[0015] In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being an insert operation, other operation events except the first operation event and the last operation event are deleted from the associated sequence.
[0016] In some embodiments, deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes:
[0017] In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being a delete operation, other operation events except the first operation event are deleted in the associated sequence.
[0018] In some embodiments, deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes:
[0019] In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being a delete operation, all operation events in the associated sequence are deleted.
[0020] In some embodiments, deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes:
[0021] In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being an insert operation, other operation events except the last operation event are deleted from the associated sequence.
[0022] In some embodiments, the merging process further includes:
[0023] According to the event arrangement order of the event subsequence, multiple first operation events representing the first appearance of the key value are determined in the event subsequence, and the multiple first operation events are read in sequence until the reading is completed to obtain a first set, wherein each time the first operation event is read, the following steps are performed: determining whether the operation type of the first operation event currently read is an insert operation; if so, deleting all operation events mapped to the key value corresponding to the first operation event currently read in the event subsequence; otherwise, not processing;
[0024] According to the reverse order of the event arrangement order of the event subsequence, multiple second operation events representing the first appearance of the key value are determined in the event subsequence, and the multiple second operation events are read in sequence until the reading is completed to obtain a second set, wherein each time the second operation event is read, the following steps are performed: determining whether the operation type of the second operation event currently read is an insert operation; if so, not processing; otherwise, deleting all operation events mapped to the key value corresponding to the second operation event currently read in the event subsequence;
[0025] The first set and the second set are concatenated to generate an operation event sequence that can be executed in parallel and corresponds to the event subsequence.
[0026] In a second aspect, the present application provides a data processing device, comprising:
[0027] a sequence segmentation module, configured to obtain an original event sequence of a source database through data change capture, and segment the original event sequence into a plurality of event subsequences, wherein the event subsequences include a plurality of operation events generated in the source database, and each operation event has corresponding attribute information, wherein the attribute information includes an operation type and a key value of the operation event;
[0028] a merging processing module, configured to perform a merging processing flow on each event subsequence, combine multiple event subsequences that have undergone the merging processing flow to obtain a target event sequence, and synchronize the target event sequence to a target database, so that the target database performs data update according to the target event sequence;
[0029] Wherein, in the merging process, the merging processing module is used to:
[0030] Determine, in the operation subsequence, a plurality of associated sequences representing mappings to the same key value, and perform the following steps for each associated sequence:
[0031] Determine the first operation event and the last operation event in the association sequence according to the event arrangement order of the association sequence;
[0032] According to the operation type of the first operation event and the operation type of the last operation, redundant operation events are deleted in the associated sequence to update the associated sequence.
[0033] In a third aspect, the present application provides an electronic device comprising: one or more processors; and one or more memories coupled to the one or more processors and storing instructions thereon, wherein when the instructions are executed individually or collectively by the one or more processors, the electronic device executes a method according to any one of the first aspects.
[0034] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing machine-executable instructions, which, when executed by one or more processors of a machine, cause the machine to perform any one of the methods of the first aspect.
[0035] Compared with the prior art, the present invention has the following advantages:
[0036] The present application provides a data processing method, device, electronic device and storage medium. First, the original event sequence of the source database is obtained through data change capture, and then the original event sequence is divided into multiple event subsequences. Each event subsequence is combined after a merge processing process to obtain a target event sequence. The target event sequence is synchronized to the target database so that the target database updates data according to the target event sequence. Since the operation events occurring in the source database can be synchronized to the target database in real time, data delay can be reduced. Moreover, in the merge processing process, redundant operation events are deleted in each associated sequence that represents the same key value mapping, that is, only important operation events in the associated sequence are retained. This can reduce the number of operation events that need to be executed after the target event sequence is synchronized to the target database, reduce resource consumption, and make the operation events mapped to the same key value correspond to one target subsequence, thereby effectively improving data writing efficiency.
[0037] Furthermore, in the merge processing flow, the event subsequences can be read separately according to the order of event arrangement and its reverse order to obtain the first set and the second set. In the first set or the second set, since the change in the relative order of the operation events will not affect the final execution result, the operation events in the first set or the second set can be executed in parallel, thereby further improving efficiency.
[0038] It should be understood that the invention summary is not intended to identify the key or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings are included to provide a further understanding of the present application. They are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present application and, together with this specification, serve to explain the principles of the present application. In the accompanying drawings:
[0040] Figure 1 This is a flowchart of a data processing method exemplarily provided in this application;
[0041] Figure 2 is a schematic diagram of a method for updating an association sequence exemplarily provided in this application;
[0042] Figure 3 This is a schematic diagram of a merge processing flow provided by this application as an example;
[0043] Figure 4 is a schematic diagram of a data processing device exemplarily provided in this application;
[0044] Figure 5 This is a schematic diagram of an electronic device exemplarily provided in this application. DETAILED DESCRIPTION
[0045] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that the description of these embodiments is merely for illustrative purposes and helps those skilled in the art to understand and implement the present disclosure without placing any restriction on the scope of the present disclosure. The disclosure described herein can be implemented in a manner different from that described below.
[0046] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
[0047] References in this disclosure to "one embodiment," "an embodiment," "an exemplary embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in conjunction with an exemplary embodiment, whether or not explicitly described, those skilled in the art will recognize that such feature, structure, or characteristic may be combined with other embodiments.
[0048] It should be understood that although the terms "first" and "second" and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. The term "and / or" as used herein includes any and all combinations of one or more of the listed terms.
[0049] The terms used herein are intended only to describe specific embodiments and are not intended to limit exemplary embodiments. As used herein, the singular forms "a," "an," and "the" also include the plural forms, unless the context clearly indicates otherwise. As used herein, "a group of elements" or "a set of elements" is intended to include one or more elements. It should also be understood that the terms "comprise," "include," "have," "have," "include," and / or "comprising," when used herein, specify the presence of the features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.
[0050] As used in this application, the term "circuitry" may refer to one or more or all of the following:
[0051] (a) Hardware circuit implementation only (e.g., implementation only in analog and / or digital circuits)
[0052] (b) a combination of hardware circuitry and software, such as (where applicable):
[0053] (i) a combination of analog and / or digital hardware circuitry and software / firmware; and
[0054] (ii) any portion of a hardware processor (including a digital signal processor) with software, software, and memory that work together to enable a device such as a mobile phone or server to perform various functions, and
[0055] (c) Hardware circuits and / or processors, such as a microprocessor or portion of a microprocessor, that require software (e.g., firmware) to operate, but where software is not required for operation, the software may not be present.
[0056] This definition of "circuitry" applies to all uses of this term in this application, including in any claims. As another example, as used in this application, the term "circuitry" also includes merely a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware implementation. The term "circuitry" also includes, for example, if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing network device.
[0057] Figure 1 1 is a flow chart of a data processing method 100 provided by this application. Figure 1 , the data processing method 100 includes steps S11 and S12, wherein:
[0058] S11, obtain the original event sequence of the source database through data change capture, and divide the original event sequence into multiple event subsequences.
[0059] The original operation sequence can be a sequence of insert (INSERT), delete (DELETE), and update (UPDATE) operations captured in real time by monitoring the source database over a specified time period. Since each "update" operation can be split into a corresponding "delete" operation and an "insert" operation, the original operation sequence can also be converted into a sequence consisting only of insert and delete operations. The order of the operation events in the original operation sequence is consistent with the order in which the data is changed in the source database.
[0060] In step S11, after obtaining the original event sequence, an interval of any length can be extracted from the original event sequence, and each operation event corresponding to the interval is considered as an event subsequence. Multiple event subsequences can be continuous, and the order of the multiple event subsequences is consistent with the order of the extracted intervals.
[0061] An event subsequence includes multiple operation events generated within the source database, and each operation event has corresponding attribute information, including the operation type and key value of the operation event. The operation type can include the insert, delete, and update operations mentioned above, or it can include only insert and delete operations. This embodiment uses the latter as an example. The key value can be the key value corresponding to a primary key in the source database for the operation event.
[0062] For example, to simplify the description, each operation event of the operation type of insert operation is recorded as I, and each operation event of the operation type of delete operation is recorded as D. Assume that an event subsequence is: M1 = I1, I2, D3, ..., D m, where m is the number of operation events contained in the event subsequence.
[0063] S12, performing a merging process on each event subsequence, combining multiple event subsequences after the merging process to obtain a target event sequence, and synchronizing the target event sequence to a target database so that the target database updates data according to the target event sequence.
[0064] In step S12, the multiple event subsequences after the merging process may be reassembled according to the arrangement order of the intercepted intervals to obtain a target event sequence.
[0065] Based on the above approach, the operation events in the target event sequence are executed in sequence in the target database, so that new data is loaded into the target database.
[0066] For example, assume that an event subsequence after the merge process is M2 = I1, I2, D3, ..., D n , where n is the number of operation events retained in the event subsequence after the merge processing flow, and n is less than or equal to m. Therefore, compared with synchronizing the original event sequence to the target database, the above method can reduce the number of operation events that need to be executed, thereby reducing resource consumption.
[0067] In some embodiments, the merge processing flow may include: S121, determining multiple association sequences that represent mappings to the same key value in the operation subsequence. In other words, each association sequence corresponds to a single key value. The association sequence can be a sequence of operation events with the key value that are sequentially queried in the operation subsequence according to the order of events in the operation subsequence. For example, assuming that the association sequence N0=I1,I 2… D 10 , where I1,I 2… D 10 With the same key value.
[0068] In S121, after determining the associated sequences in the operation subsequence, the following steps S122 may be performed for each associated sequence:
[0069] S1221: Determine the first operation event and the last operation event in the associated sequence according to the order in which the events in the associated sequence are arranged.
[0070] Exemplarily, it is assumed that the associated sequence N=I1, I2, wherein I1 is confirmed as the first operation event and I2 is confirmed as the last operation event.
[0071] S1222: Delete redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation, so as to update the associated sequence.
[0072] In some embodiments, in step S1222, redundant operation events are deleted within the associated sequence based on the operation type of the first operation event and the operation type of the last operation, including: determining whether the number of operation events of the associated sequence is less than or equal to one; if so, not processing the associated sequence; otherwise, deleting redundant operation events within the associated sequence based on the operation type of the first operation event and the operation type of the last operation.
[0073] In other words, if the number of operation events in the associated sequence is less than or equal to one, the first operation event is the same as the last operation event, and the current associated sequence is not processed.
[0074] In some embodiments, in step S1222, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, including:
[0075] S12221: In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being an insert operation, delete other operation events except the first operation event and the last operation event in the associated sequence.
[0076] For example, assume that the associated sequence N1=D1,I 2… I 10 , where D1 is the first operation event, I 10 If it is the last operation event, then delete D1 and I in the associated sequence N1. 10 Other operation events except D1, I 10 .
[0077] In some embodiments, in step S1222, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, including:
[0078] S12222: In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being a delete operation, delete the other operation events except the first operation event in the associated sequence.
[0079] For example, assume that the associated sequence N2 = D1, I 2… D 10 , where D1 is the first operation event, D 10 If D1 is the last operation event, other operation events except D1 are deleted from the associated sequence N2, that is, only D1 is retained.
[0080] In some embodiments, in step S1222, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, including:
[0081] S12223: In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being a delete operation, delete all operation events in the associated sequence.
[0082] For example, assume that the associated sequence N3 = I1, I 2… D 10 , where I1 is the first operation event, D 10 If it is the last operation event, all operation events in the associated sequence N3 are deleted, that is, no operation event is retained.
[0083] In some embodiments, in step S1222, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, including:
[0084] S12224: In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being an insert operation, delete the other operation events except the last operation event in the associated sequence.
[0085] For example, assume that the associated sequence N1=I1,I 2… I 10 , where I1 is the first operation event, I 10 If it is the last operation event, then delete I in the associated sequence N1. 10 Other operation events except I 10 .
[0086] Figure 2 2 is a schematic diagram of a method 200 for updating an association sequence provided by the present application. The method 200 may be used to update the association sequence in step S121. For example, Figure 2, the method 200 may start from step S21, in which it may be determined whether the operation type of the first operation event is a deletion operation or an insertion operation. If the operation type of the first operation event is a deletion operation, the method 200 may proceed to step S22, and further determine whether the operation type of the last operation event is a deletion operation or an insertion operation. If the operation type of the last operation event is a deletion operation, the method 200 may proceed to step S12222, and delete other operation events except the first operation event. If the operation type of the last operation is an insertion operation, the method 200 may proceed to step S12221, and delete other operation events except the first operation event and the last operation event. If the operation type of the first operation event is an insertion operation, the method 200 may proceed to step S23, and further determine whether the operation type of the last operation event is a deletion operation or an insertion operation. If the operation type of the last operation event is a deletion operation, the method 200 may proceed to step S12223, and delete all operation events. If the operation type of the last operation event is an insertion operation, the method 200 may proceed to step S12224, and delete other operation events except the last operation event.
[0087] Based on the above method, in the merge processing flow, redundant operation events are deleted in each associated sequence that represents the mapping of the same key value, that is, only important operation events in the associated sequence are retained. This can reduce the number of operation events that need to be executed after the target event sequence is synchronized to the target database, reduce resource consumption, and make the operation events mapped to the same key value correspond to a target subsequence, thereby effectively improving data writing efficiency.
[0088] Figure 3 This is a schematic diagram of a merge processing flow 300 provided by this application. Figure 3 In some embodiments, the merging process further includes:
[0089] S31: According to the order of the events in the event subsequence, determine the first operation events that represent the first appearance of multiple key values in the event subsequence, read the multiple first operation events in sequence until the reading is completed, and obtain the first set. Wherein, each time the first operation event is read, the following is executed starting from S311: S311: Determine whether the operation type of the first operation event currently read is an insert operation. If so, execute S312 and delete all operation events mapped to the key value corresponding to the first operation event currently read in the event subsequence; otherwise, execute S313 and do nothing.
[0090] For example, assume an event subsequence M3 = I1, D2, D3, D4, I5, where I1, D3, and I5 have the same key value key1, and D2 and D4 have the same key value key2. Then I1 and D2 are the first operation events of key1 and key2, respectively. Since I1 is an insert operation and D2 is a delete operation, I1, I3, and D5 are deleted in the event subsequence M3, while D2 and D4 are retained. That is, the first set includes D2 and D4.
[0091] S32, in reverse order of the event arrangement order of the event subsequence, determine multiple second operation events representing the first appearance of the key value in the event subsequence, read the multiple second operation events in sequence until the reading is completed, and obtain a second set. Wherein, each reading is executed starting from S321: S321, determine whether the operation type of the second operation event currently read is an insert operation; if so, execute S322 without processing; otherwise, execute S323, and delete all operation events mapped to the key value corresponding to the second operation event currently read in the event subsequence.
[0092] For example, assume an event subsequence I1, D2, D3, D4, I5, where I1, D3, I5 have the same key value key1, and D2, D4 have the same key value key2. Then I5 and D4 are the second operation events of key1 and key2 respectively. Since I5 is an insert operation and D4 is a delete operation, D2 and D4 are deleted in the event subsequence M3, that is, the second set is I1, D3, I5.
[0093] S33: Concatenate the first set and the second set to generate an operation event sequence that can be executed in parallel and corresponds to the event subsequence.
[0094] Exemplarily, the second set is concatenated after the first set to generate a parallel executable operation event sequence M3′=D2, D4, I1, D3, I5 corresponding to the event subsequence M3.
[0095] After step S33, the operation events in the first set can be executed in parallel. After all the operation events in the first set have been executed, the operation events in the second set can be further executed in parallel without affecting the final execution result. For example, in the parallelizable operation event sequence M3', D2 and D4 can be executed in parallel, and then I1, D3, and I5 can be executed in parallel, thereby further improving efficiency.
[0096] Figure 4 4 is a schematic diagram of a data processing device 400 provided by the present application, comprising: a sequence segmentation module 401 and a merging processing module 402, wherein:
[0097] Sequence segmentation module 401 is used to obtain the original event sequence of the source database through data change capture, and segment the original event sequence into multiple event subsequences. The event subsequences include multiple operation events generated in the source database, and each operation event has corresponding attribute information, including the operation type and key value of the operation event;
[0098] A merge processing module 402 is configured to perform a merge processing process on each event subsequence, combine multiple event subsequences that have undergone the merge processing process to obtain a target event sequence, and synchronize the target event sequence to a target database so that the target database updates data according to the target event sequence.
[0099] In the merging process, the merging module 402 is used to:
[0100] In the operation subsequence, determine the association sequences of multiple representation mappings to the same key value, and perform the following steps for each association sequence:
[0101] According to the order of events in the associated sequence, determine the first operation event and the last operation event in the associated sequence;
[0102] According to the operation type of the first operation event and the operation type of the last operation, redundant operation events are deleted in the associated sequence to update the associated sequence.
[0103] In some embodiments, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, and the merging processing module 402 is used to:
[0104] Determine whether the number of operation events of the associated sequence is less than or equal to one;
[0105] If so, the associated sequence is not processed;
[0106] Otherwise, redundant operation events are deleted in the associated sequence based on the operation type of the first operation event and the operation type of the last operation event.
[0107] In some embodiments, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, and the merging processing module 402 is used to:
[0108] In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being an insert operation, other operation events except the first operation event and the last operation event are deleted in the associated sequence.
[0109] In some embodiments, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, and the merging processing module 402 is used to:
[0110] In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being a delete operation, other operation events except the first operation event are deleted in the associated sequence.
[0111] In some embodiments, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, and the merging processing module 402 is used to:
[0112] In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being a delete operation, all operation events in the associated sequence are deleted.
[0113] In some embodiments, based on the operation type of the first operation event and the operation type of the last operation event, redundant operation events are deleted in the associated sequence, and the merging processing module 402 is used to:
[0114] In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being an insert operation, other operation events except the last operation event are deleted in the associated sequence.
[0115] In some embodiments, during the merging process, the merging module 402 is further configured to:
[0116] According to the order of the events in the event subsequence, multiple first operation events representing the first appearance of the key value are determined in the event subsequence, and the multiple first operation events are read in sequence until the reading is completed to obtain a first set, wherein each time the first operation event is read, the following steps are performed: determining whether the operation type of the first operation event currently read is an insert operation; if so, deleting all operation events mapped to the key value corresponding to the first operation event currently read in the event subsequence; otherwise, not processing;
[0117] According to the reverse order of the event arrangement order of the event subsequence, multiple second operation events representing the first appearance of the key value are determined in the event subsequence, and the multiple second operation events are read in sequence until the reading is completed to obtain a second set, wherein each time the second operation event is read, the following steps are performed: determining whether the operation type of the second operation event currently read is an insert operation; if so, not processing; otherwise, deleting all operation events mapped to the key value corresponding to the second operation event currently read in the event subsequence;
[0118] The first set and the second set are concatenated to generate an operation event sequence that can be executed in parallel and corresponds to the event subsequence.
[0119] Further, if Figure 5 An exemplary embodiment of the present application also provides an electronic device 500, comprising one or more memories 501 and one or more processors 502, wherein the one or more memories 501 are coupled to the one or more processors 502 and store instructions thereon, and the instructions can be executed individually or collectively by the one or more processors 502, so that the electronic device 500 performs any method as in the first aspect.
[0120] It should be understood that the processor mentioned in the embodiments of the present application may be a CPU, or may be other general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0121] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory, dynamic random access memory, synchronous dynamic random access memory, double data rate synchronous dynamic random access memory, enhanced synchronous dynamic random access memory, synchronously linked dynamic random access memory, and direct memory bus random access memory.
[0122] The present application also provides a non-transitory computer-readable storage medium storing machine-executable instructions, wherein the computer-executable instructions can be executed by one or more processors of a machine. The machine may include the electronic device mentioned above, etc. When the computer-executable instructions are executed by the one or more processors, the machine performs any of the methods mentioned above.
[0123] A computer-readable storage medium may include a propagated data signal embodying computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination thereof. The computer-readable storage medium may be connected to an instruction execution system, device, or apparatus to communicate, propagate, or transmit the program for use. The program code on the computer-readable storage medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or any combination of the above.
[0124] The basic concepts have been described above. It will be apparent to those skilled in the art that the above disclosures are merely illustrative and do not constitute limitations on this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and amendments to this application. Such modifications, improvements, and amendments are suggested in this application and remain within the spirit and scope of the exemplary embodiments of this application.
[0125] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "one embodiment," "an embodiment," or "an alternative embodiment" mentioned twice or multiple times in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application may be appropriately combined.
[0126] Some aspects of the present application can be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software can be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs...), smart cards, and flash memory devices (e.g., cards, sticks, key drives...).
[0127] A computer-readable medium may include a propagated data signal embodying computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination thereof. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium that can be connected to an instruction execution system, apparatus, or device to communicate, propagate, or transmit the program for use. The program code on the computer-readable medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or any combination of the above.
[0128] Similarly, it should be noted that, in order to simplify the presentation of this application and thus facilitate understanding of one or more embodiments of the invention, the foregoing descriptions of the embodiments of this application sometimes combine multiple features into a single embodiment, figure, or description thereof. However, this disclosure method does not mean that the subject matter of this application requires more features than those recited in the claims. In fact, an embodiment may have fewer features than all of the features of a single embodiment disclosed above.
[0129] In some embodiments, numbers are used to describe the quantity of components and attributes. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise stated, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the description and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of the present application are approximate values, in specific embodiments, the settings of such numerical values are as accurate as possible within the feasible range.
[0130] Although the present application has been described with reference to the current specific embodiments, ordinary technicians in this technical field should recognize that the above embodiments are only used to illustrate the present application, and various equivalent changes or substitutions can be made without departing from the spirit of the present application. Therefore, as long as the changes and modifications to the above embodiments are within the scope of the essential spirit of the present application, they will fall within the scope of the claims of the present application.
Claims
1. A data processing method, characterized in that: include: Acquire an original event sequence of a source database through data change capture, and divide the original event sequence into multiple event subsequences, each of which includes multiple operation events generated in the source database, and each operation event has corresponding attribute information, including an operation type and a key value of the operation event; A merging process is performed on each event subsequence, and multiple event subsequences after the merging process are combined to obtain a target event sequence. The target event sequence is synchronized to a target database so that the target database updates data according to the target event sequence. The merging process includes: Determine, in the operation subsequence, a plurality of associated sequences representing mappings to the same key value, and perform the following steps for each associated sequence: Determine the first operation event and the last operation event in the association sequence according to the event arrangement order of the association sequence; According to the operation type of the first operation event and the operation type of the last operation, redundant operation events are deleted in the associated sequence to update the associated sequence.
2. The method according to claim 1, wherein The deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes: Determining whether the number of operation events in the associated sequence is less than or equal to one; If so, do not process the associated sequence; Otherwise, redundant operation events are deleted from the associated sequence according to the operation type of the first operation event and the operation type of the last operation event.
3. The method according to claim 1 or 2, wherein: The deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes: In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being an insert operation, other operation events except the first operation event and the last operation event are deleted from the associated sequence.
4. The method according to claim 1 or 2, wherein: The deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes: In response to the operation type of the first operation event being a delete operation and the operation type of the last operation event being a delete operation, other operation events except the first operation event are deleted in the associated sequence.
5. The method according to claim 1 or 2, wherein: The deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes: In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being a delete operation, all operation events in the associated sequence are deleted.
6. The method according to claim 1 or 2, wherein: The deleting redundant operation events in the associated sequence according to the operation type of the first operation event and the operation type of the last operation event includes: In response to the operation type of the first operation event being an insert operation and the operation type of the last operation event being an insert operation, other operation events except the last operation event are deleted from the associated sequence.
7. The method according to claim 1 or 2, wherein: The merging process also includes: According to the event arrangement order of the event subsequence, multiple first operation events representing the first appearance of the key value are determined in the event subsequence, and the multiple first operation events are read in sequence until the reading is completed to obtain a first set, wherein each time the first operation event is read, the following steps are performed: determining whether the operation type of the first operation event currently read is an insert operation; if so, deleting all operation events mapped to the key value corresponding to the first operation event currently read in the event subsequence; otherwise, not processing; According to the reverse order of the event arrangement order of the event subsequence, multiple second operation events representing the first appearance of the key value are determined in the event subsequence, and the multiple second operation events are read in sequence until the reading is completed to obtain a second set, wherein each time the second operation event is read, the following steps are performed: determining whether the operation type of the second operation event currently read is an insert operation; if so, not processing; otherwise, deleting all operation events mapped to the key value corresponding to the second operation event currently read in the event subsequence; The first set and the second set are concatenated to generate an operation event sequence that can be executed in parallel and corresponds to the event subsequence.
8. A data processing device, characterized in that: include: a sequence segmentation module, configured to obtain an original event sequence of a source database through data change capture, and segment the original event sequence into a plurality of event subsequences, wherein the event subsequences include a plurality of operation events generated in the source database, and each operation event has corresponding attribute information, wherein the attribute information includes an operation type and a key value of the operation event; a merging processing module, configured to perform a merging processing flow on each event subsequence, combine multiple event subsequences that have undergone the merging processing flow to obtain a target event sequence, and synchronize the target event sequence to a target database, so that the target database performs data update according to the target event sequence; Wherein, in the merging process, the merging processing module is used to: Determine, in the operation subsequence, a plurality of associated sequences representing mappings to the same key value, and perform the following steps for each associated sequence: Determine the first operation event and the last operation event in the association sequence according to the event arrangement order of the association sequence; According to the operation type of the first operation event and the operation type of the last operation, redundant operation events are deleted in the associated sequence to update the associated sequence.
9. An electronic device comprising: one or more processors; as well as One or more memories coupled to the one or more processors and storing thereon instructions, which, when executed individually or collectively by the one or more processors, cause the electronic device to perform the method according to any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium storing machine-executable instructions, which, when executed by one or more processors of a machine, cause the machine to perform the method of any one of claims 1-7.