An incremental data synchronization method, apparatus, computer device, and storage medium

Abstract

This application, belonging to the fields of artificial intelligence and big data, relates to an incremental data synchronization method. The method includes acquiring incremental data and operation logs within the current synchronization time period; parsing the operation logs to extract physically deleted data and non-standard operation data; merging the non-standard operation data with the incremental data to obtain first merged data; merging the first merged data with the deleted data to obtain second merged data; and synchronizing the second merged data to a data warehouse. This application also provides an incremental data synchronization device, computer equipment, and storage medium. Furthermore, this application relates to blockchain technology, where incremental data and operation logs can be stored in the blockchain. This application effectively ensures data consistency between the synchronizing parties.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and big data technology, and in particular to incremental data synchronization methods, apparatus, computer equipment and storage media. Background Technology

[0002] As a data source for intelligent business analysis, data warehouses need to access data from various enterprise application systems and ensure that the data in the data warehouse is consistent with the data in the application systems.

[0003] Currently, data warehouses generally use incremental synchronization for data entry. However, existing incremental data synchronization methods cannot capture non-standard addition and modification records that do not update timestamps when adding or modifying data, as well as records physically deleted from the business database. This leads to inconsistencies between the data warehouse and upstream databases (such as business databases), affecting data traceability and verification. Although logical deletion can be used to replace physical deletion, as the business database runs longer, it accumulates excessive redundant data, resulting in high operating costs and slow speeds, making it counterproductive. Summary of the Invention

[0004] The purpose of this application is to provide an incremental data synchronization method, apparatus, computer device, and storage medium, the main objective of which is to ensure data consistency between the synchronizing parties.

[0005] To address the aforementioned technical problems, this application provides an incremental data synchronization method, employing the following technical solution:

[0006] Retrieve incremental data and operation logs within the current synchronization time period;

[0007] The operation log is parsed to extract physical deletion data and non-standard operation data.

[0008] The non-standard operation data is merged with the incremental data to obtain the first merged data. Then, the first merged data is merged with the deleted data to obtain the second merged data.

[0009] The second merged data is synchronized to the data warehouse.

[0010] Furthermore, the step of extracting physical deletion data and non-standard operation data from the operation log includes:

[0011] Extract the operation identifier of the first primary key from the operation log, wherein the operation log includes multiple first primary keys;

[0012] The operation type of the first primary key is determined based on the operation identifier;

[0013] All primary keys with the operation type of deletion are merged to obtain the physically deleted data, and non-standard operation data are identified from all primary keys with the operation types of addition and modification.

[0014] Furthermore, the step of determining non-standard operation data from all first primary keys whose operation type is add operation and modify operation includes:

[0015] Extract the primary keys that meet the non-standard operation conditions from all primary keys whose operation type is add or modify, and use them as secondary primary keys;

[0016] By combining all the second primary keys, we obtain the non-standard operation data.

[0017] Furthermore, prior to the step of merging the non-standard operation data with the incremental data, the method further includes:

[0018] Extract the second primary key from the non-standard operation data and save the second primary key to the non-standard operation table;

[0019] The step of merging the non-standard operation data with the incremental data includes:

[0020] Extract the second primary key of the non-standard operation type from the non-standard operation table, and merge the second primary key into the incremental data.

[0021] Furthermore, prior to the step of merging the first merged data with the deleted data, the method further includes:

[0022] Save the first primary key of each operation type of physical deletion in the physical deletion data to the physical deletion table;

[0023] The step of merging the first merged data with the deleted data includes:

[0024] Extract the first primary key of the operation type physical deletion from the physical deletion table, and determine the redundant data in the first merged data based on the first primary key extracted from the physical deletion table;

[0025] Delete the redundant data in the first merged data.

[0026] Furthermore, after the step of synchronizing the second merged data to the data warehouse, the method further includes:

[0027] Delete the physical deletion table;

[0028] Alternatively, delete all instances of the first primary key in the physical deletion table.

[0029] Furthermore, the step of synchronizing the second merged data to the data warehouse includes:

[0030] Obtain historical data from the data warehouse;

[0031] Identify identical data in the second merged data and the historical data as duplicate data;

[0032] Delete the duplicate data in the second merged data to obtain the target data;

[0033] The target data is synchronously updated to the data warehouse.

[0034] To address the aforementioned technical problems, this application also provides an incremental data synchronization device, which employs the following technical solution:

[0035] The acquisition module is used to acquire incremental data and operation logs within the current synchronization time period;

[0036] The extraction module is used to parse the operation log to extract physical deletion data and non-standard operation data from the operation log;

[0037] The merging module is used to merge the non-standard operation data with the incremental data to obtain the first merged data, and then merge the first merged data with the deleted data to obtain the second merged data.

[0038] The synchronization module is used to synchronize the second merged data to the data warehouse.

[0039] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0040] The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the incremental data synchronization method described above.

[0041] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0042] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the incremental data synchronization method described above.

[0043] Compared with existing technologies, the embodiments of this application have the following main advantages: Incremental data and operation logs within the current synchronization time period are obtained; the operation logs are parsed to extract physically deleted data and non-standard operation data; the non-standard operation data is merged with the incremental data to obtain first merged data; the first merged data is then merged with the deleted data to obtain second merged data; and the second merged data is synchronized to the data warehouse. This application solves the problem in existing technologies where data anomalies or omissions occur due to records where timestamps are not modified during data addition or modification, and / or physically deleted data. This ensures the consistency of the data between the two parties. Attached Figure Description

[0044] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0046] Figure 2 This is a flowchart of an embodiment of the incremental data synchronization method according to this application;

[0047] Figure 3 This is a schematic diagram of one embodiment of the incremental data synchronization device according to this application;

[0048] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0050] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0051] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0052] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0053] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0054] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0055] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0056] It should be noted that the incremental data synchronization method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the incremental data synchronization device is generally set in the server / terminal device.

[0057] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0058] Continue to refer to Figure 2 A flowchart of an embodiment of the incremental data synchronization method according to this application is shown. The incremental data synchronization method includes the following steps:

[0059] Step S201: Obtain incremental data and operation logs within the current synchronization time period.

[0060] In this embodiment, the incremental data synchronization method operates on an electronic device (e.g., Figure 1 The server / terminal device shown can obtain incremental data and operation logs within the current synchronization period via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future-developed wireless connection methods.

[0061] The aforementioned synchronization time period is the time period for data synchronization, which can be set by the operator. In some implementations, incremental data and operation logs from the previous current synchronization time period are synchronized. For example, if the current synchronization time period is 1 day and the current date is 2022-08-15, then incremental data and operation logs from the previous current synchronization time period (i.e., 2022-08-14) are synchronized.

[0062] Step S202: Parse the operation log to extract physical deletion data and non-standard operation data from the operation log.

[0063] In this embodiment, the aforementioned physically deleted data represents data that has been actually deleted and cannot be queried; the aforementioned non-standard operation data represents data for which the timestamp was not modified when data was added or modified.

[0064] In some implementations, after the aforementioned current synchronization time period is reached, the operation logs in the business database are synchronized to an intermediate database (such as the Kafka platform) through the GoldenGate platform. Then, the operation logs on the intermediate database are consumed, and the operation logs are parsed using a log parsing tool (such as the fn_dblog tool) to extract physical deletion data and non-standard operation data.

[0065] Step S203: Merge the non-standard operation data with the incremental data to obtain the first merged data, and then merge the first merged data with the deleted data to obtain the second merged data.

[0066] In this embodiment, non-standard operation data is merged with incremental data to overwrite and synchronize the non-standard data into the incremental data, resulting in first merged data. This solves the problem of data inconsistency caused by records where timestamps are not modified when adding or modifying data. At the same time, the first merged data is merged with the deleted data, and physically deleted data is removed from the first merged data to obtain second merged data. This solves the problem of incremental data after merging standard operation data due to physical deletion, further improving the consistency of data between the two parties.

[0067] In some implementations, non-standard operational data is merged with the incremental data using synchronization tools such as Sqoop or ETL tools.

[0068] Step S204: Synchronize the second merged data to the data warehouse.

[0069] In this embodiment, the second merged data is overwritten and synchronized to the data warehouse; for example, if the age of user A in the data warehouse is 20 years old, while the age of user A in the second merged data is 22 years old, after the second merged data is synchronized to the data warehouse, the age of user A in the data warehouse is modified to 22 years old.

[0070] This application solves the problem of data anomalies or omissions caused by records without timestamp modification during data addition or modification and / or physical data deletion in existing technologies, by extracting deleted and added / modified data from operation logs and confirming non-standard operation data through added / modified data and incremental data, and then merging the non-standard data with incremental data, and merging the merged incremental data with deleted data.

[0071] In some optional implementations of this embodiment, the step of extracting physical deletion data and non-standard operation data from the operation log includes:

[0072] Extract the operation identifier of the first primary key from the operation log, wherein the operation log includes multiple first primary keys;

[0073] The operation type of the first primary key is determined based on the operation identifier;

[0074] All primary keys with the operation type of deletion are merged to obtain the physically deleted data, and non-standard operation data are identified from all primary keys with the operation types of addition and modification.

[0075] In this embodiment, the operation log includes multiple primary keys; in some implementations, the operation log is displayed in the form of a table, and each row of data in the operation log table is a primary key.

[0076] The above operation types include deletion, addition, and modification operations, with different operation identifiers for each. For example, the operation identifier for deletion is D, the operation identifier for addition is I, and the operation identifier for modification is U. Once the operation identifier is obtained, the operation type can be determined based on the operation identifier.

[0077] In practical applications, the primary key in the operation log is divided into physically deleted data and non-standard operation data according to the different operation types. This is used in the subsequent step S203 to merge the physically deleted data, non-standard operation data and incremental data, thereby ensuring the consistency of data between the two synchronizing parties.

[0078] In some optional implementations of this embodiment, the step of determining non-standard operation data from all first primary keys whose operation type is add operation and modify operation includes:

[0079] Extract the primary keys that meet the non-standard operation conditions from all primary keys whose operation type is add or modify, and use them as secondary primary keys;

[0080] By combining all the second primary keys, we obtain the non-standard operation data.

[0081] In this embodiment, the aforementioned non-standard operation condition is the first primary key whose timestamp has not been modified in the operation type of add operation and modification operation. The first primary key that meets the non-standard operation condition is extracted separately and used as the second primary key. Finally, all the second primary keys are merged to obtain the non-standard operation data.

[0082] In some optional implementations of this embodiment, step S203, before the step of merging the non-standard operation data with the incremental data, further includes:

[0083] Extract the second primary key from the non-standard operation data and save the second primary key to the non-standard operation table;

[0084] In this embodiment, the second primary key is first stored in the non-standard operation table, which is a temporary table. The non-standard operation table is temporarily stored so that the second primary key can be retrieved from the non-standard operation table and merged with the incremental data, thereby improving the synchronization efficiency of the incremental data.

[0085] Step S203, the step of merging the non-standard operation data with the incremental data includes:

[0086] Extract the second primary key of the non-standard operation type from the non-standard operation table, and merge the second primary key into the incremental data.

[0087] In this embodiment, during the merging process, the timestamp and secondary primary key information are first extracted from the secondary primary key. The position of the secondary primary key in the incremental data is determined based on the timestamp of the secondary primary key. If the incremental data contains multiple timestamps that are the same as the timestamp of the secondary primary key, the position of the secondary primary key in the incremental data is determined based on the secondary primary key information. For example, if the secondary primary key information is "Name: Zhang San San", the position of "Name: Zhang San San" is matched from the incremental data to determine the position of the secondary primary key in the incremental data. After the position of the secondary primary key in the incremental data is determined, the secondary primary key is overwritten into the incremental data, thereby completing the merging of the secondary primary key and the incremental data.

[0088] Furthermore, after executing step S204, the non-standard operation table is deleted to free up system memory and reduce system load.

[0089] In some optional implementations of this embodiment, step S203, before the step of merging the first merged data with the deleted data, further includes:

[0090] Save the first primary key of each operation type of physical deletion in the physical deletion data to the physical deletion table;

[0091] In this embodiment, the first primary key, which is of the physical deletion type, is first saved to the physical deletion table. The physical deletion table is a temporary table and is temporarily stored so that the first primary key can be retrieved from the physical deletion table and merged with the incremental data, thereby improving the synchronization efficiency of the incremental data.

[0092] Step S203, the step of merging the first merged data with the deleted data includes:

[0093] Extract the first primary key of the operation type physical deletion from the physical deletion table, and determine the redundant data in the first merged data based on the first primary key extracted from the physical deletion table;

[0094] Delete the redundant data in the first merged data.

[0095] In this embodiment, after extracting the first primary key from the physical deletion table, the timestamp and first primary key information are extracted from the first primary key. The position of the first primary key in the incremental data is determined based on the timestamp of the first primary key. If the first merged data contains multiple timestamps with the same timestamp as the first primary key, the position of the first primary key in the first merged data is determined based on the first primary key information. For example, if the first primary key information is ID number: 441900XXXXXXXXXXX201, then the position of ID number: 441900XXXXXXXXXXX201 is matched in the first merged data to determine the position of the first primary key in the first merged data. After the position of the first primary key in the first merged data is determined, the data at the position of the first primary key in the first merged data is identified as redundant data, and the redundant data is deleted, thereby completing the merging of the first merged data and the deleted data.

[0096] In some optional implementations of this embodiment, step S204, after the step of synchronizing the second merged data to the data warehouse, further includes:

[0097] Delete the physical deletion table;

[0098] Alternatively, delete all primary keys in the physical deletion table.

[0099] In this embodiment, after step S204 is executed, if the physical deletion table is a temporary table, the physical deletion table can be deleted directly. If the physical deletion table is not a temporary table, all the first primary keys in the physical deletion table can be deleted to release system memory and reduce system load.

[0100] In some optional implementations of this embodiment, step S204, the step of synchronizing the second merged data to the data warehouse, includes:

[0101] Obtain historical data from the data warehouse;

[0102] Identify identical data in the second merged data and the historical data as duplicate data;

[0103] Delete the duplicate data in the second merged data to obtain the target data;

[0104] The target data is synchronously updated to the data warehouse.

[0105] In this embodiment, the data warehouse stores multiple historical data. While acquiring incremental data and operation logs within the current synchronization time period, the database identifier of the upstream database that generated the incremental data and operation logs is also acquired, so as to match the corresponding historical data from the data warehouse based on the database identifier.

[0106] In practical applications, since incremental data may contain duplicate data with historical data in the data warehouse, in order to reduce data redundancy, the same data between incremental data and historical data in the data warehouse tool is identified as duplicate data. After deleting the duplicate data in the second merged data, the remaining data in the second merged data is the target data. Then, the target data and historical data are merged to achieve the purpose of incremental data synchronization.

[0107] It should be emphasized that, to further ensure the privacy and security of the aforementioned incremental data and operation logs, the aforementioned incremental data and operation logs can also be stored in a node of a blockchain.

[0108] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0109] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0110] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0111] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0112] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0113] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of an incremental data synchronization device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0114] like Figure 3 As shown, the incremental data synchronization device 300 described in this embodiment includes: an acquisition module 301, an extraction module 302, a merging module 303, and a synchronization module 304. Wherein:

[0115] The acquisition module 301 is used to acquire incremental data and operation logs within the current synchronization time period.

[0116] Extraction module 302 is used to parse the operation log to extract physical deletion data and non-standard operation data from the operation log;

[0117] The merging module 303 is used to merge the non-standard operation data with the incremental data to obtain the first merged data, and then merge the first merged data with the deleted data to obtain the second merged data.

[0118] The synchronization module 304 is used to synchronize the second merged data to the data warehouse.

[0119] This application solves the problem of data anomalies or omissions caused by records without timestamp modification during data addition or modification and / or physical data deletion in existing technologies, by extracting deleted and added / modified data from operation logs and confirming non-standard operation data through added / modified data and incremental data, and then merging the non-standard data with incremental data, and merging the merged incremental data with deleted data.

[0120] In some optional implementations of this embodiment, the extraction module 302 includes an extraction submodule, a first determining submodule, and a second determining submodule. Wherein:

[0121] An extraction submodule is used to extract the operation identifier of the first primary key from the operation log, wherein the operation log includes multiple first primary keys;

[0122] The first determining submodule is used to determine the operation type of the first primary key based on the operation identifier;

[0123] The second determination submodule is used to merge all primary keys with the operation type of deletion to obtain the physically deleted data, and to determine the non-standard operation data from all primary keys with the operation types of addition and modification.

[0124] In some optional implementations of this embodiment, the data determination submodule includes an extraction unit and a merging unit. Wherein:

[0125] The extraction unit is used to extract the first primary key that meets the non-standard operation conditions from all first primary keys whose operation type is add operation and modify operation, and use it as the second primary key;

[0126] The merge unit is used to merge all the second primary keys to obtain non-standard operation data.

[0127] In some optional implementations of this embodiment, a first storage module is also included.

[0128] in:

[0129] The first storage module is used to extract the second primary key from the non-standard operation data and save the second primary key to the non-standard operation table;

[0130] The aforementioned merging module 303 includes a first merging submodule. Wherein:

[0131] The first merging submodule is used to extract the second primary key of the non-standard operation type from the non-standard operation table and merge the second primary key into the incremental data.

[0132] In some optional implementations of this embodiment, a second storage module is also included.

[0133] in:

[0134] The second storage module is used to save the first primary key of each operation type of physical deletion in the physical deletion data to the physical deletion table.

[0135] The aforementioned merging module 303 includes a third determining submodule and a first deleting submodule.

[0136] in:

[0137] The third determining submodule is used to extract the first primary key of the operation type physical deletion type from the physical deletion table, and determine the redundant data in the first merged data based on the first primary key extracted from the physical deletion table;

[0138] The first deletion submodule is used to delete the redundant data in the first merged data.

[0139] In some optional implementations of this embodiment, a deletion module is also included. Wherein:

[0140] The deletion module is used to delete the physical deletion table; or, to delete all of the first primary keys in the physical deletion table.

[0141] In some optional implementations of this embodiment, the synchronization module 304 includes an acquisition submodule, an identification submodule, a second deletion submodule, and a synchronization submodule. Wherein:

[0142] The acquisition submodule is used to acquire historical data from the data warehouse;

[0143] The identification submodule is used to identify identical data between the second merged data and the historical data as duplicate data.

[0144] The second deletion submodule is used to delete the duplicate data in the second merged data to obtain the target data;

[0145] The synchronization submodule is used to synchronize and update the target data to the data warehouse.

[0146] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0147] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0148] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0149] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for incremental data synchronization methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0150] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, such as executing computer-readable instructions for the incremental data synchronization method.

[0151] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0152] This application solves the problem of data anomalies or omissions caused by records without timestamp modification during data addition or modification and / or physical data deletion in existing technologies, by extracting deleted and added / modified data from operation logs and confirming non-standard operation data through added / modified data and incremental data, and then merging the non-standard data with incremental data, and merging the merged incremental data with deleted data.

[0153] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the incremental data synchronization method described above.

[0154] This application solves the problem of data anomalies or omissions caused by records without timestamp modification during data addition or modification and / or physical data deletion in existing technologies, by extracting deleted and added / modified data from operation logs and confirming non-standard operation data through added / modified data and incremental data, and then merging the non-standard data with incremental data, and merging the merged incremental data with deleted data.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0156] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. An incremental data synchronization method, characterized by, Includes the following steps: Retrieve incremental data and operation logs within the current synchronization time period; The operation log is parsed to extract physical deletion data and non-standard operation data. Extract the second primary key from the non-standard operation data and save the second primary key to the non-standard operation table; Save the first primary key of each operation type of physical deletion in the physically deleted data to the physical deletion table; The non-standard operation data is merged with the incremental data to obtain the first merged data. Then, the first merged data is merged with the deleted data to obtain the second merged data. Specifically, this includes: extracting the second primary key (operation type: non-standard operation) from the non-standard operation table; extracting the timestamp and second primary key information from the second primary key; determining the position of the second primary key in the incremental data based on the timestamp of the second primary key; if the incremental data contains multiple timestamps identical to the second primary key's timestamp, determining the position of the second primary key in the incremental data based on the second primary key information; and finally, determining the position of the second primary key in the incremental data after the position of the second primary key is determined. The second primary key is overwritten into the incremental data to obtain the first merged data. Then, the first primary key (operation type: physical deletion) is extracted from the physical deletion table. A timestamp and primary key information are extracted from the first primary key. The position of the first primary key in the incremental data is determined based on the timestamp. If the first merged data contains multiple timestamps identical to the first primary key, the position of the first primary key in the first merged data is determined based on the primary key information. The data at the position of the first primary key in the first merged data is identified as redundant data. The redundant data in the first merged data is then deleted to obtain the second merged data. The second merged data is synchronized to the data warehouse.

2. The incremental data synchronization method of claim 1, wherein, The steps for extracting physically deleted data and non-standard operation data from the operation log include: Extract the operation identifier of the first primary key from the operation log, wherein the operation log includes multiple first primary keys; The operation type of the first primary key is determined based on the operation identifier; All primary keys with the operation type of deletion are merged to obtain the physically deleted data, and non-standard operation data are identified from all primary keys with the operation types of addition and modification.

3. The incremental data synchronization method of claim 2, wherein, The steps for determining non-standard operation data from all first primary keys whose operation type is add or modify include: Extract the primary keys that meet the non-standard operation conditions from all primary keys whose operation type is add or modify, and use them as secondary primary keys; By combining all the second primary keys, we obtain the non-standard operation data.

4. The incremental data synchronization method of claim 2, wherein, Following the step of synchronizing the second merged data to the data warehouse, the method further includes: Delete the physical deletion table; Alternatively, delete all instances of the first primary key in the physical deletion table.

5. The incremental data synchronization method according to any one of claims 1 to 4, characterized in that, The step of synchronizing the second merged data to the data warehouse includes: Obtain historical data from the data warehouse; Identify identical data in the second merged data and the historical data as duplicate data; Delete the duplicate data in the second merged data to obtain the target data; The target data is synchronously updated to the data warehouse.

6. An incremental data synchronization device, characterized in that, include: The acquisition module is used to acquire incremental data and operation logs within the current synchronization time period; The extraction module is used to parse the operation log to extract physical deletion data and non-standard operation data from the operation log; The first storage module is used to extract the second primary key from the non-standard operation data and save the second primary key to the non-standard operation table; The second storage module is used to save the first primary key of each operation type of physical deletion in the physical deletion data to the physical deletion table; The merging module is used to merge the non-standard operation data with the incremental data to obtain first merged data. Then, it merges the first merged data with the deleted data to obtain second merged data. Specifically, this includes: extracting a second primary key from the non-standard operation table, extracting a timestamp and second primary key information from the second primary key, determining the position of the second primary key in the incremental data based on the timestamp of the second primary key, and if the incremental data contains multiple timestamps identical to the second primary key's timestamp, determining the position of the second primary key in the incremental data based on the second primary key information. After confirmation, the second primary key is overwritten into the incremental data to obtain the first merged data. Then, the first primary key of the operation type physical deletion is extracted from the physical deletion table. The timestamp and first primary key information are extracted from the first primary key. The position of the first primary key in the incremental data is determined based on the timestamp of the first primary key. If the first merged data contains multiple timestamps with the same timestamp as the first primary key, the position of the first primary key in the first merged data is determined based on the first primary key information. The data at the position of the first primary key in the first merged data is identified as redundant data. The redundant data in the first merged data is deleted to obtain the second merged data. The synchronization module is used to synchronize the second merged data to the data warehouse.

7. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the incremental data synchronization method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the incremental data synchronization method as described in any one of claims 1 to 5.

Images