Incremental database table continuous delivery method and device based on log, equipment and medium

Abstract

The invention provides a log-based incremental database table continuous delivery method, device and equipment and a medium, and belongs to the technical field of database synchronism, the method comprises the following steps: regularly extracting incremental data in a preset time period from a data log file of an information system according to a set period and compressing the incremental data to obtain an incremental data file and storing the incremental data file; performing md5 verification on the stored incremental data file, and uploading the incremental data file and a verification file to a server file system; the client system regularly downloads the incremental data file and the verification file from the server file system according to a set period, and verifies the incremental data file by using the verification file; and the client system performs data updating operation on the target database after grouping the incremental data files passing the verification, and sends an updating result to the user. According to the invention, the data synchronization efficiency is improved, the data transmission and processing cost is reduced, the manual intervention is reduced, and the stability and reliability of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of database synchronization, and particularly relates to a method, device, equipment and medium for continuous delivery of incremental database tables based on logs. Background Art

[0002] With the development of big data and cloud computing, various information systems will generate a large amount of data. As the core component of the information system, the database undertakes the task of data storage, and the update and synchronization of data in the database become the key to ensuring the business continuity and data consistency of the information system. Traditional data synchronization methods are difficult to meet the requirements of efficient and real-time data synchronization due to problems such as large data volume, low transmission efficiency, and high resource consumption. Incremental data synchronization, by only transmitting the data that has changed since the last synchronization, significantly improves the efficiency of data synchronization and has become a common method for current database data synchronization.

[0003] However, the existing incremental data synchronization methods still have the following problems in actual use: First, the data formats are not unified, making the data exchange between different systems complex and increasing the difficulty and cost of synchronization. Second, the verification mechanism is imperfect, resulting in the risk of data being tampered with or lost during transmission, thereby affecting the accuracy and reliability of the data. In addition, the low degree of automation is also a major problem in incremental data synchronization, resulting in the need for manual intervention in monitoring and adjustment, reducing the overall work efficiency and flexibility. Therefore, how to achieve efficient and reliable incremental data synchronization while ensuring data security has become an urgent problem to be solved. Summary of the Invention

[0004] In a first aspect, an embodiment of the present application provides a method for continuous delivery of incremental database tables based on logs, including the following steps: S1. Extract the incremental data within a preset time period from the data log file of the information system at regular intervals according to a set period, compress it, obtain an incremental data file, and store it; S2. Perform md5 verification on the stored incremental data file, and upload the incremental data file and the verification file to the server file system; S3. The client system downloads the incremental data file and the verification file from the server file system at regular intervals according to a set period, and uses the verification file to verify the incremental data file; S4. After decompressing and grouping the incremental data file that passes the verification, the client system performs a data update operation on the target database and sends the update result to the user.

[0005] Further, the specific steps of step S1 are as follows: S11. Preset the collection period, and set the collection time points within the collection period, with two adjacent collection time points serving as the collection period; S12. Set a timed collection task for each collection time point within each collection period; S13. In response to the collection task, obtain the data change records within the collection period to which the current collection time point belongs from the corresponding data log file of the source database of the information system; S14. After formatting and compressing the obtained data change records, obtain the incremental data file, and then name the incremental data file and store it in the preset file structure.

[0006] Furthermore, the specific steps of step S14 are as follows: S141. Obtain the source database type in advance, determine the fields in the source database as source fields, and specify the corresponding target fields for each source field in the target database; S142. Set conversion rules according to the data types of the source fields and the target fields, and add each conversion rule to the field type conversion rule library; S143. Parse the obtained data change records, extract the source fields to which each new piece of data belongs, find the corresponding conversion rules from the field type conversion rules, and then perform data format conversion according to the conversion rules; S144. Convert the new data after completing the data format conversion into a JSON format file; S145. Perform duplicate removal operation based on the version number and automatic null value filling operation according to the target field format on the new data in the JSON format file to complete data cleaning; S146. Perform streaming compression on the JSON format file after completing data cleaning using the Deflate algorithm of the zlib library, and use a dynamic compression threshold and construct an incremental dictionary according to the data similarity of adjacent collection periods during the compression process; S147. Name the compressed incremental data file according to the collection time and save it in the preset folder structure.

[0007] Furthermore, the specific steps of step S2 are as follows: S21. Calculate the md5 encoding for the stored incremental data file using the md5 algorithm; S22. Obtain the file size and generation timestamp of the incremental data file, and generate an md5 verification file together with the md5 encoding; S23. Save the md5 verification file in the same folder structure as the corresponding incremental data file; S24. Upload the incremental data file and the corresponding md5 verification file to the STP server file system or a compliant OSS file system.

[0008] Further, the specific steps of step S3 are as follows: S31. Predetermine the update cycle and update time point; the update cycle is the same as the collection cycle, and within the same cycle, the update time point is after the collection time point; S32. Set a timed update task for the update time point within each update cycle; S33. The client system responds to the update task and downloads the latest new data file and the corresponding md5 verification file from the STP server file system or the OSS file system; S34. Calculate the md5 encoding for the new data file, and obtain the generation timestamp and file size of the new data file; S35. Check whether the calculated md5 encoding, the file size of the new data file, and the generation timestamp are all consistent with the md5 encoding, file size, and generation timestamp recorded in the md5 verification file; If they are all consistent, proceed to step S4; If there are inconsistent items, proceed to step S36; S36. Initiate a re-download and return to step S33.

[0009] Further, the specific steps of step S4 are as follows: S41. Predetermine to divide the collection period into sub-periods; S42. After decompressing the verified incremental data file by the client system, obtain the incremental data, classify it according to the sub-periods and table structures, and then merge the incremental data of each sub-period within the same table structure to generate a data update task queue; S43. The client system executes the data update task queue in a transaction management manner, performs incremental data updates in a mixed manner of batch insertion and differential update, and performs exception detection for data conflicts and format errors during the update process. When an exception is detected, a queue to be repaired is automatically generated; S44. The client system feeds back the update result.

[0010] Further, the specific steps of step S43 are as follows: S431. The client system responds to the data update task queue and starts a database transaction; S432. The client system sequentially retrieves data update tasks from the data update task queue; S433. The client system detects the type of incremental data in the retrieved data update task; If it is a newly added table, use the batch insertion method to update the incremental data to the target database, and go to step S434; If it is an existing table, use the differential update method to update the differences between the incremental data and the existing table to the target database; S434. Detect whether there is a primary key conflict or an incremental constraint conflict during the incremental data update process; If so, go to step S435; If not, go to step S437; S435. Generate a violation record and generate a repair queue for the violation record; S436. The client system performs a database transaction rollback and executes the repair queue generated by the violation record to repair the incremental data, and goes to step S44; S437. The client system rebuilds the index and updates the statistical information for the updated target database; The specific steps of step S44 are as follows: S441. The client counts the successfully updated data and the records of the update failures, and generates an update result; S442. The client system feeds back the update result to the system administrator or the user.

[0011] In a second aspect, an embodiment of the present application further provides a log-based incremental database table continuous delivery system, including: An incremental data file storage module, which extracts the incremental data within a preset time period from the data log file of the information system at a set period and compresses it to obtain an incremental data file and stores it; An incremental data file upload module, which performs md5 verification on the stored incremental data file and uploads the incremental data file and the verification file to the server file system; An incremental data file download and verification module, which downloads the incremental data file and the verification file from the server file system at a set period in the client system and uses the verification file to verify the incremental data file; An incremental data file update module, which decompresses the verified incremental data file and groups it in the client system, then performs a data update operation on the target database, and sends the update result to the user.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the log-based incremental database table continuous delivery method as described in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the log-based incremental database table continuous delivery method described in the first aspect are implemented.

[0014] As can be seen from the above technical solutions, the present invention has the following advantages: In the log-based incremental database table continuous delivery method, device, equipment and medium provided by the present application, during the data processing process, by adopting data compression, format conversion, integrity verification, and transaction management, the data synchronization efficiency is improved, the data transmission and processing costs are reduced, manual intervention is reduced, and the stability and reliability of the system are improved. Finally, the log-based incremental database table continuous delivery is realized, ensuring the consistency, accuracy and timeliness of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a schematic flowchart of the log-based incremental database table continuous delivery method of the present invention.

[0017] Figure 2 It is a schematic diagram of the log-based incremental database table continuous delivery system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In the following, the specific steps of the log-based incremental database table continuous delivery method will be described in detail, and various embodiments of the present disclosure will be described more comprehensively. The present disclosure can have various embodiments, and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents and / or alternative solutions falling within the spirit and scope of the various embodiments of the present disclosure.

[0019] Exemplarily, with the development of big data and cloud computing technologies, various information systems continuously generate massive amounts of data. As the data storage foundation of information systems, the update and synchronization of database data are crucial for maintaining business continuity and data consistency. Traditional data synchronization methods have become difficult to meet the requirements of efficient and real-time data synchronization due to challenges such as large data volume, low transmission efficiency, and huge resource consumption. The incremental data synchronization technology emerged. It only transmits the data that has changed since the last synchronization, greatly improving the efficiency of data synchronization and becoming the mainstream method for database data synchronization currently.

[0020] However, there are still many deficiencies in the existing incremental data synchronization technology in practical applications. Firstly, the lack of uniformity in data formats makes the data exchange process between different systems complex and cumbersome, increasing the difficulty and cost of synchronization. Secondly, the verification mechanism is not sound enough, and there is a risk of data being tampered with or lost during transmission, thus damaging the accuracy and credibility of the data. Moreover, the low level of automation is also a major problem faced by incremental data synchronization. Frequent manual monitoring and adjustment are required, reducing the overall work efficiency and system flexibility. Therefore, how to achieve efficient and robust incremental data synchronization while ensuring data security has become an urgent problem to be solved.

[0021] To address the above problems, this embodiment provides a method for continuous delivery of incremental database tables based on logs. By periodically extracting and compressing incremental data, the burden of data transmission and storage is reduced. Using MD5 verification ensures the integrity of the data, preventing the data from being tampered with or damaged during transmission. The timed download and verification on the client system ensure the timely update of the data. Through grouping and batch updating, the efficiency of data update is improved, reducing the pressure on the database.

[0022] Next, the technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0023] Please refer to Figure 1 The flowchart of the method for continuous delivery of incremental database tables based on logs in a specific embodiment is shown. The method includes the following steps: S1. Periodically extract the incremental data within a preset time period from the data log file of the information system according to a set cycle and compress it to obtain an incremental data file and store it; It should be noted that extracting incremental data at regular intervals according to the set period can accurately obtain data changes, reduce unnecessary data processing, and improve data processing efficiency; compressing incremental data can reduce storage space occupancy and facilitate storage and transmission; S2. Perform md5 verification on the stored incremental data file, and upload the incremental data file and the verification file to the server file system; It should be noted that performing md5 verification on the incremental data file ensures the integrity and accuracy of the file during storage and transmission; uploading it to the server file system facilitates client download and acquisition; S3. The client system regularly downloads the incremental data file and the verification file from the server file system at the set period, and uses the verification file to verify the incremental data file; It should be noted that the client regularly downloads and verifies the file, thereby ensuring that the data obtained by the client is complete and not tampered with, to ensure accurate subsequent data update; S4. After the client system decompresses and groups the verified incremental data file, it performs a data update operation on the target database and sends the update result to the user; It should be noted that grouping and updating the verified data to the target database improves the efficiency and accuracy of data update; sending the update result to the user enables the user to timely understand the data update status.

[0024] Through the process of incremental data extraction, processing, transmission to client update in this embodiment, automatic incremental update of data is achieved, ensuring data consistency and timeliness, improving the efficiency of data synchronization, and reducing data transmission and processing costs.

[0025] Further, as a refinement and extension of the specific implementation manner of the above embodiment, in order to fully illustrate the specific implementation process in this embodiment, another method for continuous delivery of an incremental database table based on logs is provided. This method includes the following steps: S1. Extract the incremental data within a preset time period from the data log file of the information system at regular intervals according to the set period and compress it to obtain an incremental data file for storage; The specific steps of step S1 are as follows: S11. Preset the collection period in advance, and set collection time points within the collection period, and use two adjacent collection time points as the collection time period; S12. Set a timed collection task for the collection time points within each collection period; S13. Respond to the collection task to obtain the data change records within the collection time period to which the current collection time point belongs from the corresponding data log file of the source database of the information system; S14. Format the obtained data change records, compress them to obtain an incremental data file, name the incremental data file, and store it in a pre-set file structure; It should be noted that by setting the collection period, time point, and task, the data collection process can be precisely controlled; S2. Perform md5 verification on the stored incremental data file, and upload the incremental data file and the verification file to the server file system. The specific steps of step S2 are as follows: S21. Calculate the stored incremental data file using the md5 algorithm to obtain the md5 code; S22. Obtain the file size and generation timestamp of the incremental data file, and generate an md5 verification file together with the md5 code; S23. Save the md5 verification file to the same folder structure as the corresponding incremental data file; S24. Upload the incremental data file and the corresponding md5 verification file to the STP server file system or a compliant OSS file system; It should be noted that by obtaining the file size and timestamp and generating a verification file, the comprehensive verification of the added data file is ensured; saving the verification file to the same folder structure facilitates management and matching; uploading to multiple file systems ensures applicability; S3. The client system regularly downloads the incremental data file and the verification file from the server file system at a set period, and uses the verification file to verify the incremental data file. The specific steps of step S3 are as follows: S31. Predetermine the update period and update time point; the update period is the same as the collection period, and in the same period, the update time point is after the collection time point; S32. Set a timed update task for the update time point within each update period; S33. The client system responds to the update task and downloads the latest new data file and the corresponding md5 verification file from the STP server file system or the OSS file system; S34. Calculate the md5 code for the new data file, and obtain the generation timestamp and file size of the new data file; S35. Check whether the calculated md5 code, the file size of the new data file, and the generation timestamp are all consistent with the md5 code, file size, and generation timestamp recorded in the md5 verification file; If they are all consistent, go to step S4; If there are inconsistent items, go to step S36; S36. Start re-downloading and return to step S33; It should be noted that by setting the update period and time point and coordinating with the collection period, the data can be updated in a timely manner; and the three types of verifications comprehensively ensure the integrity and accuracy of the downloaded files, preventing incorrect data from entering the subsequent processing process; S4. After the client system groups the verified incremental data files, it performs a data update operation on the target database and sends the update result to the user; the specific steps of step S4 are as follows: S41. First, divide the collection period into sub-periods; S42. After the client system decompresses the verified incremental data files to obtain incremental data, it classifies the data according to the sub-periods and table structures, and then merges the incremental data of each sub-period within the same table structure to generate a data update task queue; S43. The client system executes the data update task queue in a transaction management manner, performs incremental data updates in a mixed manner of batch insertion and differential update, and performs exception detection for data conflicts and format errors during the update process. When an exception is detected, a repair queue to be repaired is automatically generated; S44. The client system feeds back the update result; It should be noted that dividing the collection period into sub-periods, classifying and merging the data, and generating a task queue can improve the efficiency of data update; through transaction management and exception detection, the stability and reliability of data update are ensured, and data errors and inconsistencies are avoided.

[0026] In an embodiment of the present invention, based on step S14, a possible embodiment will be given below to non-restrictively elaborate on its specific implementation.

[0027] The specific steps of step S14 are as follows: S141. First, obtain the type of the source database, determine the fields in the source database as source fields, and specify corresponding target fields for each source field in the target database; S142. Set conversion rules according to the data types of the source fields and the target fields, and add each conversion rule to the field type conversion rule library; S143. Analyze the obtained data change records, extract the source fields to which each new data belongs, find the corresponding conversion rules from the field type conversion rules, and then perform data format conversion according to the conversion rules; S144. Convert the new data after data format conversion into a JSON format file; S145. Perform duplicate removal operation based on the version number and automatic filling operation for null values according to the target field format on the new data in the JSON format file to complete data cleaning; the specific steps of step S145 are as follows: S1451. Record the version number of the field to which the new data belongs; S1452. Determine whether there is data with the same version number in the newly added data; If so, go to step S1453; If not, go to step S1454; S1453. Remove one of the newly added data with the same version number; S1454. Determine whether there are null values in the newly added data; If so, go to step S1455; If not, go to step S146; S1455. Determine the fields in the target database corresponding to the null values in the newly added data, and determine whether the determined fields allow null values; If so, go to step S146; If not, go to step S1456; S1456. Fill the null values according to the data format of the determined fields; Exemplarily, if the field corresponding to the target database in the newly added data is the price field, since the price field in the products table structure of the target database does not allow null values, the default value is set to 0.00, and the default value of the corresponding product_name field is 'Unknown'; S146. Use the Deflate algorithm of the zlib library to perform streaming compression on the JSON format file after data cleaning, and use a dynamic compression threshold and construct an incremental dictionary based on the data similarity of adjacent acquisition cycles during the compression process; The specific steps of step S146 are as follows: S1461. Determine whether the source database is incrementally updated for the first time; If so, go to step S1462; If not, go to step S1463; S1462. Add the field definitions and corresponding data types of the incremental data in the JSON format file to the incremental dictionary, and go to step S165; S1463. Compare the incremental data with the incremental dictionary to determine whether there are new field definitions; If there are, go to step S1464; If not, go to step S1466; S1464. Add the newly added field data definitions and corresponding data types to the incremental dictionary; S1465. Determine whether the sum of the data values in the incremental data and the size of the incremental dictionary exceeds the preset compression threshold; If so, go to step S1466; If not, go to step S1467; S1466. Stream-compress the incremental dictionary and the data values in the incremental data using the Deflate algorithm of the zlib library, and proceed to step S147; S1467. Determine that the incremental dictionary and incremental data do not need to be compressed, and use the original incremental dictionary and incremental data as the compressed incremental data file; S147. Name the compressed incremental data file according to the collection time and save it to the pre-set folder structure; It should be noted that obtaining the source database type and setting the field conversion rules can be compatible with different database types, enabling accurate data transmission between different databases; data cleaning ensures data quality by removing duplicate and incorrect data; using a specific algorithm to compress and construct an incremental dictionary improves compression efficiency and reduces data storage and transmission costs.

[0028] In an embodiment of the present invention, based on steps S43 and S44, a possible embodiment will be given below to non-restrictively elaborate on its specific implementation.

[0029] The specific steps of step S43 are as follows: S431. The client system responds to the data update task queue to start a database transaction; S432. The client system sequentially retrieves data update tasks from the data update task queue; S433. The client system detects the type of incremental data in the retrieved data update task; If it is a new table, use the batch insert method to update the incremental data to the target database, and proceed to step S434; If it is an existing table, use the differential update method to update the difference between the incremental data and the existing table to the target database; S434. Detect whether there is a primary key conflict or an incremental constraint conflict during the incremental data update process; If so, proceed to step S435; If not, proceed to step S437; S435. Generate violation records and generate a repair queue for the violation records; S436. The client system executes a database transaction rollback and executes the repair queue for the generated violation records to repair the incremental data, and proceeds to step S44; S437. The client system rebuilds the index and updates the statistical information for the updated target database; It should be noted that selecting an appropriate update method according to the data type improves the update efficiency; detecting conflicts and rolling back transactions ensures data consistency; rebuilding the index and updating the statistical information maintain the query performance of the database.

[0030] Step S44 is specifically as follows: S441. The client counts the successfully updated data and the records of failed updates, and generates an update result. S442. The client system feeds back the update result to the system administrator or the user. It should be noted that by counting the update result and feeding it back, the user can clearly understand the data update situation. In step S42, the incremental data file is decompressed by combining with the incremental dictionary to obtain the incremental data.

[0031] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0032] As Figure 2 shown, the following is an embodiment of the log-based incremental database table continuous delivery device provided by the embodiments of the present disclosure. This device and the log-based incremental database table continuous delivery methods of the above embodiments belong to the same inventive concept. The details not described in detail in the embodiments of the log-based incremental database table continuous delivery device can refer to the embodiments of the log-based incremental database table continuous delivery method.

[0033] The device includes: An incremental data file storage module, which extracts the incremental data within a preset time period from the data log file of the information system at a set period and compresses it to obtain an incremental data file and stores it. An incremental data file upload module, which performs md5 verification on the stored incremental data file and uploads the incremental data file and the verification file to the server file system. An incremental data file download and verification module, which at the client system downloads the incremental data file and the verification file from the server file system at a set period and uses the verification file to verify the incremental data file. An incremental data file update module, which at the client system decompresses the verified incremental data file and groups it, then performs a data update operation on the target database and sends the update result to the user.

[0034] This embodiment ensures the automation and efficiency of the data processing process through the incremental data file storage module, the incremental data file upload module, the incremental data file download and verification module, and the incremental data file update module.

[0035] The method for continuous delivery of an incremental database table based on logs provided by the embodiments of the present application can be applied to an electronic device. Those skilled in the art can understand that the structure of the electronic device involved in the embodiments of the present invention does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In the embodiments of the present invention, the electronic device includes, but is not limited to, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described herein and / or claimed.

[0036] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a key, a camera, a display screen, and a SIM card interface, etc.

[0037] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0038] The processor may include one or more processing units. For example, the processor may include a central processing unit (CPU), etc., an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0039] Among them, the processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.

[0040] A memory can also be set in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can save the instructions or data that the processor has just used or recycled. If the processor needs to use the instruction or data again, it can directly call it from this memory. This avoids repeated accesses, reduces the waiting time of the processor, and thus improves the system efficiency.

[0041] The above-mentioned electronic device implements the technical solution of the method for continuous delivery of an incremental database table based on a log in the present application, which extracts incremental data within a preset time period from the data log file of the information system at a set period and compresses it to obtain an incremental data file and stores it; performs md5 verification on the stored incremental data file, and uploads the incremental data file and the verification file to the server file system; the client system downloads the incremental data file and the verification file from the server file system at a set period, and uses the verification file to verify the incremental data file; after grouping the incremental data files that pass the verification by the client system, it performs a data update operation on the target database and sends the update result to the user, achieving the beneficial effects of improving data synchronization efficiency, reducing data transmission and processing costs, and reducing manual intervention.

[0042] In the storage medium provided by the present application, there is a program product capable of implementing the method for continuous delivery of an incremental database table based on a log.

[0043] The method for continuous delivery of an incremental database table based on a log includes: extracting incremental data within a preset time period from the data log file of the information system at a set period and compressing it to obtain an incremental data file and storing it; performing md5 verification on the stored incremental data file, and uploading the incremental data file and the verification file to the server file system; the client system downloads the incremental data file and the verification file from the server file system at a set period, and uses the verification file to verify the incremental data file; after grouping the incremental data files that pass the verification by the client system, it performs a data update operation on the target database and sends the update result to the user.

[0044] In some possible implementation manners, the method for continuous delivery of an incremental database table based on a log of the present disclosure can be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0045] The storage medium of the present disclosure may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0046] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log-based incremental database table continuous delivery method, characterized in that: The steps include: S1. Extract incremental data within a preset period of time from the data log file of the information system according to the set cycle and compress it to obtain and store the incremental data file; S2. Perform md5 verification on the stored incremental data files, and upload the incremental data files and verification files to the server file system; S3. The client system downloads the incremental data file and verification file from the server file system at a set period, and uses the verification file to verify the incremental data file; S4. After the client system decompresses and groups the verified incremental data files, it performs data update operations on the target database and sends the update results to the user.

2. The method for continuous delivery of incremental database tables based on logs according to claim 1, characterized in that: The specific steps of step S1 are as follows: S11. Preset the collection cycle and set the collection time point within the collection cycle, with two adjacent collection time points as the collection period; S12. Set a scheduled collection task for each collection time point within the collection cycle; S13. Respond to the collection task and obtain the data change record within the collection period of the current collection time point from the data log file corresponding to the source database of the information system; S14. The acquired data change records are formatted and compressed to obtain an incremental data file, and the incremental data file is named and stored in a pre-set file structure.

3. The log-based incremental database table continuous delivery method according to claim 2 is characterized in that: The specific steps of step S14 are as follows: S141. Pre-acquire the source database type, determine the fields in the source database as source fields, and specify corresponding target fields for each source field in the target database; S142. Set conversion rules according to the data type of the source field and the data type of the target field, and add each conversion rule to the field type conversion rule library; S143. Parse the acquired data change records, extract the source fields to which each newly added data belongs, and search for the corresponding conversion rules from the field type conversion rules, and then perform data format conversion according to the conversion rules; S144. Convert the newly added data after completing the data format conversion into a JSON format file; S145. Perform a deduplication operation based on the version number and an automatic filling operation of the empty value according to the target field format on the newly added data of the JSON format file to complete the data cleaning; S146. The JSON format file after data cleaning is stream compressed using the Deflate algorithm of the zlib library, and a dynamic compression threshold is used in the compression process, and an incremental dictionary is constructed according to the data similarity of adjacent collection cycles; S147. Name the compressed incremental data file according to the acquisition time and save it to a pre-set folder structure.

4. The method for continuous delivery of incremental database tables based on logs according to claim 2 is characterized in that: The specific steps of step S2 are as follows: S21. Calculate the stored incremental data file using the md5 algorithm to obtain the md5 code; S22. Get the file size and generation timestamp of the incremental data file, and generate an md5 verification file together with the md5 encoding; S23. Save the md5 verification file to the same folder structure as the corresponding incremental data file; S24. Upload the incremental data file and the corresponding md5 verification file to the STP server file system or an OSS file system that meets the requirements.

5. The method for continuous delivery of incremental database tables based on logs according to claim 4 is characterized in that: The specific steps of step S3 are as follows: S31 pre-set update cycle and update time point; the update cycle is the same as the acquisition cycle, and the update time point in the same cycle is after the acquisition time point; S32. Set a scheduled update task for each update period at the update time point; S33. The client system responds to the update task by downloading the latest new data files and the corresponding md5 verification files from the STP server file system or OSS file system; S34. Calculate the md5 code for the new data file and obtain the generation timestamp and file size of the new data file; S35. Check whether the calculated md5 code, the file size of the newly added data file, and the generation timestamp are consistent with the md5 code, file size, and generation timestamp recorded in the md5 verification file; If they are consistent, go to step S4; If there is any inconsistency, go to step S36; S36. Start re-downloading and return to step S33.

6. The method for continuous delivery of incremental database tables based on logs according to claim 5 is characterized in that: The specific steps of step S4 are as follows: S41. Pre-select the collection period to be divided into sub-periods; S42. The client system decompresses the verified incremental data file to obtain the incremental data, and classifies it according to the sub-period and table structure, and then merges the incremental data of each sub-period in the same table structure to generate a data update task queue; S43. The client system executes the data update task queue in a transaction management manner, performs incremental data updates in a hybrid manner of batch insertion and differential update, and performs abnormal detection of data conflicts and format errors during the update process, and automatically generates a queue to be repaired when an abnormality is detected; S44. The client system feeds back the update result.

7. The method for continuous delivery of incremental database tables based on logs according to claim 6 is characterized in that: The specific steps of step S43 are as follows: S431. The client system responds to the data update task queue to start the database transaction; S432. The client system sequentially takes out data update tasks from the data update task queue; S433. The client system detects the type of incremental data in the data update task; If it is a newly added table, the incremental data is updated to the target database by batch insert, and the process goes to step S434; If it is an existing table, the difference update method is used to update the difference between the incremental data and the existing table to the target database; S434. Detect whether there is a primary key conflict or incremental constraint conflict during the incremental data update process; If yes, go to step S435; If not, proceed to step S437; S435. Generate violation records and generate a repair queue for the violation records; S436. The client system executes database transaction rollback and generates a repair queue for violation records to perform incremental data repair, and then proceeds to step S44; S437. The client system rebuilds the index for the updated target database and updates the statistics; The specific steps of step S44 are as follows: S441. The client counts the successfully updated data and the failed update records and generates the update results; S442. The client system feeds back the update result to the system administrator or user.

8. A log-based incremental database table continuous delivery system, characterized in that: include: The incremental data file storage module extracts incremental data within a preset period from the data log file of the information system according to a set period and compresses it to obtain and store the incremental data file; The incremental data file upload module performs md5 verification on the stored incremental data files and uploads the incremental data files and verification files to the server file system; The incremental data file download and verification module downloads the incremental data file and verification file from the server file system at a set period in the client system, and uses the verification file to verify the incremental data file; The incremental data file update module performs data update operations on the target database and sends the update results to the user after the client system decompresses and groups the incremental data files that have passed the verification.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the log-based incremental database table continuous delivery method as described in any one of claims 1 to 7 when executing the program.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the log-based incremental database table continuous delivery method as described in any one of claims 1 to 7 are implemented.

Images