A method for increasing the sorting of large amounts of log data

Through multi-threaded parsing and path file naming, the redo log file parsing speed and data disordered data are solved, efficient sorting and synchronization of large data volumes are achieved, high-speed business needs are met, and stable and low-cost is provided. solution.

CN111858534BActive Publication Date: 2024-11-15SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010729406.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-27
Publication Date
2024-11-15
Estimated Expiration
2040-07-27

AI Technical Summary

Technical Problem

The existing technology cannot effectively and concurrently parse the redo log files generated at high speed, resulting in the parsing speed that cannot reach the log generation speed. The data after multi-threaded parsing is disordered and it is difficult to sort correctly.

Method used

The Redo log files are parsed by multi-threading, sorted by path file naming, dropping and crawling, and the log file name is obtained by FetchOracleRedoLogFile. The ExecuteOracleLogMiner processor generates data streams for every 10,000 pieces, and data transmission and sorting is carried out through the FetchRedoBySequential processor. Use the FragmentNumber and NextFragmentNumber attributes to judge the file parsing status to ensure orderly data transmission.

Benefits of technology

It realizes efficient sorting and synchronizing large data volumes, meets T-level business needs, has simple processor configuration, low development cost, and stable operation, and is suitable for a wide range of application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111858534B_ABST
    Figure CN111858534B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of big data ETL, and specifically provides a method for increasing the amount of large log data sorting, multi-threaded parsing of multiple batches of unordered data after Redo logs, sorting by path file naming, disk placement and crawling, and then transmitting them backward to the target source in a correct and orderly queue to complete data synchronization. Compared with the prior art, the present invention can use multi-threaded parsing to achieve the T level of synchronized data. It meets the incremental needs of most businesses, and the processor has simple configuration, low development cost, stable operation, low operation difficulty, and good promotion value.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data ETL, and specifically provides a method for increasing the sorting of large log data volumes. Background Art

[0002] Today's society is a rapidly developing society, with advanced technology, information flow, closer communication between people, and more convenient life. Big data is the product of this high-tech era. At present, with the increasing prevalence of informatization, the daily incremental data in the business is no longer limited to a few hundred KB or a few hundred GB of data. More business incremental redo logs have reached the T level. This leads to another problem: the speed of parsing logs cannot keep up with the speed of generating redo logs.

[0003] In order to ensure data consistency, that is, data DML consistency, the previous parsing of redo log files was run in a single thread, that is, parsing one by one according to the generation time of the redo logs. This parsing method cannot be concurrent, resulting in a speed that cannot reach the log generation speed, so a multi-threaded log parsing solution is on the horizon. However, the multi-threaded method has achieved the speed, but the data after each thread parses the log file is disorganized. How to rearrange this data in a correct and orderly manner has become a technical problem faced by technical personnel in this field. Summary of the invention

[0004] The present invention aims at the above-mentioned deficiencies of the prior art and provides a highly practical method for increasing the amount of log data sorting.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A method for increasing the sorting of large amounts of log data. After parsing multiple batches of unordered data from Redo logs through multi-threading, the data is sorted by path file naming, disk placement, and crawling, and then transmitted back to the target source in a correct and orderly queue to complete data synchronization.

[0007] Furthermore, before parsing the Redo log, you need to obtain the names of all redo log files in a certain period of time in FetchOracleRedoLogFile, and add the current.index (1, 2, 3...) attribute to each log file name in sequence according to the time when the log was generated.

[0008] Furthermore, when multiple threads parse Redo logs, ExecuteOracleLogMiner is the log file parsing processor. Each thread processes a log file, generates a data stream for every 10,000 records and transmits it backward. At the same time, the FragmentNumber and NextFragmentNumber properties are set. When FragmentNumber and NextFragmentNumber are the same, it means that the log file has been parsed.

[0009] Furthermore, multiple batches of unordered data are transmitted using the processor FetchRedoBySequential. First, the processor is checked to see if there is a state saved in the data table. If the state of the data table is not found in the state, the initialization state Initial sequence (1-1) is adopted.

[0010] Preferably, the key columns in the data table are user name and data table, and the value column is the expected serial number batch data.

[0011] Furthermore, the processor FetchRedoBySequential is executed in a single thread. If the transmitted data stream cannot be identified, the logic adopted is to store it on the disk. The path is the path configured by the user ( / indata / disk_0 / nifi / datas+" / username / data table / current.index"), and the file name is FragmentNumber+NextFragmentNumber.

[0012] Furthermore, when the expected data stream is transmitted, it is first transmitted backwards, and then the disk is scanned to see if there is a next data stream file. After the data stream is transmitted, its NextFragmentNumber property is obtained. There are two cases:

[0013] 1) When FragmentNumber and NextFragmentNumber are the same, it means that the log file of current.index has been sorted. Next, scan the file name starting with 1 in the path / indata / disk_0 / nifi / datas / user1 / tableA / current.index+1.

[0014] 2) When FragmentNumber is different from NextFragmentNumber, it means that the log file of current.index has not been sorted yet. Scan the file name starting with NextFragmentNumber in the current / indata / disk_0 / nifi / datas / user1 / tableA / current.index path.

[0015] If the expected file is scanned, the file is obtained and transmitted backwards and the above steps 1) and 2) are repeated until the expected file is not scanned, which means that the expected file has not been transmitted to the processor. At this time, the state of the processor user1.tableA is updated to the attribute (current.index+FragmentNumber) corresponding to the expected data stream, waiting for the inflow of the data stream.

[0016] Furthermore, when scanning files from disk, after a current.index path is scanned, the Delete Disk File property can be set to true to delete the path.

[0017] Furthermore, if an error occurs when retrieving a file from the disk, an exception is thrown, the status is updated to the previous batch status of the error file, and a new data stream rollback is generated.

[0018] Compared with the prior art, the method for increasing the amount of log data sorting of the present invention has the following outstanding beneficial effects:

[0019] The present invention provides a method for increasing the large amount of log data sorting, provides parsing of data attributes, disk path and file name, etc., and solves the problem that a single thread cannot match the large amount of redo log increments. Multi-threaded parsing can be used to achieve a T-level of synchronized data. The incremental requirements of most businesses are met, and the processor has simple configuration, low development cost, stable operation, and low operating difficulty, so that data information is arranged into effective data information, and has a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Attached Figure 1 It is a flow chart of collecting incremental redo logs in a method of increasing the amount of log data sorting;

[0022] Attached Figure 2 It is a schematic diagram of the FetchRedoBySequential configuration parameters in the method of increasing the sorting of large amounts of log data;

[0023] Attached Figure 3 It is a schematic diagram of the FetchRedoBySequential save state in the method of increasing the sorting of large amounts of log data. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] A best embodiment is given below:

[0026] like Figure 1 As shown, the method for increasing the large amount of log data sorting in this embodiment is mainly to parse multiple batches of unordered data after multi-threading of Redo logs, sort them by path file naming, disk placement and crawling, and then transmit them backward to the target source in a correct and orderly queue to complete data synchronization.

[0027] This is achieved through the following steps:

[0028] FetchOracleRedoLogFile obtains the names of all redo log files in a certain period of time, and adds the current.index (1, 2, 3...) attribute to each log file name in sequence according to the time when the log was generated.

[0029] ExecuteOracleLogMiner is a log file parsing processor (can be set to multi-threaded), each thread processes a log file, and generates a data stream for every 10,000 records to transmit backward to avoid memory overflow. At the same time, set the FragmentNumber (the current batch number of the current redo file 1,2,3...) and NextFragmentNumber (the next batch number of the current redo log file 1,2,3...) attributes. When FragmentNumber and NextFragmentNumber are the same, it means that the log file has been parsed.

[0030] like Figure 2As shown, multiple batches of unordered data are transmitted using the processor FetchRedoBySequential. First, check whether there is a state saved in the data table in the processor. If the state of the data table is not found in the state, the initialization state Initial sequence (1-1) is adopted.

[0031] The key columns in the data table are user name and data table, and the value column is the expected serial number batch data.

[0032] like Figure 3 As shown in FIG. 5, the first batch sequence number of the 5012th redo log file. If the state of the data table is not found in the state, the initial state Initial sequence (1-1) is adopted.

[0033] The processor FetchRedoBySequential is executed in a single thread. Since the previous processor is running in multiple threads, the data transmitted is out of order. Take user1.tableA as an example. The expected data flow at the beginning of the run is 1-1, that is, the data flow with the current.index attribute of 1 and the FragmentNumber attribute of 1. However, the data flow usually transmitted is not what you expect, and may be 2-2 or 1-2. The logic adopted at this time is to store it on the disk, and the path is the path configured by the user ( / indata / disk_0 / nifi / datas+" / username / data table / current.index"), and the file name is FragmentNumber+NextFragmentNumber.

[0034] For example: / indata / disk_0 / nifi / datas / user1 / tableA / 2 / 2-3.

[0035] When the expected data stream 1-1 is transmitted, it is first transmitted backwards, and then the disk is scanned to see if there is a next data stream file. After the data stream 1-1 is transmitted, its NextFragmentNumber property is obtained. There are two cases at this time:

[0036] 1) When FragmentNumber and NextFragmentNumber are the same, it means that the log file of current.index has been sorted. Next, scan the file name starting with 1 in the path / indata / disk_0 / nifi / datas / user1 / tableA / current.index+1.

[0037] 2) When FragmentNumber is different from NextFragmentNumber, it means that the log file of current.index has not been sorted yet. Scan the file name starting with NextFragmentNumber in the current / indata / disk_0 / nifi / datas / user1 / tableA / current.index path.

[0038] If the expected file is scanned, the file is obtained and transmitted backwards and the above steps 1 and 2 are repeated until the expected file is not scanned, which means that the expected file has not been transmitted to the processor. At this time, the state of the processor user1.tableA is updated to the attribute (current.index+FragmentNumber) corresponding to the expected data stream, waiting for the inflow of the data stream.

[0039] When scanning files from disk, after a current.index path is scanned, you can set the DeleteDisk File property to true to delete the path.

[0040] When an error occurs when retrieving a file from disk, an exception is thrown. The state is updated to the previous batch of states of the error file and a new data stream is generated to roll back. For example, when an error occurs when retrieving a file named 3-4 with current.index 2, the state is set to current.index 2 and FragmentNumber 2, i.e. 2-2. At the same time, a data stream is generated with properties of current.index 2, FragmentNumber 2, and NextFragmentNumber 3. The stream is rolled back to itself and triggered again.

[0041] The above logic is used to sort and transmit multiple batches of unordered data.

[0042] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any appropriate changes or substitutions made by any ordinary technician in the technical field that conform to the claims of the present invention for increasing the sorting method of large log data volumes shall fall within the patent protection scope of the present invention.

[0043] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for increasing the amount of log data sorting, characterized in that: After the Redo logs are parsed by multiple threads, multiple batches of unordered data are sorted by path file naming, disk storage, and crawling, and then transmitted to the target source in a correct and orderly queue to complete data synchronization; Before parsing the Redo log, you need to obtain the names of all redo log files in a certain period of time in FetchOracleRedoLogFile, and add the current.index (1, 2, 3...) attribute to each log file name in sequence according to the time when the log was generated; When multi-threaded parsing of Redo logs, ExecuteOracleLogMiner is the log file parsing processor. Each thread processes a log file, generates a data stream for every 10,000 records and transmits it backward. At the same time, the FragmentNumber and NextFragmentNumber properties are set. When FragmentNumber and NextFragmentNumber are the same, it means that the log file has been parsed. Multiple batches of unordered data are transmitted using the processor FetchRedoBySequential. First, check whether there is a state saved in the data table in the processor. If the state of the data table is not found in the state, the initial state Initial sequence (1-1) is adopted. The key column in the data table is the user name and data table, and the value column is the expected serial number batch data; The processor FetchRedoBySequential is executed in a single thread. If the transmitted data stream cannot be identified, the logic adopted is to store it on the disk. The path is the path configured by the user ( / indata / disk_0 / nifi / datas+" / username / data table / current.index"), and the file name is FragmentNumber+NextFragmentNumber; When the expected data stream is transmitted, it is first transmitted backwards, and then the disk is scanned to see if there is the next data stream file. After the data stream is transmitted, its NextFragmentNumber property is obtained. There are two cases: 1) When FragmentNumber and NextFragmentNumber are the same, it means that the log file of current.index has been sorted. Next, scan the file name starting with 1 in the path / indata / disk_0 / nifi / datas / user1 / tableA / current.index+1; 2) When FragmentNumber is different from NextFragmentNumber, it means that the log file of current.index has not been sorted yet. Scan the file name starting with NextFragmentNumber in the current / indata / disk_0 / nifi / datas / user1 / tableA / current.index path; If the expected file is scanned, the file is obtained and transmitted backward and the above steps 1) and 2) are repeated until the expected file is not scanned, which means that the expected file has not been transmitted to the processor; at this time, the processor user1.tableA status is updated to the attribute corresponding to the expected data stream (current.index+FragmentNumber), waiting for the inflow of the data stream.

2. A method for increasing the amount of log data sorting according to claim 1, characterized in that: When scanning files from disk, after a current.index path is scanned, set the Delete Disk File property to true to delete the path.

3. A method for increasing the amount of log data sorting according to claim 2, characterized in that: If an error occurs when retrieving a file from disk, an exception is thrown and the status is updated to the previous batch status of the error file and a new data stream rollback is generated.