Log merging system and method based on big data

Abstract

The invention discloses a log merging system and method based on big data. The system comprises a data acquisition module, a log merging task adding module, a data calculation module and a calculation result presentation module. The data acquisition module, the log merging task adding module, the data calculation module and the calculation result presentation module are connected in sequence; the data acquisition module is used for inputting log index data; the log merging task adding module is used for selecting a log type, determining a merging condition and specifying a merging period; the data calculation module is used for offline data calculation and real-time data calculation; and the calculation result presentation module is used for displaying a merging task list, a merging statistical result, a merging statistical list and log details. And within the range of 3-5 seconds after the merging task is executed, the merging result can be preliminarily output for data analysis, and the completion of preparation of all data is not needed.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data log merging, and particularly relates to a big data log merging system and method based thereon. Background Art

[0002] In the existing research on big data log retrieval, it is found that when aggregating and statistically analyzing within a range of time, it is necessary to perform data segmentation in terms of time and group and aggregate according to keywords in terms of dimensions, so as to analyze and study the logs at a finer granularity. Using ElasticSearch to aggregate and statistically analyze offline logs according to time steps can initially achieve log merging and retrieval. However, in the big data scenario, the log merging takes a long time, occupies too much resources, and cannot support real-time log calculation, etc., which limits the function. Using Flink time window to aggregate and statistically analyze real-time logs can improve the timeliness of log merging and greatly reduce the occupancy of hardware resources, but the flexibility of offline log merging is limited. Summary of the Invention

[0003] The purpose of the present invention is to provide a big data log merging system and method based thereon, so as to solve the problems in the prior art that in the big data scenario, the log merging takes a long time, occupies too much resources, and cannot support real-time log calculation, etc., which limits the function.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] A big data log merging system based thereon includes a data acquisition module, a log merging task adding module, a data calculation module, and a calculation result presentation module; the data acquisition module, the log merging task adding module, the data calculation module, and the calculation result presentation module are connected in sequence;

[0006] The data acquisition module is used for inputting log index data; the log merging task adding module is used for selecting the log type, determining the merging conditions, and specifying the merging period; the data calculation module is used for offline data calculation and real-time data calculation; the calculation result presentation module is used for displaying the merging task list, the merging statistical result, the merging statistical list, and the log details.

[0007] Further, the selected log type is the selected index type; the merging conditions include: source IP, source port, destination IP, destination port, and protocol.

[0008] Further, the merging period is between 10 min and 60 min.

[0009] Further, the data calculation module includes an offline data calculation unit, a real-time data calculation unit, and a calculation result merging unit. The data calculation unit and the real-time data calculation unit perform parallel calculations and then output to the calculation result merging unit.

[0010] Furthermore, for offline data calculation: start a calculation task according to the specified log merging type, merging conditions, and merging period, cyclically fetch the log data of the most recent N days from ElasticSearch for analysis and calculation, and store the calculation results of each time window in the database.

[0011] Furthermore, for real-time data calculation: configure and start Flink calculation according to the specified merging conditions, and perform real-time stream data calculation based on a rolling time window. After the calculation of each time window in Flink is completed, output and store the data in the database.

[0012] Furthermore, for the calculation result merging unit: after the merging task is started, start a one-time scheduled task after one merging period, and identify and merge the fragment tasks from the offline window data and real-time window data according to the start time of the calculation task and store them in the database.

[0013] Furthermore, the calculation result presentation module includes a merging task list unit, a merging statistical result unit, a merging statistical list unit, and a log details unit; the merging task list unit, the merging statistical result unit, the merging statistical list unit, and the log details unit are connected in sequence;

[0014] The merging task list unit is used to present all calculation tasks in the form of tasks first;

[0015] The merging statistical result unit displays the merging results according to the time window and merging conditions, and displays the corresponding merging quantity;

[0016] The merging statistical list unit is used to present the log retrieval page to view the group of logs under the corresponding time and merging conditions;

[0017] The log details unit is used to display the formatted information after log processing and the original log before processing.

[0018] Furthermore, a big data-based log merging method includes the following steps:

[0019] After the log data is accessed, select the log type, determine the merging conditions, and specify the merging period in sequence;

[0020] After the log merging task is added, taking the current time as the dividing line, trigger both offline data calculation and real-time data calculation methods simultaneously. The two calculation tasks perform log merging calculations; the offline calculation task calculates the historical data of the most recent N days, and the real-time calculation task calculates the stream data starting from the current time;

[0021] Present the calculation results in four parts: the merging task list, the merging statistical result, the merging statistical list, and the log details.

[0022] Further, after the log data is accessed, the selected log type, merging conditions, and merging period are determined in sequence:

[0023] a. Select the log type: The log type is the index type, and the merging task supports single-log merging of all existing current logs in the situation awareness platform.

[0024] b. Determine the merging conditions: Log merging is mainly based on the five-tuple of the request information for merging, including: source IP, source port, destination IP, destination port, and protocol.

[0025] c. Specify the merging period: Considering the background of big data traffic, the log merging supports log merging tasks with a minimum of 10 minutes and a maximum of 60 minutes, which can be input manually during configuration according to the analysis requirements.

[0026] Data calculation process

[0027] a. Offline data calculation: The background starts the calculation task according to the specified log merging type, merging conditions, and merging period, polls and grabs the log data of the most recent N days from ElasticSearch for analysis and calculation, and stores the calculation results of each time window in the database.

[0028] b. Real-time data calculation: The background configures and starts the Flink calculation according to the specified merging conditions and performs real-time stream data calculation based on a rolling time window. After each time window calculation in Flink is completed, the data is output and stored in the database.

[0029] c. Boundary window merging: After the merging task is started, a one-time scheduled task after one merging period is started. According to the start time of the calculation task, the fragment tasks are identified from the offline window data and the real-time window data for merging and storing in the database.

[0030] Presentation of calculation results

[0031] a. Merging task list: Each addition of a calculation task is a task data. On the log merging page, all calculation tasks are first presented in the form of tasks.

[0032] b. Merging statistical results: The merging results are displayed according to the time window and merging conditions, and the corresponding merging quantity is also displayed.

[0033] c. Merging statistical list: View the group of logs under the corresponding time and merging conditions on the log retrieval page. The merging quantity is consistent with the number of logs queried on the log retrieval page.

[0034] d. Log details include the formatted information after log processing and the original log before processing.

[0035] Compared with the prior art, the present invention has the following technical effects:

[0036] The present invention uses a combination of ElasticSearch offline logs and Flink real-time logs:

[0037] First, it can perform quick merge responses. Within 3 to 5 seconds after executing the merge task, the merge results can be preliminarily output for data analysis without waiting for all data to be prepared. For a single ElasticSearch offline log statistics, it usually takes more than a minute and is extremely prone to memory overflow.

[0038] Second, it can make up for the flexibility of real-time data calculation. When the merge conditions and merge period change, the real-time tasks initiated based on Flink can quickly respond to real-time stream data, enabling subsequent access data to be merged according to the new merge scheme; the offline tasks based on ElasticSearch will simultaneously re-merge historical data and present it on the interface after combining with real-time data. For a pure real-time task, historical data will not be processed after the merge scheme is adjusted.

[0039] Third, it can enhance the log merge analysis ability. The combination of offline logs and real-time logs provides more possibilities for log merge analysis by improving performance, timeliness, and flexibility, and can enhance the overall log merge analysis ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a design diagram of the log merge process.

[0041] Figure 2 It is an example diagram of adding a log merge task.

[0042] Figure 3 It is an example diagram of setting log merge conditions.

[0043] Figure 4 It is an example diagram of the merge task viewing page.

[0044] Figure 5 It is an example diagram of the merge statistics result page. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0046] In the description of the present invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0047] It should also be understood that the terms used in the specification of the present invention are merely for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly dictates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0048] It should be further understood that the term " / and" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally represents an "or" relationship between the preceding and following related objects.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0050] Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0051] Structural schematic diagrams according to the disclosed embodiments of the present invention are shown in the drawings. These figures are not drawn to scale, where for the purpose of clear expression, some details are enlarged and some details may be omitted. The shapes of various regions and layers shown in the figures and their relative sizes and positional relationships are merely exemplary, and may actually deviate due to manufacturing tolerances or technical limitations, and those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0052] Please refer to Figure 1, the present invention provides a big data-based log merging system and method, which combines two merging schemes of offline logs and real-time logs, combines the flexibility and timeliness of log merging statistics, realizes the free combination of merging conditions and merging periods on the page, quickly views the log merging results, and improves the log analysis ability.

[0053] Embodiment 1:

[0054] A big data-based log merging system includes a data acquisition module, a log merging task adding module, a data calculation module, and a calculation result presentation module; the data acquisition module, the log merging task adding module, the data calculation module, and the calculation result presentation module are connected in sequence;

[0055] The data acquisition module is used to input log index data; the log merging task adding module is used to select the log type, determine the merging conditions, and specify the merging period; the data calculation module is used for offline data calculation and real-time data calculation; the calculation result presentation module is used for displaying the merging task list, the merging statistical result, the merging statistical list, and the log details.

[0056] Embodiment 2

[0057] A big data-based log merging method includes the following steps:

[0058] After the log data is accessed, the log type is selected, the merging conditions are determined, and the merging period is specified in sequence;

[0059] After the log merging task is added, taking the current time as the dividing line, two methods of offline data calculation and real-time data calculation are triggered simultaneously, and the two calculation tasks perform log merging calculation; the offline calculation task calculates the historical data of the most recent N days, and the real-time calculation task calculates the streaming data starting from the current time;

[0060] The calculation results are presented in four parts: the merging task list, the merging statistical result, the merging statistical list, and the log details.

[0061] Specifically:

[0062] 1. Log data access

[0063] The log merging function is implemented on the basis of the existing log streaming data, which is a functional supplement to the existing log streaming data and a complete log analysis module implemented on the existing standard log transfer function.

[0064] 2. Log merging task addition

[0065] The addition of the log merging task is mainly divided into three processes. Including: selecting the log type, determining the merging conditions, and specifying the merging period.

[0066] a. Select the log type: The log type is the index type. The merging task supports single-log merging for all existing current logs in the situation awareness platform.

[0067] b. Determine the merging conditions: Log merging is mainly based on the five-tuple of the request information for merging, including: source IP, source port, destination IP, destination port, and protocol. In addition, specific fields can be configured according to the log type in the configuration item to achieve personalized merging.

[0068] c. Specify the merging period: Considering the background of big data traffic, the log merging supports log merging tasks with a minimum of 10 minutes and a maximum of 60 minutes, which can be entered manually according to the analysis requirements during configuration.

[0069] 3. Data calculation process

[0070] The data calculation process of log merging includes two methods: offline data calculation and real-time data calculation. After the log merging task is added, taking the current time as the dividing line, two calculation tasks will be triggered simultaneously in the background for log merging calculation. The offline calculation task calculates the historical data of the last N days, and the real-time calculation task calculates the streaming data starting from the current time:

[0071] a. Offline data calculation: The background starts the calculation task according to the specified log merging type, merging conditions, and merging period, polls and grabs the log data of the last N days from ElasticSearch for analysis and calculation, and stores the calculation results of each time window in the database.

[0072] b. Real-time data calculation: The background configures and starts Flink calculation according to the specified merging conditions and performs real-time streaming data calculation based on the rolling time window. After each time window calculation in Flink is completed, the data is output and stored in the database.

[0073] c. Boundary window merging: Taking the 10-minute interval as an example, after the offline data calculation and real-time data calculation are triggered in this calculation task, the current 10-minute time window will occupy a small time window in both the offline task and the real-time task, and the data in these two windows needs to be merged. After the merging task is started, a one-time scheduled task after one merging period is started, and according to the start time of the calculation task, the fragment tasks are identified from the offline window data and the real-time window data for merging and storing in the database.

[0074] 4. Presentation of calculation results

[0075] The presentation of calculation results includes four parts: merging task list, merging statistical results, merging statistical list, and log details.

[0076] a. Merged task list: Each added calculation task is a task data. On the log merging page, all calculation tasks are first presented in the form of tasks. Clicking on the task details can view all the calculation results of the time windows of this merged task.

[0077] b. Merged statistical results: The merged results are displayed according to the time window and merging conditions, and the corresponding merged quantity is shown. Relatively speaking, the smaller the merged time window, the higher the log dispersion, and the more statistical results here.

[0078] c. Merged statistical list: Clicking on the details icon before each merged statistical result can jump to the log retrieval page to view the group of logs under the corresponding time and merging conditions. The merged quantity must be consistent with the number of logs queried on the log retrieval page, and more specific information about the relevant logs can be queried on the log retrieval page.

[0079] d. Log details: On the log retrieval page, clicking on the details icon before the log can view the log details information, including the formatted information after log processing and the original log before processing. The information viewed here is the most complete information of the log.

[0080] Example 3

[0081] 1. Adding log merging tasks

[0082] Trigger the merging conditions through the new button on the page, supporting custom task names, log types, merging conditions, and merging periods; after the input is completed, click Save to complete the task addition and initiate the calculation.

[0083] 2. Viewing log merging results

[0084] After the task addition is completed, a new task can be seen in the task list;

[0085] Click on the details link to enter the merged task result page, where the log statistical quantity can be seen in terms of the merging conditions and merging periods;

[0086] Click on the operation button before the data bar to jump to the log retrieval page for viewing log details.

[0087] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0088] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present invention can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0090] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal and method can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0093] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0094] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0095] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the processes in Figure 1One or more processes and / or boxes Figure 1 The functions specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes.

[0097] The above content is only to illustrate the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention falls within the protection scope of the claims of the present invention.

Claims

1. A big data log merging system, characterized in that, it includes a data acquisition module, a log merging task addition module, a data calculation module, and a calculation result presentation module; the data acquisition module, the log merging task addition module, the data calculation module, and the calculation result presentation module are connected in sequence; The data acquisition module is used to input log index data; The log merging task addition module is used to select the log type, determine the merging conditions, and specify the merging period; the data calculation module is used for offline data calculation and real-time data calculation; The calculation result presentation module is used to display the merging task list, the merging statistical result, the merging statistical list, and the log details.

2. The big data log merging system according to claim 1, characterized in that, the selected log type is the selected index type; the merging conditions include: source IP, source port, destination IP, destination port, and protocol.

3. The big data log merging system according to claim 1, characterized in that, the merging period is between 10 minutes and 60 minutes.

4. The big data log merging system according to claim 1, characterized in that, the data calculation module includes an offline data calculation unit, a real-time data calculation unit, and a calculation result merging unit. The data calculation unit and the real-time data calculation unit perform parallel calculations and then output to the calculation result merging unit.

5. The big data log merging system according to claim 4, characterized in that, Offline data calculation: Start the calculation task according to the specified log merging type, merging conditions, and merging period, cyclically grab the log data of the most recent N days from ElasticSearch for analysis and calculation, and store the calculation results of each time window in the database.

6. The big data log merging system according to claim 4, characterized in that, Real-time data calculation: Configure and start Flink calculation according to the specified merging conditions and perform real-time stream data calculation based on a rolling time window. After each time window calculation in Flink is completed, output and store the data in the database.

7. The big data log merging system according to claim 4, characterized in that, Calculation result merging unit: After the merging task is started, start a one-time scheduled task after one merging period. According to the start time of the calculation task, identify the segment tasks from the offline window data and the real-time window data for merging and storing in the database.

8. The big data log merging system according to claim 1, characterized in that, the calculation result presentation module includes a merging task list unit, a merging statistical result unit, a merging statistical list unit, and a log details unit; the merging task list unit, the merging statistical result unit, the merging statistical list unit, and the log details unit are connected in sequence; The merging task list unit is used to first present all calculation tasks in the form of tasks; The merging statistical result unit displays the merging results according to the time window and the merging conditions, and displays the corresponding merging quantity; The merging statistical list unit is used to present the log retrieval page to view the group of logs under the corresponding time and merging conditions; The log details unit is used to display the formatted information after log processing and the original log before processing.

9. A method for merging big data logs, characterized in that based on the big data log merging system according to any one of claims 1 to 8, it includes the following steps: After the log data is accessed, the log type is selected, the merging condition is determined, and the merging period is specified in sequence; After the log merging task is added, taking the current time as the dividing line, two methods of offline data calculation and real-time data calculation are triggered simultaneously, and the two calculation tasks perform log merging calculations; the offline calculation task calculates the historical data of the most recent N days, and the real-time calculation task calculates the streaming data starting from the current time; The calculation results are presented in four parts: the merging task list, the merging statistics result, the merging statistics list, and the log details.

10. According to the method for merging big data logs described in claim 9, characterized in that After the log data is accessed, the log type is selected, the merging condition is determined, and the merging period is specified in sequence: a. Select the log type: The log type is the index type, and the merging task supports single-log merging of all existing current logs in the situation awareness platform; b. Determine the merging condition: Log merging is mainly based on the five-tuple of the request information for merging, including: source IP, source port, destination IP, destination port, and protocol; c. Specify the merging period: Considering the background of big data traffic, the log merging supports log merging tasks with a minimum of 10 minutes and a maximum of 60 minutes, which are entered manually during configuration according to the analysis requirements; Data calculation process a. Offline data calculation: The background starts a calculation task according to the specified log merging type, merging condition, and merging period, polls and grabs the log data of the most recent N days from ElasticSearch for analysis and calculation, and stores the calculation results of each time window in the database; b. Real-time data calculation: The background configures and starts Flink calculation according to the specified merging condition and performs real-time streaming data calculation based on a rolling time window. After each time window calculation in Flink is completed, the data is output and stored in the database; c. Boundary window merging: After the merging task is started, a one-time scheduled task after one merging period is started, and the segment tasks are identified and merged from the offline window data and the real-time window data according to the start time of the calculation task and stored in the database; Calculation result presentation a. Merging task list: Each addition of a calculation task is a task data. On the log merging page, all calculation tasks are first presented in the form of tasks; b. Merging statistics result: The merging results are displayed according to the time window and the merging condition, and the corresponding merging quantity is displayed; c. Merging statistics list: View the group of logs corresponding to the time and merging condition on the log retrieval page. The merging quantity is the same as the number of logs queried on the log retrieval page; d. The log details include the formatted information after log processing and the original log before processing.

Images