Running log analysis method and system based on big data

Abstract

The invention relates to the field of big data, in particular to a running log analysis method and system based on big data, a big data analysis program of a running log is read, if a Map task in any node is analyzed to a preset proportion of to-be-analyzed data, a monitoring process is triggered to collect information of a file which executes the Map task and overflows to a disk, and the information of the file is sent to a server; determining a partition mode according to information of files overflowing to a disk when all the nodes executing the Map task execute the Map task; when the proportion of the analyzed data of the Map task in all the nodes to the remaining to-be-analyzed data is larger than a threshold value, files overflowing to a disk are segmented according to a partitioning mode, and newly overflowing files in the annular memory buffer area are written into new files; the node where each Reduce task is located starts at least one prefetching process, and the prefetching processes prefetch corresponding files according to the segmentation intervals; when all the Map tasks are executed, a Reduce task is newly built, and a new file serves as a partition to be read.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a method and system for analyzing operation logs based on big data. Background Art

[0002] Logs record various system operations and operation records in detail. A large amount of log information is generated in manufacturing systems, IoT systems, and security systems, such as startup, shutdown, crashes, warnings, and program operation information. Log analysis helps monitor the operating status of the system, discover and solve potential problems in a timely manner, and ensure the stable operation of the system. In addition, by monitoring and analyzing logs, abnormal behaviors and potential security threats, such as unauthorized access and data leakage, can be discovered, so that corresponding security measures can be taken.

[0003] Big data technology can process massive amounts of log data and extract valuable information from it, but log data is generated very quickly, especially in a high-concurrency environment, where the amount of data grows at a rate of several megabytes or even tens of megabytes per second. In addition, log data itself is highly diverse, including system logs, application logs, security logs, etc. These different types of data require complex preprocessing and parsing, further increasing the complexity and time overhead of processing. In addition, when the existing big data processing framework is performing data transmission (such as the Shuffle stage), network bandwidth and transmission delay become bottlenecks, and a large amount of data needs to be exchanged and sorted between different nodes, resulting in a significant increase in processing time. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a big data-based operation log analysis method, which comprises the following steps: A big data analysis program that reads the running log, when running to a Map task, if the Map task in any node analyzes a preset proportion of the data to be analyzed, the monitoring process is triggered to collect information about files overflowed to the disk when executing the Map task, and the partitioning method is determined based on the information about files overflowed to the disk when all nodes executing the Map task execute the Map task; When the ratio of the data that has been analyzed by the Map task to the remaining data to be analyzed in all nodes is greater than the threshold, the files overflowed to the disk are divided according to the partitioning method, and the newly overflowed files in the ring memory buffer are written into new files; At least one pre-fetch process is started at the node where each Reduce task is located. The pre-fetch process pre-fetches the corresponding files according to the partition interval. When all Map tasks are executed, a new Reduce task is created and the new file is read as a partition. When the big data analysis program is finished running, the log analysis results are written to the disk.

[0005] Preferably, the partitioning method is determined according to the information of the files overflowed to the disk when all nodes executing the Map task execute the Map task, specifically: The master node collects the idle status of each node executing the Reduce task, and determines the ratio based on the idle status; The monitoring process on each node executing the Map task sends the file information to the master node, and the master node aggregates the information of all files and determines the segmentation identifier set according to the ratio; The file overflowed to the disk is divided into at least two intervals according to the segmentation identifier set.

[0006] Preferably, the master node aggregates information of all files and determines a segmentation identifier set according to the ratio, specifically: All the files overflowed to the disk are merged, and the merged files are segmented according to the ratio to obtain the key of each segmentation point, and the keys of all the segmentation points are put into the segmentation identification set.

[0007] Preferably, the master node aggregates information of all files and determines a segmentation identifier set according to the ratio, specifically: Obtain the size of the file overflowed to the disk from the file information to obtain the total file size, and obtain the size of each interval according to the ratio; Splitting each file overflowed to the disk by executing the Map task according to the ratio to obtain a key set of split points and then obtain multiple key sets; Traversing each key set in the plurality of key sets, segmenting each file according to the key set, calculating the total size of each interval in all the file segmentation results, calculating the ratio of the total sizes of each interval, and obtaining a key set whose ratio is closest to the ratio; The key set whose ratio is closest to the proportion is used as the segmentation identifier set.

[0008] Preferably, the analysis program includes at least one MapReduce stage, and each MapReduce stage includes at least a Map stage program and a Reduce stage program; when the Map stage program is running, it is called a Map task, and when the Reduce stage program is running, it is called a Reduce task; the Map task runs on multiple nodes, and the Reduce task runs on multiple nodes.

[0009] On the other hand, the present invention provides an operation log analysis system based on big data, the system comprising the following modules: A partitioning module is a big data analysis program for reading running logs. When running to a Map task, if the Map task in any node analyzes a preset proportion of the data to be analyzed, the monitoring process is triggered to collect information about files overflowed to the disk when executing the Map task, and the partitioning method is determined based on the information about files overflowed to the disk when all nodes executing the Map task execute the Map task; The overflow file segmentation module is used to segment the files overflowed to the disk according to the partitioning method when the ratio of the data analyzed by the Map task to the remaining data to be analyzed in all nodes is greater than the threshold, and write the newly overflowed files in the ring memory buffer into new files; The result merging module is used to start at least one pre-fetch process at each node where the Reduce task is located, and the pre-fetch process pre-fetches the corresponding files according to the partition interval; when all Map tasks are executed, a new Reduce task is created, and the new file is read as a partition; when the big data analysis program is finished running, the log analysis results are written to the disk.

[0010] Preferably, the partitioning method is determined according to the information of the files overflowed to the disk when all nodes executing the Map task execute the Map task, specifically: The master node collects the idle status of each node executing the Reduce task, and determines the ratio based on the idle status; The monitoring process on each node executing the Map task sends the file information to the master node, and the master node aggregates the information of all files and determines the segmentation identifier set according to the ratio; The file overflowed to the disk is divided into at least two intervals according to the segmentation identifier set.

[0011] Preferably, the master node aggregates information of all files and determines a segmentation identifier set according to the ratio, specifically: All the files overflowed to the disk are merged, and the merged files are segmented according to the ratio to obtain the key of each segmentation point, and the keys of all the segmentation points are put into the segmentation identification set.

[0012] Preferably, the master node aggregates information of all files and determines a segmentation identifier set according to the ratio, specifically: Obtain the size of the file overflowed to the disk from the file information to obtain the total file size, and obtain the size of each interval according to the ratio; Splitting each file overflowed to the disk by executing the Map task according to the ratio to obtain a key set of split points and then obtain multiple key sets; Traversing each key set in the plurality of key sets, segmenting each file according to the key set, calculating the total size of each interval in all the file segmentation results, calculating the ratio of the total sizes of each interval, and obtaining a key set whose ratio is closest to the ratio; The key set whose ratio is closest to the proportion is used as the segmentation identifier set.

[0013] Preferably, the analysis program includes at least one MapReduce stage, and each MapReduce stage includes at least a Map stage program and a Reduce stage program; when the Map stage program is running, it is called a Map task, and when the Reduce stage program is running, it is called a Reduce task; the Map task runs on multiple nodes, and the Reduce task runs on multiple nodes.

[0014] Finally, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented.

[0015] In order to improve the speed of big data analysis of system operation logs, the present invention determines the partitioning method of overflowed disk files based on the analyzed data before the Map task is fully executed. At the same time, when the ratio of the data analyzed by the Map task to the remaining data to be analyzed in all nodes is greater than the threshold, the files overflowed to the disk are divided according to the partitioning method, and the newly overflowed files of the ring memory buffer are written into new files; each node where the Reduce task is located starts at least one pre-fetch process, and the pre-fetch process pre-fetches the corresponding files according to the segmentation interval; when all Map tasks are executed, a new Reduce task is created, and the new file is read as a partition. The parallelism of Map tasks and Reduce tasks is improved, and the data skew of Reduce tasks can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of Embodiment 1; Figure 2 This is a schematic diagram of the MapReduce process; Figure 3 This is a structural diagram of the second embodiment. DETAILED DESCRIPTION

[0017] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the first embodiment, the present invention provides a method for analyzing operation logs based on big data, such as Figure 1 As shown, the method comprises the following steps: S1, a big data analysis program that reads the running log, when running to the Map task, if the Map task in any node analyzes a preset proportion of the data to be analyzed, the monitoring process is triggered to collect information about files overflowed to the disk when executing the Map task, and the partitioning method is determined according to the information about files overflowed to the disk when all nodes executing the Map task execute the Map task; The operation log records the log information generated by the operation of the system. After completing the log analysis program according to the requirements, the big data analysis program is submitted to the big data platform. The big data frameworks used include but are not limited to Hadoop and Spark. The big data analysis program is a log analysis program written according to the requirements of Hadoop or Spark. After being submitted to the management node (ResourceManager) or the master node (Master), the working node or computing node will identify the Map task in the analysis program. When the node runs to the Map task, it will determine the ratio of the data that has been analyzed by the Map task to the data to be analyzed. The node running the Map task will have data to be analyzed. The data to be analyzed is the data to be analyzed by the node when executing the Map task. The size of the data to be analyzed corresponding to different nodes is not exactly the same. During the execution process, the data to be analyzed is continuously analyzed. When the ratio of the data to be analyzed reaches the preset ratio, the monitoring process on the computing node will be triggered to collect information about the files that the node executing the Map task overflows (Spill) to the disk, such as Figure 2 As shown ( Figure 2 Only one Reduce task is shown), wherein the monitoring process can be running on each node that executes the Map task; it can also be running on the master node; it can also be running on the node that executes the Map task and the master node at the same time, but the functions of the two are different. In a specific embodiment, the preset ratio is 60% or 80%. When any node running a Map task reaches the preset ratio, it indicates that the Map task is about to be completed and is about to enter the Shuffle stage. In order to improve the reading speed of the Shuffle and the balance of each Reduce task, the splitting method of the files overflowed to the disk is determined. Among them, the data to be analyzed refers to all the data to be analyzed by the Map task. For example, if a Map task needs to analyze 500M of data, then this 500M is the data to be analyzed.

[0020] Since the Shuffle fetching phase has not yet begun when the preset ratio is reached, performing various operations on the overflow to disk will not prolong the Shuffle duration. Instead, by determining the partitioning method, each Reduce task can be as balanced as possible, which can reduce the execution time of the Reduce phase. In a specific embodiment, the partitioning method is determined based on the information of the files overflowed to the disk when all nodes executing the Map task execute the Map task, specifically: The master node collects the idle status of each node executing the Reduce task, and determines the ratio based on the idle status; The monitoring process on each node executing the Map task sends the file information to the master node, and the master node aggregates the information of all files and determines the segmentation identifier set according to the ratio; The file overflowed to the disk is divided into at least two intervals according to the segmentation identifier set.

[0021] There are multiple nodes that will execute Map tasks at the same time. Different nodes have different idle conditions. More data is allocated to relatively idle nodes. Collect the idle conditions of each node that will execute Map tasks, and then determine the ratio based on the idle conditions. For example, if there are 3 Reduce tasks executed on 3 nodes, the idle ratio of these 3 nodes is 3:1:2, that is, the first node is the idlest, the third node is the second idlest, and the second node is the least idle. When splitting the files that overflow to the disk, a better splitting method is to split them according to 3:1:2. The splitting described here is the splitting of the sum of all the files that overflow to the disk when executing Map tasks, not the splitting of the files that overflow to the disk when a certain node that executes Map tasks. For example, if there are 4 nodes that execute Map tasks, and the files that overflow to the disk are File1, File2, File3, and File4, File1+File2+File3+File4 are split as a whole into 3:1:2, rather than splitting each of File1, File2, File3, and File4 into 3:1:2.

[0022] The master node aggregates all file information to obtain a split identifier set, for example, the split identifier set is {Key100, Key180}. For each node that executes the Map task, the files that overflow to the disk will be split according to the split identifier set. For example, Key1-Key100 of File1 is divided into one area, Key101-Key180 is divided into one area, and Key181-Key1000 is divided into one area. Similarly, the above splitting method will be used for File2-File4. There are three Reduce tasks. The first Reduce task will read the first area in File1-File4, the second Reduce task will read the second area in File1-File4, and the third Reduce task will read the third area in File1-File4.

[0023] There are many ways to determine the segmentation mark set according to the ratio. When the preset ratio is relatively small, it will take a long time to complete all the Map tasks. At this time, the files overflowed to the disk can be read. At this time, the master node summarizes the information of all files and determines the segmentation mark set according to the ratio, specifically: All the files overflowed to the disk are merged, and the merged files are segmented according to the ratio to obtain the key of each segmentation point, and the keys of all the segmentation points are put into the segmentation identification set.

[0024] Merge the files overflowed to disk by the Map tasks in all nodes running Map tasks into a large file, and then sort the contents of the large file by key, and split the sorted large file according to the ratio, that is, the merging process will perform sort operations, but no combine operations. For example, if the large file is 300M and the split ratio is 3:1:2, the first split area is 150M, and the split point is assumed to be Key100, the second split area is 50M, and the split point is assumed to be Key800, and the remaining 100M is the third split area, then the split identifier set is Key100, Key800.

[0025] After obtaining the segmentation identifier set, each file that overflows to the disk when executing the Map task will be segmented according to the identifiers in the segmentation identifier set. For example, if the file that overflows to the disk when the first node executing the Map task is File1, Key100 and Key800 will be found to segment File1 into three parts. Of course, for File1, the ratio of these three parts is not necessarily equal to the ratio determined based on the idle situation, but the ratio of the sum of the first part, the sum of the second part, and the sum of the third part of File1-File4 is equal to the ratio determined based on the idle situation.

[0026] Although the above method is accurate in segmentation, since a large part of the data to be analyzed has not been executed, the files overflowed to the disk at this time are quite different from the files overflowed to the disk after all the Map tasks are executed. It may happen that although the files segmented by the segmentation identifier set are similar to the ratio, the difference between the files finally overflowed to the disk by the segmentation identifier set and the ratio is large. When the preset ratio is relatively large, the Map task is basically completed. At this time, the accuracy will be improved, but there is not much time to determine the segmentation identifier set. In one embodiment, the master node summarizes the information of all files and determines the segmentation identifier set according to the ratio, specifically: Obtain the size of the file overflowed to the disk from the file information to obtain the total file size, and obtain the size of each interval according to the ratio; Each Map task may spill to disk during execution. Get the total file size of all Map tasks spilled to disk and determine the size of each interval according to the ratio. For example, if there are two Map tasks, one spills to disk with 100M and the other spills to disk with 400M, the total file size is 200M. For the files spilled to disk by each Map task, according to the ratio of 3:1:2, the size of the first interval is 300M, the size of the second interval is 100M, and the size of the third interval is 200M.

[0027] Splitting each file overflowed to the disk by executing the Map task according to the ratio to obtain a key set of split points and then obtain multiple key sets; Divide the files that overflow to disk from each Map task according to the ratio. Assuming there are two Map tasks, each task corresponds to an overflow file. Divide the overflow file according to the ratio, and there will be two corresponding key sets. The first key set corresponds to the files that overflow to disk from the first Map task, and the second key set corresponds to the files that overflow to disk from the second Map task. Each key set records the split position of the files that overflow to disk from the corresponding Map task. For example, the key set corresponding to the first Map task is (Key10, Key58), and the key set corresponding to the second Map task is (Key14, Key63).

[0028] Traversing each key set in the plurality of key sets, segmenting each file according to the key set, calculating the total size of each interval in all the file segmentation results, calculating the ratio of the total sizes of each interval, and obtaining a key set whose ratio is closest to the ratio; The key set whose ratio is closest to the proportion is used as the segmentation identifier set.

[0029] Traverse multiple key sets, that is, perform the following operations on each key set in turn: split the files overflowed to disk by each Map task according to the key set, and then calculate the total size of the same interval of files overflowed to disk by all Map tasks. For example, for the first key set (Key10, Key58), there are three intervals of files overflowed to disk by splitting the first Map task, with sizes of (10M, 51M, 41M), and there are three intervals of files overflowed to disk by splitting the second Map task, with sizes of (18M, 32M, 56M), and the total size of each interval is (28M, 83M, 97M), with a ratio of 28:83:97. Similarly, for the second key set, the total size of each interval is (34M, 96M, 78M), with a ratio of 34:96:78. If the idle ratio of the three nodes is 1:3:2, it is obvious that the ratio corresponding to the second key set is closest to the ratio, so the second key set is used as the split identifier set. This method avoids data duplication between different nodes, and because this embodiment is executed when the preset ratio is relatively large, the ratio of files overflowed to disk after all Map tasks are executed is close to the division method of this embodiment.

[0030] In a more specific embodiment, if the preset ratio is greater than 80%, it is considered that the preset ratio is relatively large, and if the preset ratio is less than or equal to 80%, it is considered that the preset ratio is relatively small. In another embodiment, the execution of the above two embodiments is determined according to the size of the preset ratio. If the preset ratio is less than or equal to 80%, the former embodiment is executed, otherwise the latter embodiment is executed.

[0031] S2, when the ratio of the data that has been analyzed by the Map task to the remaining data to be analyzed in all nodes is greater than the threshold, the files overflowed to the disk are divided according to the partitioning method, and the files newly overflowed from the ring memory buffer are written into new files; In order to enable the Map task and the Reduce task to be executed in parallel, when the ratio of the data analyzed by all nodes executing the Map task to the remaining data to be analyzed by the Map task is greater than the threshold, the files overflowing to the disk are segmented according to the partitioning method, and the newly overflowed files in the ring memory buffer are written into new files. Among them, when the ratio of the data analyzed by the Map task in all nodes to the remaining data to be analyzed is greater than the threshold, there are two implementation methods, one is that the ratio of the sum of the data volume analyzed by all Map tasks to the sum of the data volume not analyzed is greater than the threshold; the other is that the ratio of the data volume analyzed by each or any one of the Map tasks in all nodes to the remaining data to be analyzed is greater than the threshold, and the remaining data to be analyzed is the data corresponding to the Map task that has not been analyzed.

[0032] When the ratios are all greater than the threshold, if there are overflowing files written to the original files, after the overflowing files are written to the original files, all Map tasks will stop overflowing the disk to the original files and overflow to the new files. S3, each node where the Reduce task is located starts at least one pre-fetch process, and the pre-fetch process pre-fetches the corresponding files according to the partition interval; when all Map tasks are executed, a new Reduce task is created, and the new file is read as a partition; when the big data analysis program is finished running, the log analysis results are written to the disk.

[0033] After the file of each Map task is split, the node where the Reduce task is located will start one or more pre-fetch processes, and read the corresponding interval files from the split files according to the correspondence between the Reduce task and the split interval. For example, the third Reduce task is on node 2, and the third Reduce task processes the data of interval 2. Then node 2 will start the pre-fetch process and read the second interval data of each split file.

[0034] For the data in the new file, a new Reduce task will be created. The node of the new Reduce task can be any node, which is not specifically limited here. For the same Map task, after it is executed on all nodes, a new Reduce task will be triggered. This new Reduce task will read the content of the new file corresponding to each Map task. Since the content of the new file is less, this process will be executed faster, so that the entire execution time of the Reduce task including the new Reduce task will be shortened. In the present invention, unless otherwise specified, the Map task is the same Map program code or the same Map stage in the analysis program, but multiple Map tasks appear due to running on different nodes at the same time; similarly, the Reduce task is also the same Reduce program code or the same Reduce stage in the analysis program, but multiple Map tasks appear due to running on different nodes at the same time. A big data analysis program for a log may include multiple MapReduce stages. The above-mentioned Reduce task and Map task appear in the same MapReduce stage. When the Map stage program is running, it is called a Map task, and when the Reduce stage program is running, it is called a Reduce task; the Map task runs on multiple nodes, and the Reduce task runs on multiple nodes. When the big data analysis program is finished running, the analysis results are written to the disk, so that the analysis results can be displayed on the monitor through front-end technologies such as UI.

[0035] In the second embodiment, the present invention provides an operation log analysis system based on big data, such as Figure 3 As shown, the system includes the following modules: A partitioning module is a big data analysis program for reading running logs. When running to a Map task, if the Map task in any node analyzes a preset proportion of the data to be analyzed, the monitoring process is triggered to collect information about files overflowed to the disk when executing the Map task, and the partitioning method is determined based on the information about files overflowed to the disk when all nodes executing the Map task execute the Map task; The overflow file segmentation module is used to segment the files overflowed to the disk according to the partitioning method when the ratio of the data analyzed by the Map task to the remaining data to be analyzed in all nodes is greater than the threshold, and write the newly overflowed files in the ring memory buffer into new files; The result merging module is used to start at least one pre-fetch process at each node where the Reduce task is located, and the pre-fetch process pre-fetches the corresponding files according to the partition interval; when all Map tasks are executed, a new Reduce task is created, and the new file is read as a partition; when the big data analysis program is finished running, the log analysis results are written to the disk.

[0036] Preferably, the partitioning method is determined according to the information of the files overflowed to the disk when all nodes executing the Map task execute the Map task, specifically: The master node collects the idle status of each node executing the Reduce task, and determines the ratio based on the idle status; The monitoring process on each node executing the Map task sends the file information to the master node, and the master node aggregates the information of all files and determines the segmentation identifier set according to the ratio; The file overflowed to the disk is divided into at least two intervals according to the segmentation identifier set.

[0037] Preferably, the master node aggregates information of all files and determines a segmentation identifier set according to the ratio, specifically: All the files overflowed to the disk are merged, and the merged files are segmented according to the ratio to obtain the key of each segmentation point, and the keys of all the segmentation points are put into the segmentation identification set.

[0038] Preferably, the master node aggregates information of all files and determines a segmentation identifier set according to the ratio, specifically: Obtain the size of the file overflowed to the disk from the file information to obtain the total file size, and obtain the size of each interval according to the ratio; Splitting each file overflowed to the disk by executing the Map task according to the ratio to obtain a key set of split points and then obtain multiple key sets; Traversing each key set in the plurality of key sets, segmenting each file according to the key set, calculating the total size of each interval in all the file segmentation results, calculating the ratio of the total sizes of each interval, and obtaining a key set whose ratio is closest to the ratio; The key set whose ratio is closest to the proportion is used as the segmentation identifier set.

[0039] Preferably, the analysis program includes at least one MapReduce stage, and each MapReduce stage includes at least a Map stage program and a Reduce stage program; when the Map stage program is running, it is called a Map task, and when the Reduce stage program is running, it is called a Reduce task; the Map task runs on multiple nodes, and the Reduce task runs on multiple nodes.

[0040] In a third embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first embodiment is implemented.

[0041] Through the description of the above implementation methods, technicians in this field can clearly understand that each implementation method can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for analyzing operation logs based on big data, characterized in that: The method comprises the following steps: A big data analysis program that reads the running log, when running to a Map task, if the Map task in any node analyzes a preset proportion of the data to be analyzed, the monitoring process is triggered to collect information about files overflowed to the disk when executing the Map task, and the partitioning method is determined based on the information about files overflowed to the disk when all nodes executing the Map task execute the Map task; When the ratio of the data that has been analyzed by the Map task to the remaining data to be analyzed in all nodes is greater than the threshold, the files overflowed to the disk are divided according to the partitioning method, and the newly overflowed files in the ring memory buffer are written into new files; At least one pre-fetch process is started at the node where each Reduce task is located. The pre-fetch process pre-fetches the corresponding files according to the partition interval. When all Map tasks are executed, a new Reduce task is created and the new file is read as a partition. When the big data analysis program is finished running, the log analysis results are written to the disk.

2. The method according to claim 1, characterized in that The partitioning method is determined according to the information of the files overflowed to the disk when all nodes executing the Map task execute the Map task, specifically: The master node collects the idle status of each node executing the Reduce task, and determines the ratio based on the idle status; The monitoring process on each node executing the Map task sends the file information to the master node, and the master node aggregates the information of all files and determines the segmentation identifier set according to the ratio; The file overflowed to the disk is divided into at least two intervals according to the segmentation identifier set.

3. The method according to claim 2, characterized in that The master node aggregates the information of all files and determines a segmentation identifier set according to the ratio, specifically: All the files overflowed to the disk are merged, and the merged files are segmented according to the ratio to obtain the key of each segmentation point, and the keys of all the segmentation points are put into the segmentation identification set.

4. The method according to claim 2, characterized in that The master node aggregates the information of all files and determines a segmentation identifier set according to the ratio, specifically: Obtain the size of the file overflowed to the disk from the file information to obtain the total file size, and obtain the size of each interval according to the ratio; Splitting each file overflowed to the disk by executing the Map task according to the ratio to obtain a key set of split points and then obtain multiple key sets; Traversing each key set in the plurality of key sets, segmenting each file according to the key set, calculating the total size of each interval in all the file segmentation results, calculating the ratio of the total sizes of each interval, and obtaining a key set whose ratio is closest to the ratio; The key set whose ratio is closest to the proportion is used as the segmentation identifier set.

5. The method according to claim 1, characterized in that The analysis program includes at least one MapReduce stage, and each MapReduce stage includes at least a Map stage program and a Reduce stage program; when the Map stage program is running, it is called a Map task, and when the Reduce stage program is running, it is called a Reduce task; the Map task runs on multiple nodes, and the Reduce task runs on multiple nodes.

6. A big data-based operation log analysis system, characterized in that: The system includes the following modules: A partitioning module is a big data analysis program for reading running logs. When running to a Map task, if the Map task in any node analyzes a preset proportion of the data to be analyzed, the monitoring process is triggered to collect information about files overflowed to the disk when executing the Map task, and the partitioning method is determined based on the information about files overflowed to the disk when all nodes executing the Map task execute the Map task; The overflow file segmentation module is used to segment the files overflowed to the disk according to the partitioning method when the ratio of the data analyzed by the Map task to the remaining data to be analyzed in all nodes is greater than the threshold, and write the newly overflowed files in the ring memory buffer into new files; The result merging module is used to start at least one pre-fetching process on each node where the Reduce task is located, and the pre-fetching process pre-fetches the corresponding files according to the segmentation interval; When all Map tasks are executed, a new Reduce task is created to read the new file as a partition; when the big data analysis program is finished running, the log analysis results are written to the disk.

7. The system according to claim 6, characterized in that The partitioning method is determined according to the information of the files overflowed to the disk when all nodes executing the Map task execute the Map task, specifically: The master node collects the idle status of each node executing the Reduce task, and determines the ratio based on the idle status; The monitoring process on each node executing the Map task sends the file information to the master node, and the master node aggregates the information of all files and determines the segmentation identifier set according to the ratio; The file overflowed to the disk is divided into at least two intervals according to the segmentation identifier set.

8. The system according to claim 7, characterized in that The master node aggregates the information of all files and determines a segmentation identifier set according to the ratio, specifically: All the files overflowed to the disk are merged, and the merged files are segmented according to the ratio to obtain the key of each segmentation point, and the keys of all the segmentation points are put into the segmentation identification set.

9. The system according to claim 7, characterized in that The master node aggregates the information of all files and determines a segmentation identifier set according to the ratio, specifically: Obtain the size of the file overflowed to the disk from the file information to obtain the total file size, and obtain the size of each interval according to the ratio; Splitting each file overflowed to the disk by executing the Map task according to the ratio to obtain a key set of split points and then obtain multiple key sets; Traversing each key set in the plurality of key sets, segmenting each file according to the key set, calculating the total size of each interval in all the file segmentation results, calculating the ratio of the total sizes of each interval, and obtaining a key set whose ratio is closest to the ratio; The key set whose ratio is closest to the proportion is used as the segmentation identifier set.

10. A computer storage device, wherein a computer program is stored on the storage device, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.

Images