Log supplementary collection method and device, electronic equipment and storage medium
By monitoring the difference in line counts between log files and collected files, the system automatically identifies and fills in the missing target log files, thus solving the problem of missing data in multiple log collection tasks and reducing operation and maintenance costs.
Patent Information
- Application Number
- CN202511034117.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-25
- Publication Date
- 2025-11-11
AI Technical Summary
In scenarios involving multiple log collection tasks, existing technologies cannot automatically identify which specific collection task or file is experiencing an anomaly, leading to data loss and requiring manual intervention, which increases operational costs.
By monitoring the difference between the number of lines in the log file and the number of lines in the collected file, the system automatically identifies the target log file with missing data and automatically collects supplementary data according to the time period for supplementary collection, thus avoiding the impact on files that are not missing.
It enables automatic supplementation of missing data under multiple log collection tasks, reducing operation and maintenance costs and minimizing the need for manual intervention.
Smart Images
Figure CN120929330A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a log supplementation collection method, apparatus, electronic device and storage medium. Background Technology
[0002] In real-world business environments, Filebeat, as a log collection tool, can simultaneously collect log data from multiple hosts. When multiple collection tasks run concurrently, the downstream Kafka cluster may experience network fluctuations or performance bottlenecks, causing delays or even blockages in data transmission for some tasks. In this situation, Filebeat can only provide overall collection and transmission metrics and cannot identify which specific collection task or file is experiencing an anomaly. When individual files are missing data, it is usually necessary to stop collection, roll back to the beginning, and then restart collection. While this will repair the lost data, it also causes tasks that did not lose data to repeatedly collect some data. Manual troubleshooting is time-consuming and significantly increases operational costs. Summary of the Invention
[0003] This application provides a log supplementation collection method, apparatus, electronic device, and storage medium to solve the technical problem of how to automatically supplement the collection of only the tasks that are missing in the collection in a scenario with multiple log collection tasks.
[0004] Firstly, this application provides a log supplementation collection method, the method comprising:
[0005] Identify the multiple first log files corresponding to the multi-log collection task;
[0006] For each of the first log files, the following collection status monitoring steps are performed: monitor the number of the first line in the first log file and the number of the second line in the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file to be collected, and the time period for supplementing collection of the target log file is determined;
[0007] The target log file is supplemented with data according to the specified supplementary collection time period.
[0008] Optionally, monitoring the number of the first line in the first log file and the number of the second line in the first collected file includes:
[0009] At each first preset time interval, obtain the first number of lines added to the first log file within the first preset time interval, and obtain the second number of lines added to the first collected file within the first preset time interval;
[0010] Compare the number in the first row with the number in the second row.
[0011] Optionally, when the comparison result between the second row count and the first row count indicates missing collected data, the first log file is used as the target log file to be supplemented for collection, and the supplementary collection time period for the target log file is determined, including:
[0012] Obtain the comparison result between the number of the second row and the number of the first row within a first preset time period;
[0013] When the comparison result indicates that the collected data is missing, obtain the comparison result corresponding to N consecutive first preset durations;
[0014] If N consecutive comparison results all indicate missing collected data, then the first log file is determined as the target log file to be collected.
[0015] The supplementary collection time period for the target log file is determined based on the time periods corresponding to all the missing comparison results in the collected data.
[0016] Optionally, the comparison result indicates missing collected data, including:
[0017] If the number in the second row is different from the number in the first row, then the collected data is determined to be missing;
[0018] or,
[0019] The proportion of missing data is determined based on the number of the second row and the number of the first row; if the proportion of missing data is greater than a preset threshold, then the data is determined to be missing.
[0020] Optionally, supplementary collection is performed on the target log file according to the supplementary collection time period, including:
[0021] Obtain the task of collecting the target log file;
[0022] Determine the target data acquisition task to be executed during the supplementary data acquisition time period within the data acquisition task;
[0023] The target log file is supplemented according to the target collection task.
[0024] Optionally, when the comparison result indicates that the collected data is missing, the method further includes:
[0025] An alarm message is generated; wherein the alarm message is used to indicate that the collected data is missing.
[0026] Optionally, before determining the multiple first log files corresponding to the multi-log collection task, the method further includes:
[0027] Install a status monitor script on the host machine that performs the data acquisition status monitoring, so as to perform data acquisition status monitoring steps on the first log file according to the status monitor script.
[0028] Secondly, this application provides a log supplementation and collection device, the device comprising:
[0029] The determination module is used to determine the multiple first log files corresponding to the multi-log collection task;
[0030] The monitoring module is used to perform the following collection status monitoring steps for each of the first log files: monitoring the number of the first line in the first log file and the number of the second line in the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file for supplementary collection, and the supplementary collection time period of the target log file is determined;
[0031] The supplementary data collection module is used to supplement the collection of data from the target log file according to the supplementary data collection time period.
[0032] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0033] Memory, used to store computer programs;
[0034] When a processor executes a program stored in memory, it implements the log supplementation and collection method described in any embodiment of the first aspect.
[0035] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log supplementation collection method as described in any embodiment of the first aspect.
[0036] Compared with the prior art, the above-mentioned technical solution provided in this application has the following advantages: The method provided in this application determines multiple first log files corresponding to multiple log collection tasks; for each first log file, the following collection status monitoring steps are performed: monitoring the number of the first line of the first log file and the number of the second line of the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file to be collected, and the supplementary collection time period of the target log file is determined; supplementary collection is performed on the target log file according to the supplementary collection time period. This method, in a multi-log collection task scenario, identifies multiple first log files corresponding to the multiple log collection tasks; for each first log file, it monitors the number of the first line and the number of the second line of the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file to be collected, and the time period for supplementing the collection of the target log file is determined. Thus, based on the time period for supplementing the collection, only the target log file is automatically collected, while other first log files that do not have missing collected data are unaffected. Since there is no need for maintenance personnel to manually check whether the collected data is complete and the collection is automatically supplemented when the collected data is missing, the maintenance cost is reduced. Attached Figure Description
[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0040] Figure 1 A system architecture diagram of a log supplementation collection method provided in one embodiment of this application;
[0041] Figure 2 A flowchart illustrating a log supplementation collection method provided in one embodiment of this application;
[0042] Figure 3 A schematic diagram of the structure of a log supplementation and collection device provided in one embodiment of this application;
[0043] Figure 4 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0045] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0046] First, a brief explanation of the existing technology: Filebeat is an open-source log data collector that can collect data from multiple files simultaneously. In production scenarios where Filebeat collects log data and sends it to Kafka, during simultaneous collection by multiple tasks, performance fluctuations in the downstream Kafka can sometimes cause individual tasks to be blocked or even lose data. However, the Filebeat component only monitors overall collection and sending metrics and cannot pinpoint the specific abnormal task and the file being collected. When data collection for individual files is missing, it is usually necessary to stop collection, rewind the start of collection, and then restart collection. While this will repair the lost data, it also causes tasks that did not lose data to repeatedly collect some data. Moreover, the cost of manually troubleshooting abnormal tasks and recollecting data by operations personnel is very high.
[0047] For example, a Filebeat data collector might be tasked with simultaneously collecting data from two files: info.log and warn.log. When downstream data verification detects missing data, it's impossible to pinpoint which file is missing; it's possible only one file is missing. In this case, the solution is to rewind the Filebeat collection time to a point before the data loss occurred and then restart collection. The result is that while the missing data from the missing file is recovered, the other file, which doesn't have missing data, might contain duplicate data.
[0048] To address the technical challenge of automatically supplementing missing data only for tasks in multi-log collection scenarios, this application provides a log supplementation method, apparatus, electronic device, and storage medium. This method automatically supplements data only for the target log file based on the supplementation time period, while other log files without missing data remain unaffected. Since manual checks for data integrity by maintenance personnel are unnecessary and data is automatically supplemented when missing data occurs, maintenance costs are reduced.
[0049] The first embodiment of this application provides a log supplementation collection method, which can be applied to, for example... Figure 1 The system architecture shown includes at least a log data acquisition module 101 and a status monitoring module 102, which establish a communication connection. Specifically, this system architecture can be configured on a host machine in a Kafka cluster. The log data acquisition module 101 can be a Filebeat collector, which collects logs from its corresponding log files according to the collection task. The status monitoring module 102 can be a status monitor, which monitors the number of log lines in the log file and the number of lines successfully collected in the collection file, and compares the two. When the status monitor determines that the log data in the collection file corresponding to a certain log file is missing, it can determine that the log file is a log file that needs to be supplemented for collection. At this time, a supplementary collection task can be generated and sent to the Filebeat collector. After the Filebeat collector loads the supplementary collection task, it can fill in the missing log data, while other collection tasks are unaffected.
[0050] Next, based on this system architecture, the log supplementation collection method will be described in detail, such as... Figure 2 The log supplementation collection method includes:
[0051] Step 201: Determine the multiple first log files corresponding to the multi-log collection task.
[0052] For example, if a multi-log collection task is to collect the info and error files on the host machine 10.127.0.1, the Task Manager will send the collection task to the Filebeat collector on the host machine 10.127.0.1. In this case, there are two log files: an info file and an error file.
[0053] Step 202: For each first log file, perform the following collection status monitoring steps: monitor the number of the first line in the first log file and the number of the second line in the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file to be collected, and the time period for supplementing the collection of the target log file is determined.
[0054] The status monitor can monitor the collection status of each first log file. For example, for an info file, the status monitor can monitor the number of the first line in the info file and the number of the second line in the corresponding first collection file, and compare the number of the first line and the number of the second line. If the comparison result shows that the collected data is missing, the info file can be used as the target log file to be collected, and the time period for supplementing the collection of the info file can be determined.
[0055] In one embodiment, monitoring the number of the first line in the first log file and the number of the second line in the first collected file includes: at each first preset time interval, obtaining the number of the first line added to the first log file within the first preset time interval, and obtaining the number of the second line added to the first collected file within the first preset time interval; comparing the number of the first line and the number of the second line.
[0056] In this embodiment, the first preset duration can be, for example, 1 minute. The status monitor can compare the number of new lines in the first log file (first line count) and the number of new collection lines in the first collection file (second line count) every minute, and compare the number of file lines and the number of collection lines to determine whether there is any missing collection data based on the comparison result.
[0057] In one embodiment, the comparison result characterizes missing data, including: if the number of the second row is different from the number of the first row, then it is determined that the data is missing; or, the proportion of missing data is determined based on the number of the second row and the number of the first row; if the proportion of missing data is greater than a preset threshold, then it is determined that the data is missing.
[0058] In this embodiment, when determining whether collected data is missing, it can be determined based on whether the number of the first row and the number of the second row are the same. For example, if the number of the second row is different from the number of the first row, it is directly determined that the collected data is missing. Alternatively, it can be determined by the proportion of missing collected data determined by the number of the second row and the number of the first row. For example, a preset threshold can be set, such as 10%. If the number of the second row is less than 10% of the number of the first row, it is determined that the collected data is missing. When it is determined that the collected data is missing, a collection missing signal can be sent to the task manager responsible for the collection task. Of course, the preset threshold can be set as needed. This embodiment uses a preset threshold of 10% as an example, which does not represent a limitation on the specific value of the preset threshold.
[0059] In one embodiment, when the comparison result between the second row number and the first row number indicates that the collected data is missing, the first log file is used as the target log file to be collected, and the collection time period of the target log file is determined, including: obtaining the comparison result between the second row number and the first row number within a first preset time period; when the comparison result indicates that the collected data is missing, obtaining the comparison results corresponding to N consecutive first preset time periods; if all N consecutive comparison results indicate that the collected data is missing, then the first log file is determined as the target log file to be collected; and the collection time period of the target log file is determined according to the time period corresponding to all comparison results of the missing collected data.
[0060] In this embodiment, to avoid frequent data collection, supplementary data collection can be performed only when the comparison results for N consecutive first preset durations all indicate missing data. For example, the comparison results of the second row number and the first row number within the first preset duration can be obtained. When the comparison results indicate missing data, the comparison results corresponding to N consecutive first preset durations can be obtained. If all N consecutive comparison results indicate missing data, the first log file is determined as the target log file to be supplemented. The supplementary data collection time period of the target log file is determined according to the time periods corresponding to all comparison results indicating missing data.
[0061] Specifically, for example, if a multi-log collection task collects info and error files on the host machine 10.127.0.1, the Task Manager will send the collection task to the Filebeat collector on the host machine 10.127.0.1. In this case, there are two log files: an info file and an error file.
[0062] The status monitor can monitor the collection status of each first log file. For example, for an info file, the status monitor can monitor the number of the first line of the info file and the number of the second line of the corresponding first collection file, and compare the first and second line counts. For an error file, the status monitor can monitor the number of the first line of the error file and the number of the second line of the corresponding first collection file, and compare the first and second line counts.
[0063] The log supplementation and collection process is as follows:
[0064] At 10:01, the status monitor comparison results showed that the info file had increased by 100 lines and 100 lines had been collected, while the error file had increased by 10 lines and 10 lines had been collected. It was determined that there were no missing data collected during this period.
[0065] At 10:02 AM, the status monitor compared the results and found that the info file had added 10 lines (10 lines collected), and the error file had added 10 lines (5 lines collected). It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0066] At 10:03 AM, the status monitor compared the results and found that the info file had added 10 lines (10 lines collected), and the error file had added 10 lines (2 lines collected). It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0067] At 10:04, the status monitor compared the results: the info file added 10 lines, collecting 10 lines; the error file added 10 lines, collecting 0 lines. It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0068] At 10:05, the status monitor compared the results: the info file added 10 lines, collecting 10 lines; the error file added 10 lines, collecting 0 lines. It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to Task Manager.
[0069] At 10:06, the status monitor compared the results: the info file added 10 lines, collecting 10 lines; the error file added 10 lines, collecting 0 lines. It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0070] The Task Manager has received five consecutive signals indicating missing data collection from the error file on the 10.127.0.1 host machine. This indicates missing data. Therefore, a supplementary data collection task can be generated. For example, the supplementary task could collect data from the error file on the 10.127.0.1 host machine between 10:02 and 10:06 and send it to Kafka. The Task Manager then assigns this supplementary task to the Filebeat data collector. Filebeat loads the new supplementary task, begins execution, and completes the supplementary data collection.
[0071] In this embodiment, only the error file can be automatically collected according to the supplementary collection time period (10:02 to 10:06). Other first log files that have not experienced missing data, such as the info file, are not affected. Since there is no need for maintenance personnel to manually check whether the collected data is complete and the data is automatically supplemented when it is missing, the maintenance cost is reduced.
[0072] In one embodiment, when the comparison result indicates that the collected data is missing, the method further includes: generating an alarm message; wherein the alarm message is used to indicate that the collected data is missing.
[0073] In this embodiment, when the comparison result indicates that the collected data is missing, an alarm message can be generated. The alarm message can indicate that the collected data is missing. Of course, the alarm message can also be a collection missing signal sent to the task manager responsible for the collection task.
[0074] Step 203: Perform supplementary collection on the target log file according to the supplementary collection time period.
[0075] This method, in a multi-log collection task scenario, identifies multiple first log files corresponding to the multiple log collection tasks; for each first log file, it monitors the number of the first line and the number of the second line of the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file to be collected, and the time period for supplementing the collection of the target log file is determined. Thus, based on the time period for supplementing the collection, only the target log file is automatically collected, while other first log files that do not have missing collected data are unaffected. Since there is no need for maintenance personnel to manually check whether the collected data is complete and the collection is automatically supplemented when the collected data is missing, the maintenance cost is reduced.
[0076] In one embodiment, supplementing the collection of a target log file according to the supplementary collection time period includes: obtaining the collection task of the target log file; determining the target collection task to be executed during the supplementary collection time period in the collection task; and supplementing the collection of the target log file according to the target collection task.
[0077] In this embodiment, when supplementing the target log file according to the supplementary collection time period, the collection task of the target log file can be obtained first, the target collection task executed in the supplementary collection time period can be determined, and then the target log file can be supplemented according to the target collection task. That is, the collection task in the supplementary collection time period is repeatedly executed in the target log file to ensure the accuracy of the supplementary collection data.
[0078] In one embodiment, before determining the multiple first log files corresponding to the multiple log collection tasks, the method further includes: installing a status monitor script on the host machine that performs collection status monitoring, so as to perform collection status monitoring steps on the first log files according to the status monitor script.
[0079] In this embodiment, a status monitor script can be installed on the host machine that performs the data collection status monitoring. The data collection status monitoring steps are then performed on the first log file according to the status monitor script. Specifically, a status monitor script can be injected into the Filebeat installation package first. When the host machine installs Filebeat, the status monitor script is registered to run on a scheduled basis. Then, a task manager service is deployed. The task manager can receive signals sent by the status monitor script and can issue supplementary data collection tasks to Filebeat.
[0080] In one specific embodiment, the status monitor can monitor changes in the number of lines in the log file and changes in the number of lines collected. By comparing the two values, it can detect whether collected data is missing. When collected data is missing, a missing data signal is sent to the supplementary collection manager. The supplementary collection manager generates a supplementary collection task based on the missing file and the collection task, and sends the supplementary collection task to the Filebeat collector. The Filebeat collector determines which file and which time range of data to collect based on the supplementary collection task. Specifically, the supplementary collection manager can be the task manager responsible for the collection task. This application uses a timed script to count the increase in the number of lines in the file every minute and obtains the increase in the number of collected lines through the Filebeat monitoring port. By comparing the two values, it concludes whether there is missing collected data. After receiving a missing data signal for 5 consecutive minutes, the supplementary collection manager determines that the file is missing data, then reads the collection configuration of the file, generates a supplementary collection task, and the time range of the supplementary collection is the missing range. The task is sent to Filebeat, and after Filebeat loads the task, it fills in the missing data.
[0081] Specifically, for example, if a multi-log collection task collects info and error files on the host machine 10.127.0.1, the Task Manager will send the collection task to the Filebeat collector on the host machine 10.127.0.1. In this case, there are two log files: an info file and an error file.
[0082] The status monitor can monitor the collection status of each first log file. For example, for an info file, the status monitor can monitor the number of the first line of the info file and the number of the second line of the corresponding first collection file, and compare the first and second line counts. For an error file, the status monitor can monitor the number of the first line of the error file and the number of the second line of the corresponding first collection file, and compare the first and second line counts.
[0083] The log supplementation and collection process is as follows:
[0084] At 10:01, the status monitor comparison results showed that the info file had increased by 100 lines and 100 lines had been collected, while the error file had increased by 10 lines and 10 lines had been collected. It was determined that there were no missing data collected during this period.
[0085] At 10:02 AM, the status monitor compared the results and found that the info file had added 10 lines (10 lines collected), and the error file had added 10 lines (5 lines collected). It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0086] At 10:03 AM, the status monitor compared the results and found that the info file had added 10 lines (10 lines collected), and the error file had added 10 lines (2 lines collected). It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0087] At 10:04, the status monitor compared the results: the info file added 10 lines, collecting 10 lines; the error file added 10 lines, collecting 0 lines. It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0088] At 10:05, the status monitor compared the results: the info file added 10 lines, collecting 10 lines; the error file added 10 lines, collecting 0 lines. It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to Task Manager.
[0089] At 10:06, the status monitor compared the results: the info file added 10 lines, collecting 10 lines; the error file added 10 lines, collecting 0 lines. It was determined that data collection in the error file was incomplete during this period, and the results were uploaded to the Task Manager.
[0090] At this point, the Task Manager has received five consecutive signals indicating missing data collection from the error file on the 10.127.0.1 host machine. This indicates missing data, and a supplementary data collection task can be generated. For example, the supplementary task could collect data from the error file on the 10.127.0.1 host machine between 10:02 and 10:06 and send it to Kafka. The Task Manager then assigns the supplementary task to the Filebeat data collector. Filebeat loads the new supplementary task, begins execution, and completes the supplementary data collection.
[0091] In this embodiment, missing data collection can be automatically detected and reported without manual verification and investigation. Furthermore, a supplementary data collection task can be automatically generated and run without manual operation and maintenance. This embodiment can automatically supplement data collection only for error files based on the supplementary collection time period (10:02–10:06), while other first log files without missing data, such as info files, remain unaffected. Because it eliminates the need for maintenance personnel to manually check the completeness of collected data and automatically supplements missing data, maintenance costs are reduced.
[0092] Based on the same technical concept, the second embodiment of this application provides a log supplementation and collection device, such as... Figure 3 The device includes:
[0093] The determination module 301 is used to determine the multiple first log files corresponding to the multi-log collection task;
[0094] The monitoring module 302 is used to perform the following collection status monitoring steps for each of the first log files: monitoring the number of the first line in the first log file and the number of the second line in the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file for supplementary collection, and the supplementary collection time period of the target log file is determined;
[0095] The supplementary collection module 303 is used to supplement the collection of the target log file according to the supplementary collection time period.
[0096] In a multi-log collection task scenario, this device identifies multiple first log files corresponding to the multi-log collection task. For each first log file, it monitors the number of the first line and the number of the second line of the first collection file. The first collection file is generated based on log collection from the first log file. When the comparison between the number of the second line and the number of the first line indicates missing collected data, the first log file is used as the target log file to be collected, and the time period for supplementing the collection of the target log file is determined. Thus, based on the time period for supplementing the collection, only the target log file is automatically collected, while other first log files without missing collected data are unaffected. Since there is no need for maintenance personnel to manually check whether the collected data is complete and the collection is automatically supplemented when the collected data is missing, the maintenance cost is reduced.
[0097] like Figure 4 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.
[0098] Memory 113 is used to store computer programs;
[0099] In one embodiment of this application, the processor 111, when executing the program stored in the memory 113, implements the log supplementation collection method provided in any of the foregoing method embodiments.
[0100] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0101] The communication interface is used for communication between the aforementioned terminal and other devices.
[0102] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0103] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0104] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the log supplementation collection method provided in any of the foregoing method embodiments.
[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0107] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.
[0108] It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. In the description, suffixes such as "module," "part," or "unit" used to denote elements are used solely for illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.
[0109] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A log supplementation collection method, characterized in that, The method includes: Identify the multiple first log files corresponding to the multi-log collection task; For each of the first log files, the following collection status monitoring steps are performed: monitor the number of the first line in the first log file and the number of the second line in the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file to be collected, and the time period for supplementing collection of the target log file is determined; The target log file is supplemented with data according to the specified supplementary collection time period.
2. The method according to claim 1, characterized in that, Monitoring the first line number of the first log file and the second line number of the first collected file includes: At each first preset time interval, obtain the first number of lines added to the first log file within the first preset time interval, and obtain the second number of lines added to the first collected file within the first preset time interval; Compare the number in the first row with the number in the second row.
3. The method according to claim 2, characterized in that, When the comparison result between the second row count and the first row count indicates missing collected data, the first log file is selected as the target log file to be supplemented for collection, and the supplementary collection time period for the target log file is determined, including: Obtain the comparison result between the number of the second row and the number of the first row within a first preset time period; When the comparison result indicates that the collected data is missing, obtain the comparison result corresponding to N consecutive first preset durations; If N consecutive comparison results all indicate missing collected data, then the first log file is determined as the target log file to be collected. The supplementary collection time period for the target log file is determined based on the time periods corresponding to all the missing comparison results in the collected data.
4. The method according to claim 3, characterized in that, The comparison results indicate missing data, including: If the number in the second row is different from the number in the first row, then the collected data is determined to be missing; or, The proportion of missing data is determined based on the number of the second row and the number of the first row; if the proportion of missing data is greater than a preset threshold, then the data is determined to be missing.
5. The method according to claim 1, characterized in that, Supplementary data collection is performed on the target log file according to the specified supplementary collection time period, including: Obtain the task of collecting the target log file; Determine the target data acquisition task to be executed during the supplementary data acquisition time period within the data acquisition task; The target log file is supplemented according to the target collection task.
6. The method according to claim 3, characterized in that, When the comparison result indicates that the collected data is missing, the method further includes: An alarm message is generated; wherein the alarm message is used to indicate that the collected data is missing.
7. The method according to claim 1, characterized in that, Before determining the multiple first log files corresponding to the multi-log collection task, the method further includes: Install a status monitor script on the host machine that performs the data acquisition status monitoring, so as to perform data acquisition status monitoring steps on the first log file according to the status monitor script.
8. A log supplementation and collection device, characterized in that, The device includes: The determination module is used to determine the multiple first log files corresponding to the multi-log collection task; The monitoring module is used to perform the following collection status monitoring steps for each of the first log files: monitoring the number of the first line in the first log file and the number of the second line in the first collection file; wherein, the first collection file is generated based on log collection from the first log file; when the comparison result between the number of the second line and the number of the first line indicates that the collected data is missing, the first log file is used as the target log file for supplementary collection, and the supplementary collection time period of the target log file is determined; The supplementary data collection module is used to supplement the collection of data from the target log file according to the supplementary data collection time period.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the log supplementation and collection method according to any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the log supplementation and collection method as described in any one of claims 1-7.