A data acquisition monitoring method, device and medium based on InfluxDB database

Abstract

The present application discloses a data collection monitoring method, device and medium based on InfluxDB database, the method includes: determining the data collection task to be executed, the content of the data collection task includes downloading pages and extracting structured data; obtaining the log information of the execution process of the data collection task within the preset time period at each preset time interval; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period; obtaining the actual collection speed of the data collection task according to the log information; judging whether the data collection task is abnormal according to the speed threshold in the pre-set monitoring information and the actual collection speed, the monitoring information is configured according to the data collection task. It can help establish a complete monitoring mechanism for the task collection process, help data collection personnel to promptly discover collection anomalies such as download failure and download error page during the data collection process, and provide timely feedback and repair.

Description

Technical Field

[0001] The present application relates to the field of data acquisition, and specifically to a data acquisition and monitoring method, device and medium based on an InfluxDB database. Background Art

[0002] In the era of big data, more and more companies have established big data platforms to meet various data requirements. Big data platforms are inseparable from data. The generation of data requires data collectors to collect data. The data collection process is inevitably accompanied by download failures, error pages, speed reduction and other problems. If a series of problems faced in data collection cannot be discovered and handled in time, the progress of task collection may be slowed down, thereby affecting work efficiency. Therefore, how to discover data collection problems in a timely manner and provide data collectors with an effective data collection monitoring method has become a problem that needs to be studied and solved urgently. Based on this, the present invention proposes a data collection monitoring method, device and medium based on InfluxDB database. Summary of the invention

[0003] In order to solve the above problems, the present application proposes a data collection and monitoring method, device and medium based on the InfluxDB database, the method comprising:

[0004] Determine a data collection task to be executed, wherein the content of the data collection task includes downloading a page and extracting structured data; obtain log information during the execution of the data collection task within the preset time period at intervals of a preset time period; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period; obtain the actual collection speed of the data collection task according to the log information; determine whether the data collection task is abnormal according to a speed threshold in the preset monitoring information and the actual collection speed, and the monitoring information is configured according to the data collection task.

[0005] In one example, after obtaining the log information during the execution of the data collection task within the preset time period, the method further includes: transmitting the log information to the InfluxDB database, so as to obtain the actual collection speed by utilizing the characteristics of the InfluxDB database that supports time series functions and can calculate data in real time.

[0006] In one example, after determining whether the data collection task is abnormal, the method further includes:

[0007] Determine that the data collection task is abnormal; send an alarm email to the mailbox of the data collection personnel to remind the data collection personnel to handle the abnormality.

[0008] In one example, the method further includes: determining a monitoring cycle, each data acquisition task corresponds to a monitoring cycle; the monitoring cycle includes a start time and an end time, which are used to limit the start time and the end time of the data acquisition task; the time between two inspections of the data acquisition task is determined as the inspection cycle, and based on the inspection cycle, periodically judging whether the data acquisition task has any abnormalities.

[0009] In one example, the actual acquisition speed of the data acquisition task is calculated according to the log information, specifically including: determining the time period from the start time of the monitoring cycle to the current time as the acquisition time; using the data acquisition amount C1 within the acquisition time and the acquisition time S1 to The actual acquisition speed V1 is obtained.

[0010] In one example, the method further includes: recording the actual acquisition speed in the previous preset time period of the data acquisition task as the historical acquisition speed in the current preset time period of the data acquisition task; comparing the actual acquisition speed with the historical acquisition speed to obtain a first difference; if the first difference exceeds a first preset threshold, it is considered that the acquisition task is abnormal.

[0011] In one example, the method further includes: estimating the task size of the data collection task to obtain an estimated data collection amount; using the estimated data collection amount C2 and the monitoring period S2 to Obtain a theoretical acquisition speed V2; compare the theoretical acquisition speed V2 with the actual acquisition speed V1 to obtain a second difference; if the second difference exceeds a second preset threshold, it is considered that the acquisition task is abnormal.

[0012] In one example, the method further includes: comparing the actual start time with the start time of the monitoring period to obtain a third difference, and if the third difference exceeds a third preset threshold, it is considered that the acquisition task is started abnormally.

[0013] The present application also provides a data acquisition and monitoring device based on the InfluxDB database, including:

[0014] at least one processor; and a memory in communication with the at least one processor; wherein,

[0015] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute:

[0016] Determine a data collection task to be performed, wherein the data collection task includes downloading a page and extracting structured data;

[0017] At every preset time period, log information of the execution process of the data collection task within the preset time period is obtained; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period;

[0018] Obtaining the actual acquisition speed of the data acquisition task according to the log information;

[0019] According to the speed threshold value in the preset monitoring information and the actual acquisition speed, it is determined whether the data acquisition task is abnormal, and the monitoring information is configured according to the data acquisition task.

[0020] The present application also provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0021] Determine a data collection task to be performed, wherein the data collection task includes downloading a page and extracting structured data;

[0022] At every preset time period, log information of the execution process of the data collection task within the preset time period is obtained; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period;

[0023] Obtaining the actual acquisition speed of the data acquisition task according to the log information;

[0024] According to the speed threshold value in the preset monitoring information and the actual acquisition speed, it is determined whether the data acquisition task is abnormal, and the monitoring information is configured according to the data acquisition task.

[0025] This application can help establish a complete monitoring mechanism for the task collection process, help data collectors promptly discover collection anomalies such as download failures and download error pages during the data collection process, and provide timely feedback and repairs, thereby ensuring the smooth progress of data collection work. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 This is a schematic diagram of a data collection and monitoring method based on the InfluxDB database in an embodiment of the present application;

[0028] Figure 2 This is a schematic diagram of a data acquisition and monitoring device based on the InfluxDB database in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0030] The present invention proposes a data collection monitoring method, device and medium based on InfluxDB database. Data collection mainly includes downloading pages and extracting structured data. According to the download log in the data collection process, the real-time data download amount and data extraction amount in the data collection process are stored. The InfluxDB database supports time-related related functions and can calculate a large amount of data in real time. After calculating the download speed and extraction speed, and comparing them with the minimum threshold of the collection speed set by the collection personnel or the threshold set by historical data, it is determined whether the data collection task is abnormal, and finally the purpose of monitoring data collection and issuing early warning is achieved.

[0031] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, the embodiment of the present application provides a method, comprising:

[0033] S101, determining a data collection task to be executed, wherein the data collection task includes downloading a page and extracting structured data;

[0034] S102, every preset time period, obtaining log information during the execution of the data collection task within the preset time period; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period;

[0035] Among them, the preset time period refers to the interval time for obtaining log information, which can be set according to the different types of data collection tasks. For example, it can be set to obtain once every five minutes, or once every half an hour. Log information may include task start time, data download volume, data extraction volume, task end time and other information. The first preset time period here is preferably set to be shorter. If the time is longer, such as obtaining log information in a data collection task once a day, abnormalities in the data collection task will not be discovered in time, and the accuracy of the log information cannot be guaranteed. Obtaining the log information of data collection at a fixed interval each time can timely update the information of the downloaded data, providing a basis for timely discovery of abnormalities in data collection.

[0036] S103: Calculate the actual collection speed of the data collection task according to the log information.

[0037] The actual acquisition speed mentioned here includes download speed and extraction speed, which refers to the download speed or extraction speed from the time the task is set to the present, not the instantaneous speed. If the first preset time is set short enough, such as obtaining log information of the data acquisition task every ten seconds or even shorter, the data download speed or data extraction speed at this time can also be regarded as the instantaneous speed of the current data download or extraction.

[0038] S103, judging whether the data collection task is abnormal according to a speed threshold value preset in the monitoring information and the actual collection speed, wherein the monitoring information is configured according to the data collection task.

[0039] The monitoring information here refers to the type of information that needs to be monitored in the log information. For example, the monitoring information may include the speed threshold, which is the minimum threshold of the current acquisition speed, which is the minimum threshold of the actual download speed and the minimum threshold of the actual extraction speed. After configuring the monitoring information, the monitoring information is monitored to determine whether the data collection task is proceeding normally.

[0040] After calculating the actual download speed and actual extraction speed, they are compared with the minimum threshold of actual download speed and the minimum threshold of actual extraction speed in the configured monitoring information. If the actual download speed and the actual extraction speed are lower than the set minimum threshold, it is considered that the current data collection task is abnormal and needs to be processed by the data collection personnel.

[0041] In one embodiment, since the data collection process requires both the time sequence of data collection to be marked and the real-time calculation of a large amount of data to be performed, the acquired log information can be transmitted to the InfluxDB database by utilizing the characteristics of supporting time-related functions and being able to calculate a large amount of data in real time, making it more convenient to mark the time sequence and obtain the actual collection speed.

[0042] In one embodiment, the data collection process is inevitably accompanied by problems such as download failure, error page, and speed reduction. If an abnormality occurs in the data collection task, but the data collection personnel fail to handle it in time, it will delay the progress of the data collection task. Based on this, after determining that the data collection task is abnormal, an alarm can be issued in the data collection monitoring system. The alarm method can be displayed on the display screen, sent a text message, sent an alarm email to the data collection personnel's mailbox, sound signal alarm, etc., to promptly remind the data collection personnel to handle the abnormality that occurs, and prevent the progress of the overall data collection from being delayed due to some data collection abnormalities.

[0043] In one embodiment, since it is impossible for staff to work every day, the data collection task also needs to consider the start and stop of terminals and equipment. A monitoring cycle can be added to the monitoring information, and the monitoring cycle is used to set the start and stop time of the data collection task. For example, the data collection task can be daily, weekly or monthly. The start time and end time of the monitoring cycle can also be added to the monitoring information. The start time of the monitoring cycle is the theoretical task start time of the data collection task, and the end time of the monitoring cycle is the theoretical deadline of the data collection task. An inspection cycle can also be added to the monitoring information to periodically determine whether the collection task is abnormal. For example, the inspection cycle can be set to half an hour, that is, every half an hour, according to the monitoring information configured by the data collection task, the log information of the data collection task is inspected. By setting the monitoring cycle of the data collection task, the time of the data collection task is limited. The start time and end time of the monitoring cycle are estimated based on the theoretical download or extraction speed and the amount of data collection tasks. If an abnormality occurs during the data collection task, the actual progress will be different from the theoretical progress, which can also be used to determine whether the data collection task is abnormal. By setting the inspection cycle, the real-time monitoring of log information is changed to periodic monitoring, which reduces the workload of monitoring data collection tasks.

[0044] It should be noted that, compared with the start time of the monitoring cycle, the actual start time of the data collection task refers to the time when the download or data extraction actually starts after the task is created, or the time when the download speed starts to appear. It may be different from the theoretical task start time, that is, the start time of the monitoring cycle.

[0045] In one embodiment, when calculating the download speed and extraction speed of the data collection task based on the log information, for the convenience of description, the time period from the start time of the monitoring cycle, i.e., the theoretical start time of the data collection task to the current time point, is called the collection time. The data collection amount C1 within the collection time is divided by the collection time S1, that is, The actual acquisition speed V1 within the acquisition time can be obtained.

[0046] In one embodiment, due to abnormal situations occurring during the data collection process, for example, when the network speed is reduced, if it is not discovered in time that the network speed is generally lower than before, the efficiency of data collection will be greatly reduced, thereby affecting the completion time of the data collection task. Based on this, the actual collection speed of the data collection task in the previous preset time period can be recorded as the historical collection speed of the data collection task in the current preset time period. That is, the collection speed calculated based on the log information obtained last time is used as the historical collection speed when the log information is obtained next time for the same data collection task. The actual collection speed of the data collection task in the current preset time period is compared with the historical collection speed to obtain a first difference. If the first difference exceeds the first preset threshold, it is considered that the collection task is abnormal. Or if the current collection speed is overall on a downward trend compared to the historical collection speed, it can also be considered that the collection task is abnormal.

[0047] In one embodiment, during the monitoring of the data collection task, the current actual collection speed can also be compared with the average speed of the entire monitoring cycle. If the fluctuation is large, it can also be considered that the collection task is abnormal. Based on this, in order to obtain the average speed of the data collection task in the entire monitoring cycle S2, it is first necessary to estimate the task volume of this data collection task to obtain the estimated data collection volume C2 of the data collection task, and divide the estimated data collection volume C2 by the duration of the monitoring cycle S2, that is, Thus, the theoretical acquisition speed V2 of the data acquisition task in the entire monitoring cycle S2 is obtained, which can also be said to be the average acquisition speed. The theoretical acquisition speed V2 is compared with the actual acquisition speed V1 in the current preset time period to obtain a second difference. If the second difference exceeds the second preset threshold, it is considered that the acquisition task is abnormal.

[0048] In one embodiment, since there may be no download speed for a period of time after the data collection task starts at the beginning of the task, it can be said that there is an abnormality in the startup of the data collection task. Therefore, the actual start time of the task can be compared with the start time of the monitoring period to obtain a third difference. If the third difference exceeds the third preset threshold, it is considered that the startup of the collection task is abnormal.

[0049] like Figure 2 As shown, the embodiment of the present application also provides a data acquisition and monitoring device based on the InfluxDB database, including:

[0050] at least one processor; and,

[0051] a memory communicatively connected to the at least one processor; wherein,

[0052] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0053] Determine a data collection task to be performed, wherein the data collection task includes downloading a page and extracting structured data;

[0054] At every preset time period, log information of the execution process of the data collection task within the preset time period is obtained; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period;

[0055] Obtaining the actual acquisition speed of the data acquisition task according to the log information;

[0056] According to the speed threshold value in the preset monitoring information and the actual acquisition speed, it is determined whether the data acquisition task is abnormal, and the monitoring information is configured according to the data acquisition task.

[0057] The embodiment of the present application also provides a non-volatile computer storage medium for deep hash retrieval, storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0058] Determine a data collection task to be performed, wherein the data collection task includes downloading a page and extracting structured data;

[0059] At every preset time period, log information of the execution process of the data collection task within the preset time period is obtained; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period;

[0060] Obtaining the actual acquisition speed of the data acquisition task according to the log information;

[0061] According to the speed threshold value in the preset monitoring information and the actual acquisition speed, it is determined whether the data acquisition task is abnormal, and the monitoring information is configured according to the data acquisition task.

[0062] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0063] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0064] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0065] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0068] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0069] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0070] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0071] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0072] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data collection and monitoring method based on InfluxDB database, characterized in that: include: Determine a data collection task to be performed, wherein the data collection task includes downloading a page and extracting structured data; At every preset time period, log information of the execution process of the data collection task within the preset time period is obtained; the log information includes the actual start time of the data collection task and the amount of data collected within the preset time period; Obtaining the actual acquisition speed of the data acquisition task according to the log information; According to the speed threshold value in the preset monitoring information and the actual acquisition speed, it is determined whether the data acquisition task is abnormal, and the monitoring information is configured according to the data acquisition task; After obtaining the log information during the execution of the data collection task within the preset time period, the method further includes: The log information is transmitted to the InfluxDB database, so as to obtain the actual acquisition speed by utilizing the characteristics of the InfluxDB database that supports time series functions and can calculate data in real time.

2. The method according to claim 1, characterized in that After determining whether the data collection task is abnormal, the method further includes: Determining that an abnormality occurs in the data collection task; Send an alarm email to the mailbox of the data collector to remind the data collector to handle the exception.

3. The method according to claim 1, characterized in that The method further comprises: Determine a monitoring cycle, each of the data collection tasks corresponds to one monitoring cycle; The monitoring period includes a start time and an end time, which are used to define the start time and the end time of the data collection task; The time between two inspections of the data collection task is determined as an inspection cycle, and based on the inspection cycle, it is periodically determined whether the data collection task is abnormal.

4. The method according to claim 3, characterized in that The calculating the actual acquisition speed of the data acquisition task according to the log information specifically includes: The time period from the start time of the monitoring cycle to the current time is determined as the collection time; According to the data collection amount C1 within the collection time and the collection time S1, The actual acquisition speed V1 is obtained.

5. The method according to claim 4, characterized in that The method further comprises: Recording the actual acquisition speed in the previous preset time period of the data acquisition task as the historical acquisition speed in the current preset time period of the data acquisition task; Comparing the actual acquisition speed with the historical acquisition speed to obtain a first difference; If the first difference exceeds a first preset threshold, it is considered that the acquisition task is abnormal.

6. The method according to claim 4, characterized in that The method further comprises: estimating the task size of the data collection task to obtain an estimated data collection amount; Through the estimated data collection amount C2 and the monitoring period S2, Get the theoretical acquisition speed V2; Comparing the theoretical acquisition speed V2 with the actual acquisition speed V1 to obtain a second difference; If the second difference exceeds a second preset threshold, it is considered that the acquisition task is abnormal.

7. The method according to claim 4, characterized in that The method further comprises: The actual start time is compared with the start time of the monitoring period to obtain a third difference. If the third difference exceeds a third preset threshold, it is considered that the acquisition task is started abnormally.

8. A data acquisition and monitoring device based on InfluxDB database, characterized in that: include: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to perform the method as claimed in any one of claims 1 to 7.

Images