Adaptively adjusted data acquisition method and device

Abstract

The present disclosure relates to an adaptive adjustment data collection method and device. A data collection method is disclosed, comprising: sampling data at a first time interval; sending the sampled data to a server at a second time interval, the second time interval being n times the first time interval, n being an integer greater than or equal to 3; wherein the determination of the sampled data to be sent is as follows: determining the difference between the maximum value and the minimum value in the sampled data during the current second time interval; determining whether to send all the sampled data during the current second time interval or to resample the sampled data based on the difference.

Description

Technical Field

[0001] This disclosure generally relates to operation and maintenance monitoring technologies, and more specifically to data acquisition technologies. Background Technology

[0002] Data acquisition technology is a crucial foundational technology for monitoring the operational status of cloud resource pools. By deploying monitoring agents on the monitored servers or terminals, status information of specified monitoring items, such as performance indicators like CPU, memory, hard disk, and network, is collected to monitor the operational status of the resource pool. Monitoring data can be used for system management, job scheduling, fault detection, and recovery, and the quality of the monitoring data directly impacts the effectiveness of the data acquisition methods.

[0003] Currently, the data acquisition technology of cloud resource pool monitoring systems mainly faces the following three problems:

[0004] 1) Sampling time

[0005] When adding new monitoring items to a monitoring system, the randomness of their startup time makes it impossible to synchronize them with the sampling times of existing monitoring items. This is especially true for sampling data from around the same time, where significant timestamp discrepancies exist between monitoring items, affecting the comparability of monitoring data.

[0006] 2) Sampling frequency

[0007] Data is collected at a fixed frequency. If the frequency is too low, the sampling interval becomes longer. When an anomaly occurs in the monitored item, the data change cannot be obtained in time, resulting in the loss of critical data. If the sampling frequency is too high, the sampling interval becomes shorter. When the system is running in a stable state, a large amount of similar data will be collected. This will not only excessively consume the terminal's computing and network resources, but also increase the computing and storage pressure on the monitoring server.

[0008] 3) Sampling characteristics

[0009] Currently, the mainstream sampling features for monitoring data are the values ​​at the current sampling time or the average values ​​within the sampling interval. When the sampling interval is long, the data at the current sampling time or the average value cannot reflect the actual trend of the monitored item's status change, especially when the monitored item is abnormal, as such data features lack sufficient information. When the sampling interval is short, since the normal operating time of the system is much longer than the abnormal time, the collection method based on such features will result in a lot of data redundancy.

[0010] The existing data collection technical solutions mainly aim to solve the above problems by adjusting the sampling frequency. Patent CN111147079A provides a data collection method with adaptive adjustable sampling frequency, which detects the data change rate in adjacent time windows and adjusts the sampling frequency in the next time window according to the data change rate. Patent CN110474812A provides a sampling rate adaptive adjustment method, which collects the current running state indicators of the application nodes in the distributed cluster, and judges whether the sampling rate needs to be adjusted according to the preset sampling rate adjustment rule. Both of the two solutions start collecting data at a certain fixed frequency, and change to another fixed sampling frequency when the change condition is met. However, changing the sampling frequency after triggering the change condition means that the triggering event has occurred, and changing the sampling frequency after that will still miss the key data before and at the time of the event.

[0011] With the wide application of artificial intelligence technology in the fields of anomaly detection, fault diagnosis and root cause analysis, the existing cloud resource pool data collection technical solutions have been difficult to meet the needs of data quality in these application scenarios due to the deficiencies in sampling time, sampling frequency and sampling features. SUMMARY

[0012] A brief summary of the present disclosure is given in the following to provide a basic understanding of some aspects of the present disclosure. However, it should be understood that this summary is not an exhaustive overview of the present disclosure. It is not intended to identify key or critical elements of the present disclosure or to delineate the scope of the present disclosure. Its purpose is merely to present some concepts of the present disclosure in a simplified form as a prelude to the more detailed description to be given later.

[0013] In order to solve the problems existing in the prior art, the present application provides a data collection method and device which can adaptively adjust the sampling time, sampling frequency and sampling features, not only solving the problems of redundant sampling data and the inability of sampling features to depict the real data change trend, but also solving the problem of too large sampling timestamp deviation between monitoring items in the monitoring system.

[0014] According to an aspect of the present application, a data collection method is provided, comprising: sampling data at a first time interval; sending the sampled data to a server at a second time interval, the second time interval being n times of the first time interval, n being an integer greater than or equal to 3; wherein the sent sampled data is determined as follows: determining the difference between the maximum and minimum values in the data sampled during the current second time interval; determining whether to send all the sampled data during the current second time interval or to resample the sampled data based on the difference.

[0015] According to another aspect of the present application, there is provided a data acquisition apparatus comprising: a memory storing computer executable instructions; and a processor configured to perform the method as described above when executing the computer executable instructions stored in the memory.

[0016] According to another aspect of the present application, there is provided a computer readable storage medium comprising computer executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] The present disclosure can be understood more readily by reference to the following detailed description, taken in connection with the accompanying drawings, and wherein:

[0019] Figure 1 A configuration diagram of a data acquisition system according to one embodiment of the present application is shown;

[0020] Figure 2 A flowchart of a data acquisition method according to one embodiment of the present application is shown;

[0021] Figure 3 A flowchart of a data acquisition method according to one embodiment of the present application is shown;

[0022] Figure 4 A flowchart of a sampling process according to one embodiment of the present application is shown;

[0023] Figure 5 A configuration diagram of a data acquisition apparatus according to one embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The following detailed description is made with reference to the accompanying drawings, and is provided to help a comprehensive understanding of various example embodiments of the present disclosure. The following description includes various details to help the understanding of the present disclosure, but these details are only considered as examples, and the present disclosure is defined by the appended claims and their equivalents. The words and phrases used in the following description are used only to enable a clear and consistent understanding of the present disclosure. In addition, descriptions of well-known structures, functions and configurations can be omitted for clarity and conciseness. Those of ordinary skill in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0025] Figure 1A system structure embodiment of the data collection system is given, which is composed of a server and a plurality of agents. The server can be a special monitoring server, and the agents can be monitoring agents. Each monitoring agent is deployed in a target system, i.e., a monitored system, which needs to collect data. Each monitoring agent can perform running state sampling on a plurality of collection objects. The collection objects can include CPU, memory, hard disk, network performance indicators, etc.

[0026] The monitoring server can be configured to configure the monitoring agents and corresponding sampling parameters, the configuration parameters of the monitoring agents including IP address, sampling object, port number, etc., and the sampling parameters including sampling time interval, etc. Preferably, the monitoring server can also be configured with a data display module for configuring the sampling parameters of the monitoring agents, displaying real-time sampling data, querying historical sampling data, etc. In addition, the monitoring server can also be configured with a data storage module for saving the configuration parameters and sampling data, and a data transmission module responsible for data communication between the monitoring server and the monitoring agents.

[0027] The monitoring agent is configured to perform the main process of data collection, including the following steps: sampling data at a first time interval; sending the sampled data to the server at a second time interval, the second time interval being n times of the first time interval and n being an integer greater than or equal to 3. Wherein, the sampled data to be sent is determined as follows: determining the difference (hereinafter referred to as range) between the maximum value and the minimum value in the data sampled during the current second time interval; determining whether to send all the sampling data during the current second time interval or to perform secondary sampling on the data sampled during the current second time interval based on the difference.

[0028] Wherein, the secondary sampling can be performed by those skilled in the art in a suitable manner according to actual conditions. Those skilled in the art can understand that the data amount of the secondary sampling data is less than that of the all sampling data.

[0029] Figure 2 A schematic flowchart of the above data collection method is shown.

[0030] First, in step 201, data is sampled and the sampled data is recorded. Then, in step 202, it is determined whether the second time interval has been reached. If the second time interval has not been reached, the flow proceeds to step 203, waiting for the first time interval. Then, it returns to step 201 and samples the data again. When it is determined at step 202 that the second time interval has been reached, the flow proceeds to step 204, calculating the difference between the maximum value and the minimum value in the recorded sampled data. Finally, in step 205, it is determined whether to send all the sampling data during the second time interval or to perform secondary sampling on the sampled data based on the difference, and the determined data to be sent is sent to the server for further processing.

[0031] Preferably, whether the second time interval has been reached can be determined by detecting the timing of the timer. Alternatively, whether the second time interval has been reached can also be determined by setting a counter to detect the number of iterations. For example, a counter C is set to 0, and after each sampling, C = C + 1. When C = n is detected, it is determined that the second time interval has been reached.

[0032] The present application reduces the amount of sampling data transmitted as a whole, reduces data redundancy and bandwidth consumption by sampling data at a higher frequency, then transmitting the sampled data to the server at a lower frequency, and determining whether to transmit all the data or only representative subsampled data based on the range of the sampled data indicating the system running state.

[0033] According to one embodiment, the subsampled data can be determined by the following steps. First, a first adjustment threshold is set. The difference between the maximum value and the minimum value in the sampled data is compared with the first adjustment threshold. If the difference is higher than or equal to the first adjustment threshold, all the sampled data is transmitted. If the difference is lower than the first adjustment threshold, the sampled data is subsampled.

[0034] Preferably, the first adjustment threshold is set to a critical value indicating that the monitored item is abnormal.

[0035] In this embodiment, when the extreme value of the sampled data is higher than or equal to the first adjustment threshold, it indicates that the monitored item is abnormal during the current second time interval. Therefore, all the sampled data is transmitted to the monitoring server to prevent the loss of critical data. When the change of the sampled data is lower than the first adjustment threshold, it indicates that the running of the monitored object is not abnormal, so there is no need to transmit all the sampled data, only the subsampled data representing the system running state is obtained by subsampling the sampled data, and then the subsampled data is transmitted to the server. Preferably, the subsampled data can include, for example, the extreme value (e.g. the maximum value and the minimum value) in the sampled data, the sampled data at a certain time (e.g. the data sampled at the beginning and end of the second interval), or the average value of the sampled data. It can be seen that the amount of subsampled data is much smaller than the amount of all sampled data (i.e. the amount of data sampled once).

[0036] In this way, by detecting the change of the difference between the maximum value and the minimum value, the characteristics of the lower frequency subsampling are dynamically adjusted, so that the sampled data can reflect the key trend changes of the running state of the sampling object while reducing data redundancy. This is because only the full sampling data for the period when an abnormality is likely to occur is transmitted, and only the subsampled data for the period of normal operation is transmitted.

[0037] To further reduce the redundant data while ensuring that no critical data is lost when the monitored item is abnormal, a second adjustment threshold can be set to distinguish between a stable state and a fluctuating state of the object. Preferably, the second adjustment threshold is set to a critical value representing a fluctuation in the monitored item. The non-abnormal state can be further divided into a stable state and a fluctuating state. The fluctuating state refers to a state in which the monitored object can have some instability, but has not yet become abnormal. In this state, the monitored object can be checked and maintained to avoid failure.

[0038] According to a preferred embodiment, if the difference between the maximum value and the minimum value is lower than the first adjustment threshold, it can be further determined whether the difference is higher than or equal to a second adjustment threshold, wherein the second adjustment threshold is lower than the first adjustment threshold; if higher than or equal to the second adjustment threshold, the maximum value, the minimum value, the earliest sampled data, and the latest sampled data of the data sampled during the current second time interval are sent; and if lower than the second adjustment threshold, the average value of the data sampled during the current second time interval is sent.

[0039] This is because, when higher than or equal to the second adjustment threshold, it indicates that the system is running without abnormalities, but there is fluctuation. Therefore, the data that best reflects the fluctuation in this period (the maximum value, the minimum value, the earliest sampled data, and the latest sampled data) is sent to the server to help analyze the fluctuation. When the difference is lower than the second adjustment threshold, it indicates that the object is running stably, and there is no too much difference between the collected data, at this time, only the average value representing the overall running condition of this period needs to be sent to the server for recording. In this way, the amount of data sent can be further reduced.

[0040] According to another preferred embodiment, the data collection method further comprises: if the range is higher than the first adjustment threshold, and it is determined that all the sampled data is to be sent, it is further determined whether the sampled data during the previous second time interval has been sent in its entirety. If not, all the sampled data during the previous second time interval is sent together with all the sampled data during the current second time interval.

[0041] In this embodiment, when an abnormality is detected, not only all the collected data during the period of failure is sent, but also the historical data of the period before the failure is sent. In this way, the data sequence before and at the time of failure can be obtained, and critical data can be avoided.

[0042] According to another preferred embodiment, the data collection method further comprises: synchronizing the sampling time of the monitored item with the sampling time of other monitored items. In this way, the sampling time stamps of the monitored items are basically free of significant deviation, which is beneficial to the comparison between the monitored data.

[0043] According to one embodiment, the time of sampling can be synchronized with the time of sampling of the other monitoring items as follows. First, it is determined whether the time stamp of the current time and the time of sampling of the other monitoring items is substantially the same. If it is substantially the same, the sampling is triggered. If it is not substantially the same, a start-up time interval is waited for, and then the step of determining whether the time stamp of the current time and the time of sampling of the other monitoring items is substantially the same is repeated until the time stamp of the current time and the time of sampling of the other monitoring items is substantially the same.

[0044] Preferably, the first time interval pi is a factor of 60 in seconds, and the start-up time interval T is in milliseconds and satisfies 10≤T<1000. This can be achieved by using a millisecond-level timer. When it is detected that the second bit of the timer is zero, it is determined that the current time and the time of sampling of the other monitoring items is substantially the same.

[0045] In this context, substantially the same means that it is the same at the second level, that is, there can be an error at the millisecond level.

[0046] In this way, even if the sampling procedure is started randomly, it can be ensured that the time stamps of sampling of different monitoring agents are synchronized at the second level. That is, the deviation of the time stamps of sampling between different monitoring agents is at the millisecond level, that is, less than 1 second, and the deviation is controllable.

[0047] In one preferred embodiment, before the sampling is triggered, it can be further determined whether sampling has been performed in the previous second of the current time, and the sampling is triggered only in the case that sampling has not been performed in the previous second of the current time. Specifically, after it is determined that the time stamp of the current time and the time of sampling of the other monitoring items is substantially the same, it is further determined whether sampling has been performed in the previous second of the current time. If sampling has been performed in the previous second, then a start-up time interval is waited for, and the determination of whether the time stamp is the same and whether sampling has been performed in the previous second is performed again.

[0048] This is because a millisecond-level timer is used, and the start-up time interval is also at the millisecond level, so it is possible that, after sampling is performed when the second bit is 0, even if the start-up time interval is added, the second bit of the timer is still 0. If sampling is performed again at this time, it will result in that the sampling time interval is too short, that is, less than 1 second, which is also referred to as repeated sampling. Repeated sampling will result in the generation of redundant data, which is not conducive to saving computing and network resources.

[0049] In a preferred embodiment, a random delay period is waited before sending data, and the random delay period is no more than one second. This is because the sampling times of different monitoring items are synchronized, and the server has the possibility to receive a large amount of data at the same time. By introducing the random delay, the sending times of the collected data of different monitoring items can be staggered with each other, preventing the server from receiving a large amount of data at the same time, and ensuring the smooth running of the server.

[0050] An example of the present application is described below with reference to Figure 3 An example of the present application is described below with reference to

[0051] After the monitoring agent is started, the data transmission module is used to communicate with the monitoring server, to acquire the system time of the monitoring server and to synchronize. When the configuration parameters from the monitoring server are received, the configuration parameters are stored and sampling initialization is completed according to the configuration parameters, and a sampling counter is set.

[0052] Step 301: Calibrating system time

[0053] The time of the system in which the collection object is located is synchronized with the selected network standard time, to calibrate the system time.

[0054] Step 302: Setting sampling parameters

[0055] Before starting to collect data, a first sampling time interval pi is set, corresponding to a first sampling frequency fi = 1 / pi; a second sampling time interval p2 is set, corresponding to a second sampling frequency f2 = 1 / p2; a first adjustment threshold h1 for judging whether an abnormality occurs and a second adjustment threshold h2 for judging whether the running state is smooth are set, h1 > h2; a parameter n is set, n is a positive integer, and n > 3. At the same time, pi and p2 need to satisfy the following conditions:

[0056] 60% pi = 0

[0057] p2 = pi * n

[0058] That is, the first sampling time interval pi is between 1 second and 60 seconds, and is a factor of 60, and the second sampling time interval is n times the first sampling time interval.

[0059] The first sampling frequency fi is the original sampling frequency, and the second sampling frequency f2 is the frequency of transmitting the data of the original sampling. It can be seen that the second sampling frequency is lower than the first sampling frequency.

[0060] Step 303: Sampling initialization

[0061] The counter, the history data record and the related flags are all initialized to 0. For example, the sampling counter C is initialized to 0, and the history data sequence record data0 is initialized to empty. In addition, the first flag Flagl is initialized to 0, the second flag Flag2 is initialized to 0, and the third flag Flag3 is initialized to 0.

[0062] The first flag Flagl, the second flag Flag2 and the third flag Flag3 are binary single digits, whose values are 0 or 1.

[0063] The first flag Flagl indicates whether the time stamp of the current time and the sampling time of the other monitoring item is the same. When Flagl = 1, it means that the time stamp of the current time and the sampling time of the other monitoring item is approximately the same, that is, at least the same at the second level. When Flagl = 0, it means that it is not clear whether the time stamp of the current time and the sampling time of the other monitoring item is approximately the same, and further judgment is needed.

[0064] The second flag Flag2 indicates whether sampling is performed within the last second of the current time. When Flag2 = 1, it means that sampling has been performed within the last second of the current time, that is, the current time is too short (less than 1 second) from the last sampling. Even if the second bit of the timer is 0 at this time, the first sampling time interval is not met, and sampling should not be performed. When Flag2 = 0, it means that sampling is not performed within the last second of the current time, and if the second bit of the timer is 0 at this time, sampling can be triggered.

[0065] The third flag Flag3 indicates whether the data sampled during the last second sampling time interval is all sent. When Flag3 = 1, it means that the data sampled during the last second sampling time interval is not all sent. That is, the data sampled during the last second sampling time interval is sent after being sampled twice. When Flag3 = 0, it means that the data sampled during the last second sampling time interval is all sent.

[0066] Step 304: Sampling

[0067] The timer is started, and the starting time interval of the timer is T milliseconds, and 10 ≤ T < 1000. That is, the operation is performed every T milliseconds to obtain the sampling data. The specific operation steps are described below with reference to Figure 4 The specific operation steps are as follows:

[0068] 1. Synchronize the sampling time with the sampling time of the other monitoring item (steps 401-404).

[0069] In step 401, the value of the second bit of the current time of the timer is obtained every T milliseconds. For example, if the timer is H hours, M minutes, S seconds and T milliseconds, the value of the second bit is S.

[0070] In step 403, it is detected whether S is 0. If S=0, in step 404, the first flag Flagg is set to 1, and the process proceeds to step 405. If S≠0, the process repeats step 401.

[0071] The synchronization operation is only performed when a new monitoring item is started. Once the initial sampling time stamp is synchronized with other monitoring items, the subsequent sampling times are synchronized, and there is no need to repeat the detection each time. Therefore, in step 402, if Flagg=1, it indicates that the synchronization operation has been completed, and the process can continue to step 405. By setting Flagg, in the subsequent operation, it can be determined whether the synchronization has been completed by detecting the value of Flagg.

[0072] 2. Detecting the first sampling time interval (steps 405-407).

[0073] Whether the first sampling time interval is met can be determined by two conditions. 1) Since pi is a time period in seconds and is a factor of 60, S should be a multiple of pi. 2) Considering the repeated sampling problem, it should be ensured that there is no sampling in the previous second. If both conditions are met, it indicates that the first sampling time interval is met.

[0074] In step 405, S%pi is calculated. Then in step 406, if S%pi≠0, i.e., S is not 0 and is not a multiple of pi, the first sampling time interval is not met, the second flag Flag2 is set to 0, and the process returns to step 401. If S%pi=0 and in step 407, it is determined that Flag2=1, it indicates that sampling has just been performed in the previous second, the first sampling time interval is not met, and the process returns to step 401. If S%pi=0 and in step 407, it is determined that Flag2=0, it indicates that the first sampling time interval is met, and the process proceeds to step 408.

[0075] 3. Sampling and recording the sampling data (steps 408-409)

[0076] In step 408, the second flag Flag2 is set to 1, and the sampling counter C is increased by 1, i.e., C=C+1. In step 409, the current state of the monitored object is sampled once and recorded as (ti, vi), where i=C, ti is the sampling time, and vi is the sampling value. i i i i

[0077] 4. Determining the extreme value of the sampling data (step 410)

[0078] In step 410, the current sampling data can be compared with the data of the previous sampling, and the larger value of the data is recorded as the maximum value v max ​​​​the data with the smaller value is recorded as v min .

[0079] If v i > v i-1 , then v max = v i , t va = t i , v min = v i-1 , t vi = t i-1 ; otherwise, v max = v i-1 , t va = t i-1 , v min = v i , t vi = t i . Wherein V max is the maximum value in the sampled data, t va is the sampling time corresponding to the maximum value, V min is the minimum value in the sampled data, t vi is the sampling time corresponding to the minimum value.

[0080] The way to determine the extreme value of the sampled data is not limited to this, and the extreme value in the sampled data can also be determined after all the sampled data during the second sampling time interval is recorded.

[0081] 5. Detecting the second sampling time interval (step 411)

[0082] Since the second sampling time interval is n times the first sampling time interval, it is only necessary to detect whether the value of the counter C is n.

[0083] In step 411, if C is not equal to n, steps 4.1 to 4.4 are repeated; otherwise, the second sampling time interval is met, the range R = v max - v min is calculated, a set of sampled data [(t1, v1), (t2, v2), …, (t n , v n )] is obtained, and the sampling counter C is reset to 0.

[0084] 6. Determine the data sequence to be sent (step 413)

[0085] In step 413, the range R is compared with the adjustment thresholds h1 and h2, and the sampled data sequence data1 to be sent is determined based on the comparison result, and then step 401 is returned and step 305 to be described later is executed at the same time.

[0086] The method of obtaining the data sequence data1 to be sent is as follows:

[0087] When R≥h1, it is determined that an abnormality can occur, and all the sampling data need to be sent. Therefore, it is determined that the data sequence data1 includes all the sampling data [(t1, v1), (t2, v2), …, (t n ,v n )]. n ,v n )]. n ,v n )].

[0088] Then, the value of data0 is updated to [(t1, v1), (t2, v2), …, (t n ,v n )], and the third flag Flag3 is set to 0.

[0089] When R n ,v n )] is twice-sampled. Preferably, R can be compared with a second adjustment threshold h2, and the non-abnormal state can be further subdivided, and different sampling methods can be taken.

[0090] When h2≤R va ,v max ) and (t vi ,v min ) in the sampling data and the data at the initial and final sampling time (i.e., (t1, v1) and (t n ,v n )) are sent to the server. Since the data sent is a sequence, the data should be arranged according to the time sequence. If t va >t vi , the data sequence data1 is [(t1, v1), (t vi ,v min ), (t va ,v max ), (tn v n )],else data1 is [(t1,v1),(t va ,v max ),(t vi ,v min ),(t n ,v n )]. Preferably, when there is a repetition between timestamps t1, t vi , t va and t n , only one of the repeated sampling data is kept. Then, the third flag Flag3 is set to 1, and data0 is updated to [(t1,v1),(t2,v2),…,(t n ,v n )].

[0091] When R < h2, the monitored object is in a steady state. At this time, only the characteristic value of the sampling data representing the running state of the object in this period of time needs to be sent. The mean value of the sampling data data1 is [(t n ,μ)], and the third flag Flag3 is set to 1, and data0 is updated to [(t1,v1),(t2,v2),…,(t n ,v n )].

[0092] The above secondary sampling method is only an example. Those skilled in the art can adopt different secondary sampling data sequences according to needs. For example, the median of the sampling data can be sent, or the average value and the value at a specific time can be sent together, etc.

[0093] Step 305: Delayed sending

[0094] Since the sampling times of different monitoring items are synchronized, the sending times of the data are also approximately the same, so the server can receive a large amount of data at the same time. In order to ensure the smooth running of the server, a random delay is introduced, so that the sending times of the sampling data of different monitoring items are staggered with each other.

[0095] According to an embodiment, a random integer d can be calculated, and 0 < d < 1000, and the sampling data sequence data1 is sent after a delay of d milliseconds. In this way, the sending times of the sampling data of different monitoring items can be staggered within 1 second.

[0096] The above method can be implemented by using a programming language known by those skilled in the art. For example, in Python, a timer with a time interval of 100 ms can be implemented by using the sleep function of the time module, the current time can be obtained by using the datetime module, and the value of the second bit can be obtained by now.second. The threading module is used to process threads, and the first thread thread1 and the second thread thread2 are initialized.

[0097] The first sampling time interval and the second sampling time interval, the first adjustment threshold and the second adjustment threshold, and the like can be selected by those skilled in the art within a specified range according to the nature of the monitored object. For example, for monitoring the available memory rate of a Linux system, the first sampling time interval can be 5 s, the second sampling time interval can be 50 s, the first adjustment threshold can be 0.25, and the second adjustment threshold can be 0.5. The timer uses a millisecond-level timer, and the start time interval can be determined according to the set sampling time interval. For example, for a first sampling time interval of 5 s and a second sampling time interval of 50 s, the start time interval of the timer can be 100 ms.

[0098] The first thread thread1 can be used to perform the sampling process, and the second thread thread2 can be used to perform data transmission at the same time after completing the sampling during the second sampling time interval. The process is as follows: first, a random number d between 1 and 999 is generated by using the randint function of the random module, and then the sampling data sequence data1 is sent after a delay of d milliseconds by using the sleep function of the time module.

[0099] After the monitoring server receives the sampling data from the monitoring agent, the sampling data is saved to the data storage module and displayed by the data display module.

[0100] Although the process of implementing the data sampling method of the present application is described above with reference to Python, the implementation of the data sampling method of the present application is not limited thereto. Other programming languages can also be used for implementation.

[0101] The above-described data collection method can be a method that can be implemented by a computer or a processor. Specifically, it can be executed by software, hardware, or a combination of software and hardware. For example, it can be implemented by a plurality of modules that execute the steps of the above method. The above-described method can also be implemented as program code stored in a storage medium to be executed by a processor. The processor can be a special-purpose processor or a general-purpose processor.

[0102] Furthermore, this disclosure can be implemented as any combination of apparatus, system, integrated circuit, and computer program on a non-transitory computer-readable medium. A data acquisition apparatus according to an embodiment of the present invention is described below.

[0103] Figure 5 An exemplary configuration of the data acquisition device 500, which can be implemented according to embodiments of the present disclosure, is shown. The data acquisition device 500 can be deployed in a distributed manner at a monitoring server and at the monitored object.

[0104] The data acquisition device 500 is an example of a hardware device capable of applying the above aspects of this disclosure. The data acquisition device 500 can be any machine configured to perform processing and / or calculations. The data acquisition device 500 can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal digital assistant (PDA), smartphone, in-vehicle computer, or a combination thereof.

[0105] like Figure 5 As shown, the data acquisition device 500 may include one or more components that can be connected to or communicate with the bus 502 via one or more interfaces. The bus 502 may include, but is not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus. The computing device 500 may include, for example, one or more processors 504, one or more input devices 506, and one or more output devices 508. The one or more processors 504 may be any type of processor and may include, but is not limited to, one or more general-purpose processors or dedicated processors (such as dedicated processing chips). The processor 504 may, for example, be configured to implement the method for identifying unwanted information according to the present invention. The input device 506 may be any type of input device capable of inputting information to the computing device and may include, but is not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote controller. The output device 508 may be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, and / or printer.

[0106] The computing device 500 can also include or be connected to a non-transitory storage device 514, which can be any non-transitory and can enable data storage and can include, but is not limited to, a disk drive, optical storage, solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape, or any other magnetic medium, a compact disk or any other optical medium, a cache memory, and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 514 stores computer-executable instructions. The instructions, when executed by the processor 504, implement the method of identifying bad information of the present disclosure.

[0107] The computing device 500 can also include a random access memory (RAM) 510 and a read-only memory (ROM) 512. The ROM 512 can store programs, utilities, or processes to be executed in a non-volatile manner. The RAM 510 can provide volatile data storage and store instructions related to the operation of the data acquisition device 500. The data acquisition device 50 can also include a network / bus interface 516 coupled to a data link 518. The network / bus interface 516 can be any kind of device or system capable of enabling the computing device 500 to communicate with external devices and / or networks, and can include, without limitation, a modem, a network card, an infrared communication device, a wireless communication device and / or chipset (such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc.).

[0108] The subject matter of the present disclosure is presented with reference to examples of apparatus, systems, methods, and program for performing the features described in this disclosure. However, other features or variations can be contemplated in addition to or in place of the features described above. It is contemplated that the implementation of the components and functions can be accomplished with any technology that can potentially render the features as described above.

[0109] In addition, the above description provides examples, and does not limit the scope, applicability, or configuration of the claims. Changes can be made in the function and arrangement of elements discussed without departing from the spirit and scope of the disclosure. Various embodiments can omit, substitute, or add various procedures or components as appropriate. For instance, features described with respect to certain embodiments can be incorporated into other embodiments as appropriate.

[0110] In addition, in the description of the disclosure, the terms "first", "second", "third", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance and sequence.

[0111] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order nor that all illustrated operations be performed, to accomplish a desired result. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A data collection method, comprising: sampling data at a first time interval; sending the sampled data to a server at a second time interval, the second time interval being n times the first time interval, n being an integer greater than or equal to 3; wherein the sent sampled data is determined as follows: determining a difference between a maximum value and a minimum value in the sampled data during a current second time interval; determining whether to send all of the sampled data during the current second time interval or to subsample the sampled data based on the difference, comprising: determining whether the difference is higher than or equal to a first adjustment threshold; if the difference is higher than or equal to the first adjustment threshold, sending all of the sampled data; and if the difference is lower than the first adjustment threshold, subsampling the sampled data.

2. The data collection method of claim 1, wherein the twice-sampled data comprises: extreme values in the sampled data, sampled data at specific time instants, or average values of the sampled data.

3. The data collection method of claim 1, wherein, if the difference is lower than the first adjustment threshold, further determining whether the difference is higher than or equal to a second adjustment threshold, wherein the second adjustment threshold is lower than the first adjustment threshold; if higher than or equal to the second adjustment threshold, sending a maximum value, a minimum value, an earliest sampled data, and a latest sampled data of the sampled data during the current second time interval; and if lower than the second adjustment threshold, sending an average value of the sampled data during the current second time interval. if it is determined to send all of the sampled data, the data collection method further comprising: determining whether all of the sampled data during a previous second time interval has been sent; and 4. The data collection method of claim 1, wherein, if not, sending all of the sampled data during the previous second time interval together with all of the sampled data during the current second time interval. synchronizing the sampling time with a sampling time of another monitoring item. synchronizing the sampling time with the sampling time of the other monitoring item comprises:

5. The data collection method of claim 1, further comprising: determining whether a timestamp of a current time and a sampling time of the other monitoring item is substantially the same; 6. The data collection method of claim 5, wherein, if substantially the same, triggering the sampling; and if not substantially the same, waiting for a start-up time interval T, and then repeating the step of determining whether the timestamp of the current time and the sampling time of the other monitoring item is substantially the same until the timestamp of the current time and the sampling time of the other monitoring item is substantially the same. the first time interval is in seconds and is a factor of 60, and the start-up time interval T is in milliseconds and satisfies 10 < T < 1000. the timestamp of the current time and the sampling time of the other monitoring item is determined to be substantially the same when a second bit of a timer is detected to be zero.

7. The data collection method of claim 6, wherein, 9. The data collection method of claim 7, further comprising determining whether sampling has occurred within a previous second of the current time, and triggering the sampling only in the case that sampling has not occurred within the previous second of the current time.

8. The data collection method of claim 7, wherein, waiting for a random delay period before sending the data, the random delay period not exceeding one second.

11. A data collection apparatus, comprising:

10. The data collection method of claim 1, further comprising: a memory storing computer-executable instructions; and a processor configured to execute the method of any one of claims 1-10 when executing the computer-executable instructions stored in the memory. ​ ​ ​ 12. A computer-readable storage medium comprising computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1-10.

Images