A data acquisition system, method and computer program product

Abstract

The present disclosure belongs to the technical field of data collection, and particularly relates to a data collection system, method and computer program product, the system comprising: a log component and a strategy component deployed at a collected end; wherein the strategy component is configured to: acquire log data of the collected end through the log component; analyze a collection period rule of the collected end and a data update rule of the collected end according to the log data; dynamically adjust a data pulling period of the collected end according to an analysis result; and adjust a system parameter corresponding to a pulling execution effect based on the adjusted data pulling period of the collected end. The data collection system provided by the present disclosure acquires log data through the log component, analyzes a collection period rule of the collected end and a data update rule of the collected end, and dynamically adjusts a data pulling period of the collected end, so that, compared with a traditional fixed period pulling mode, a large amount of invalid data pulling operations are reduced, and a data collection effect is better.

Description

Technical Field

[0001] This disclosure belongs to the field of data acquisition technology, and specifically relates to a data acquisition system, method and computer program product. Background Technology

[0002] Data acquisition is a preliminary step in the data processing process. Pull-mode acquisition is a method in which the acquisition end actively collects business data from the database of the acquisition end. It can provide reliable support for subsequent data processing, analysis and application. The acquisition effect directly affects the accuracy and effectiveness of subsequent data application. Therefore, there are high requirements for the real-time performance of the acquired data.

[0003] Figure 1 As a common data collection system, the databases on the collected end generally use a fixed-period polling method to pull business data. Although this method can meet basic data collection needs, in actual business scenarios, the update patterns of business data are diverse. There are regular updates with fixed periods, as well as non-periodic updates with unstable update rates. The update frequency and operating status of different business systems also vary. The fixed-period retrieval strategy has obvious limitations and is difficult to adapt to various actual application scenarios. There is an urgent need to optimize the retrieval strategy to improve the data collection effect while taking into account both data real-time performance and resource utilization.

[0004] Furthermore, such as Figure 2 As shown, the existing fixed-period polling retrieval method is prone to misalignment between the retrieval trigger time and the server collection time. If the collection action occurs before the database data update is completed, historical data that has not been updated will be collected, causing data lag issues and making it unsuitable for business scenarios with high real-time requirements. Secondly, the retrieval efficiency is low and resources are wasted. The fixed-period retrieval mode cannot adapt to the dynamic changes in the frequency of business data updates. When business data updates frequently, a fixed long period will exacerbate the data lag problem. When business data updates are sparse, a fixed short period will generate a large number of invalid retrieval operations, increasing the database query pressure and causing resource consumption.

[0005] In addition, since the commonly used optimization methods in the industry are all to improve the acquisition efficiency by modifying the acquisition end, when the acquisition end equipment of different manufacturers is replaced, the evaluation and optimization work needs to be completed again. Therefore, this optimization method still has the problem of insufficient universality. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure provides a data acquisition system that offers the advantages of non-intrusive acquisition and efficient data retrieval. The system includes: Log components and policy components deployed on the client-side; among them, The strategy component is configured as follows: Log data from the collected device is obtained through the log component. Based on the log data analysis, the data collection cycle pattern of the data collection terminal and the data update pattern of the collected terminal are observed. The data retrieval cycle of the collected terminal is dynamically adjusted based on the analysis results. Optimize system parameters based on the adjusted data retrieval cycle of the collected data source to correspond to the retrieval execution effect.

[0007] Furthermore, the logging component includes: The log collection module and log retrieval module are deployed on the client side. The strategy component is configured to obtain log data from the collected end through the log collection module and the log retrieval module; wherein, the log data includes: server collection behavior data, database retrieval behavior data, and database retrieval result data.

[0008] Furthermore, the strategy component is configured as follows: The server collection time interval data is determined based on the server's collection behavior data; Based on the statistical characteristic values ​​of the server's data collection time interval, it is determined whether the server's data collection cycle is stable. If so, the server's data collection time interval is fitted to a baseline data collection cycle, which is then used as one of the patterns of the data collection cycle at the acquisition end.

[0009] Furthermore, the strategy component is configured as follows: Determine the database fetch time interval data based on the database fetch behavior data; Based on the statistical characteristic values ​​of the database retrieval time interval data, it is determined whether the database updates data periodically. If so, the database retrieval time interval is predicted based on the database retrieval time interval data, and the predicted value is used as one of the data update rules of the collected end.

[0010] Furthermore, the strategy component is also configured as follows: When determining non-periodic updates to database data, a prediction algorithm, including the weighted moving average method, is used to predict the database retrieval time interval based on the database retrieval time interval data, and the predicted value is used as one of the data update rules of the collected end.

[0011] Furthermore, the strategy component is also configured as follows: Determine whether the data collection period of the acquisition terminal is stable. If not, the data retrieval period of the acquired terminal is reverted to the preset basic minimum guaranteed period. If so, the data retrieval period of the acquired terminal is adjusted according to the relationship between the data update interval of the acquired terminal and the preset update interval threshold.

[0012] Furthermore, the strategy component is also configured as follows: When the data collection cycle at the acquisition end is determined to be stable, it is checked whether the data update interval at the acquired end is less than a preset update interval threshold. If so, the current data retrieval cycle at the acquired end is shortened; otherwise, the current data retrieval cycle at the acquired end is lengthened. When extending the current data retrieval period of the collected terminal, the constraints include: the extended data retrieval period of the collected terminal is not greater than the basic minimum guaranteed period.

[0013] Furthermore, the strategy component is configured as follows: Determine whether the effective retrieval rate corresponding to the adjusted data retrieval period of the collected end meets the requirement of being greater than the preset effective retrieval target and not continuously decreasing. If yes, maintain the current data retrieval period of the collected end; otherwise, optimize the current data retrieval period of the collected end until the effective retrieval rate meets the condition.

[0014] This disclosure also proposes a data acquisition method, including: Using the log component and policy component deployed on the client side, execute the following methods: Log data from the collected device is obtained through the log component. Based on the log data analysis, the data collection cycle pattern of the data collection terminal and the data update pattern of the collected terminal are observed. The data retrieval cycle of the collected terminal is dynamically adjusted based on the analysis results. Optimize system parameters based on the adjusted data retrieval cycle of the collected data source to correspond to the retrieval execution effect.

[0015] This disclosure also proposes a computer program product stored in a computer-readable storage medium, which, when executed by a processor, is used to implement at least the above-described data acquisition method.

[0016] Compared with the prior art, this disclosure has the following advantages: (1) The data acquisition system proposed in this disclosure deploys the log component and the strategy component on the acquisition end in a unified manner, and obtains log data through the log component, analyzes the acquisition cycle pattern of the acquisition end and the data update pattern of the acquisition end, and dynamically adjusts the data retrieval cycle of the acquisition end. Compared with the traditional fixed cycle retrieval method, it reduces a large number of invalid data retrieval operations and the data acquisition effect is better. (2) The log component and strategy component proposed in this disclosure both run on the data collection end, without the need for intrusive modification of the data collection end. They can be adapted to data collection end devices of various manufacturers, have strong universality, and do not need to be re-adapted and optimized after replacing the data collection end device, which greatly reduces the business adaptation cost. (3) The data acquisition system proposed in this disclosure has a closed-loop optimization mechanism, which iteratively optimizes the data retrieval strategy based on the effective retrieval rate to ensure the stability and accuracy of long-term data acquisition.

[0017] Other features and advantages of this disclosure will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 An existing pull-mode acquisition system is shown; Figure 2 This illustrates the principle behind the misalignment between the pull trigger time and the server acquisition time; Figure 3 A data acquisition system according to an embodiment is shown; Figure 4 The connection principle of the log component according to the embodiment is shown; Figure 5 The connection principle of the strategy components according to the embodiment is shown; Figure 6 The process of analyzing and collecting records according to the strategy component is illustrated in the embodiment; Figure 7 The process of analyzing and retrieving records according to a strategy component is illustrated in the embodiment. Figure 8 The policy component decision-making process according to an embodiment is illustrated; Figure 9 The closed-loop feedback optimization process of the strategy component according to an embodiment is illustrated. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] This disclosure discloses a data acquisition system, which includes: Log components and policy components deployed on the client-side; among them, The strategy component is configured as follows: Log data from the collected device is obtained through the log component. Based on the log data analysis, the data collection cycle pattern of the data collection terminal and the data update pattern of the collected terminal are observed. The data retrieval cycle of the collected terminal is dynamically adjusted based on the analysis results. Optimize system parameters based on the adjusted data retrieval cycle of the collected data source to correspond to the retrieval execution effect.

[0022] According to some embodiments of this disclosure, such as Figure 3 As shown, the data acquisition system proposed in this disclosure deploys four types of components, each with a clear division of labor and collaborative operation. The four types of components are: log component, strategy component, database component, and business component. Among them, the log component and strategy component are necessary components. The two work together to complete data retrieval with a closed-loop optimization mechanism. Specifically, the strategy component obtains log data from the acquired end through the log component, analyzes the acquisition cycle pattern and data update pattern of the acquired end, and dynamically adjusts the data retrieval cycle of the acquired end based on the analysis results. After the data retrieval cycle of the acquired end is adjusted, the parameters of the data acquisition system are optimized in a closed loop based on the data retrieval effect.

[0023] Furthermore, the logging component includes: The log collection module and log retrieval module are deployed on the client side. The strategy component is configured to obtain log data from the collected end through the log collection module and the log retrieval module; wherein, the log data includes: server collection behavior data, database retrieval behavior data, and database retrieval result data.

[0024] The working principle of the logging component will be further explained below with reference to Example 1: like Figure 4 As shown, the log component includes a server log collection module, a database log retrieval module, and a log preprocessing module deployed on the collected end. The log collection module and the log retrieval module are used to implement separate recording of dual log channels, which facilitates subsequent separate calls and analysis.

[0025] 1. Data Collection Log Module: This module is specifically responsible for recording the exact time and results of each data collection by the server, accurately capturing the server's data collection behavior, and providing basic data for subsequent analysis of the server's data collection cycle patterns. 2. Pull Log Module: This module is specifically responsible for recording the exact time and results of each database pull (with a focus on whether valid updated data was pulled), providing a complete record of the pull behavior and its effects. 3. Log Preprocessing Module: This module preprocesses the log data collected by the log acquisition and log retrieval modules, sorts it chronologically, and removes abnormal data such as erroneous records and data with outdated times. Samples with slight fluctuations in the server's acquisition cycle due to network latency are treated as noise and removed to ensure that the log data is clean and reliable, providing support for subsequent pattern extraction and analysis.

[0026] Furthermore, the strategy component is configured as follows: The server collection time interval data is determined based on the server's collection behavior data; Based on the statistical characteristic values ​​of the server's data collection time interval, it is determined whether the server's data collection cycle is stable. If so, the server's data collection time interval is fitted to a baseline data collection cycle, which is then used as one of the patterns of the data collection cycle at the acquisition end.

[0027] The working principle of the strategy component will be explained below with reference to Example 2: like Figure 5 As shown, the strategy components include a log collection module and an analysis module. The log collection module is used to obtain historical records after preprocessing by the log component, and the analysis module is used to perform statistical analysis on the log data obtained by the log collection module, extract the server collection cycle pattern and the database data update pattern, and construct a correlation profile between data updates and collection. The specific scheme is as follows: like Figure 6 As shown, the analysis module determines the server collection records based on the preprocessed log data, calculates the time interval between two adjacent server collections, and obtains the statistical characteristic values ​​(mean, variance, maximum, minimum, etc.) of the time interval data. By judging whether the fluctuation of the statistical characteristic values ​​of the collection time interval is less than or equal to the preset fluctuation threshold, it can be determined whether the server collection cycle is stable. If the collection period is stable: perform period fitting based on the server collection time interval data to determine the server collection baseline period, providing a basis for subsequent pull period adjustments; If the collection period is unstable: it is determined to be non-periodic collection, which may be due to human operation. It is highly random and not easy to predict. The execution module will be notified to take appropriate action to ensure the stability of the system's collection work and avoid abnormal data retrieval caused by server period fluctuations.

[0028] Furthermore, the strategy component is configured as follows: Determine the database fetch time interval data based on the database fetch behavior data; Based on the statistical characteristic values ​​of the database retrieval time interval data, it is determined whether the database updates data periodically. If so, the database retrieval time interval is predicted based on the database retrieval time interval data, and the predicted value is used as one of the data update rules of the collected end.

[0029] The working principle of the strategy component will be further explained below based on Example 2: like Figure 7 As shown, the analysis module is used to process the data update times collected by the database retrieval module, calculate the time interval between two adjacent database retrievals, obtain the statistical characteristic values ​​of the database retrieval time interval (average value and fluctuation range, etc.), identify the database data update pattern (periodic update / non-periodic update), and use the corresponding algorithm to predict the subsequent data update time. If the data update pattern is periodic update, the subsequent database retrieval time interval or database retrieval time is predicted based on the database retrieval time interval, ensuring that the prediction accuracy matches the data update rhythm.

[0030] Furthermore, the strategy component is also configured as follows: When determining non-periodic updates to database data, a prediction algorithm, including the weighted moving average method, is used to predict the database retrieval time interval based on the database retrieval time interval data, and the predicted value is used as one of the data update rules of the collected end.

[0031] The working principle of the strategy component will be further explained below based on Example 2: The strategy component is also used to predict subsequent database fetch intervals or database fetch times based on the calculated database fetch interval after determining that the database data update mode is non-periodic. For example, a weighted moving average method is used to give higher weight to recent data update intervals to predict subsequent database fetch intervals or database fetch times. This method is suitable for non-periodic data update scenarios and balances prediction accuracy with ease of engineering implementation.

[0032] Furthermore, the strategy component is also configured as follows: Determine whether the data collection period of the acquisition terminal is stable. If not, the data retrieval period of the acquired terminal is reverted to the preset basic minimum guaranteed period. If so, the data retrieval period of the acquired terminal is adjusted according to the relationship between the data update interval of the acquired terminal and the preset update interval threshold.

[0033] The working principle of the strategy component will be explained below with reference to Example 3: The strategy module includes an execution module, which is used to adjust the data retrieval cycle of the collected data based on preset rules and distribute it to the log component for execution. The specific scheme is as follows: (1) Setting constraint parameters: Set an update interval threshold: This is the dividing line between fast and slow data updates, serving as the core basis for determining whether the retrieval cycle needs to be shortened or lengthened. Set a basic minimum protection period: pull the longest red line of the period to avoid long periods of no data collection due to prediction deviation, and ensure that the data can be refreshed.

[0034] (2) Pulling cycle adjustment rules: like Figure 8 As shown, the execution module adjusts the data retrieval cycle of the collected data according to the following process, so that the retrieval action adapts to both the data collection patterns of the collected data terminal and the data update patterns of the database: If the data collection period of the acquisition terminal is unstable, the data retrieval period of the acquisition terminal will be rolled back to the preset basic minimum period without further optimization or adjustment, prioritizing the stability and reliability of the system's data collection work. If the data collection period of the acquisition terminal is stable, the data retrieval period of the acquisition terminal will be adjusted according to the correspondence between the data update interval of the acquisition terminal and the preset update interval threshold.

[0035] Furthermore, the strategy component is also configured as follows: When the data collection cycle at the acquisition end is determined to be stable, it is checked whether the data update interval at the acquired end is less than a preset update interval threshold. If so, the current data retrieval cycle at the acquired end is shortened; otherwise, the current data retrieval cycle at the acquired end is lengthened. When extending the current data retrieval period of the collected terminal, the constraints include: the extended data retrieval period of the collected terminal is not greater than the basic minimum guaranteed period.

[0036] The working principle of the strategy component will be further explained below based on Example 3: After determining that the server's data collection cycle is stable, the execution module further determines the relationship between the data update interval of the collected end and the set threshold: if the data update interval of the collected end is less than the set update interval threshold (i.e., the data updates quickly), the data retrieval cycle of the collected end is shortened to ensure that the retrieval action matches the data update rhythm and avoid missing the latest data; if the data update interval is greater than or equal to the set threshold (i.e., the data updates slowly), the current data retrieval cycle of the collected end is lengthened with the basic minimum guaranteed cycle as a constraint, that is, the adjusted data retrieval cycle of the collected end should not be greater than the basic minimum guaranteed cycle.

[0037] Furthermore, the strategy component is configured as follows: Determine whether the effective retrieval rate corresponding to the adjusted data retrieval period of the collected end meets the requirement of being greater than the preset effective retrieval target and not continuously decreasing. If yes, maintain the current data retrieval period of the collected end; otherwise, optimize the current data retrieval period of the collected end until the effective retrieval rate meets the condition.

[0038] The working principle of the strategy component will be explained below with reference to Example 4: The strategy component includes an optimization module, which continuously monitors the performance of the fetch process and builds a closed-loop optimization mechanism of "collection-optimization-analysis-adjustment" to achieve dynamic iterative optimization of the fetch cycle, such as... Figure 9 As shown, the specific scheme corresponding to this closed-loop optimization mechanism is as follows: (1) The optimization module uses the log information collected after the last pull strategy execution to determine whether each pull is the latest data before the server collects it, thereby distinguishing between valid pulls and invalid pulls; (2) Calculate the effective pull rate within a certain period and compare it with the preset effective pull rate target; (3) If the effective pull rate reaches the set target: If the effective pull rate does not show a continuous decline, the current pull cycle is maintained and no optimization is required; otherwise, the cycle is optimized. If the effective pull rate does not reach the target, the parameters of each module (such as the analysis module / execution module) are fine-tuned and the cycle is optimized. (4) Repeat the above process to achieve dynamic iterative optimization of the data retrieval cycle of the collected end, continuously improve the effective retrieval rate, and ensure the long-term stability of the system retrieval effect.

[0039] Based on the same technical concept, this disclosure also proposes a data acquisition method, including: Using the log component and policy component deployed on the client side, execute the following methods: Log data from the collected device is obtained through the log component. Based on the log data analysis, the data collection cycle pattern of the data collection terminal and the data update pattern of the collected terminal are observed. The data retrieval cycle of the collected terminal is dynamically adjusted based on the analysis results. Optimize system parameters based on the adjusted data retrieval cycle of the collected data source to correspond to the retrieval execution effect.

[0040] Based on the same technical concept, this disclosure also proposes a computer program product, which is stored in a computer-readable storage medium, and when executed by a processor, is used to at least implement the above-described data acquisition method.

[0041] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A data acquisition system, characterized in that, include: Log components and policy components deployed on the client-side; among them, The strategy component is configured as follows: Log data from the collected device is obtained through the log component. Based on the log data analysis, the data collection cycle pattern of the data collection terminal and the data update pattern of the collected terminal are observed. The data retrieval cycle of the collected terminal is dynamically adjusted based on the analysis results. Optimize system parameters based on the adjusted data retrieval cycle of the collected data source to correspond to the retrieval execution effect.

2. The system as described in claim 1, characterized in that, The logging component includes: The log collection module and log retrieval module are deployed on the client side. The strategy component is configured to obtain log data from the collected end through the log collection module and the log retrieval module; wherein, the log data includes: server collection behavior data, database retrieval behavior data, and database retrieval result data.

3. The system as described in claim 2, characterized in that, The strategy component is configured as follows: The server collection time interval data is determined based on the server's collection behavior data; Based on the statistical characteristic values ​​of the server's data collection time interval, it is determined whether the server's data collection cycle is stable. If so, the server's data collection time interval is fitted to a baseline data collection cycle, which is then used as one of the patterns of the data collection cycle at the acquisition end.

4. The system as described in claim 1, characterized in that, The strategy component is configured as follows: Determine the database fetch time interval data based on the database fetch behavior data; Based on the statistical characteristic values ​​of the database retrieval time interval data, it is determined whether the database updates data periodically. If so, the database retrieval time interval is predicted based on the database retrieval time interval data, and the predicted value is used as one of the data update rules of the collected end.

5. The system as described in claim 4, characterized in that, The policy component is also configured as follows: When determining non-periodic updates to database data, a prediction algorithm, including the weighted moving average method, is used to predict the database retrieval time interval based on the database retrieval time interval data, and the predicted value is used as one of the data update rules of the collected end.

6. The system as described in claim 1, characterized in that, The policy component is also configured as follows: Determine whether the data collection period of the acquisition terminal is stable. If not, the data retrieval period of the acquired terminal is reverted to the preset basic minimum guaranteed period. If so, the data retrieval period of the acquired terminal is adjusted according to the relationship between the data update interval of the acquired terminal and the preset update interval threshold.

7. The system as described in claim 6, characterized in that, The policy component is also configured as follows: When the data collection cycle at the acquisition end is determined to be stable, it is checked whether the data update interval at the acquired end is less than a preset update interval threshold. If so, the current data retrieval cycle at the acquired end is shortened; otherwise, the current data retrieval cycle at the acquired end is lengthened. When extending the current data retrieval period of the collected terminal, the constraints include: the extended data retrieval period of the collected terminal is not greater than the basic minimum guaranteed period.

8. The system as described in claim 1, characterized in that, The strategy component is configured as follows: Determine whether the effective retrieval rate corresponding to the adjusted data retrieval period of the collected end meets the requirement of being greater than the preset effective retrieval target and not continuously decreasing. If yes, maintain the current data retrieval period of the collected end; otherwise, optimize the current data retrieval period of the collected end until the effective retrieval rate meets the condition.

9. A data acquisition method, characterized in that, include: Using the log component and policy component deployed on the client side, execute the following methods: Log data from the collected device is obtained through the log component. Based on the log data analysis, the data collection cycle pattern of the data collection terminal and the data update pattern of the collected terminal are observed. The data retrieval cycle of the collected terminal is dynamically adjusted based on the analysis results. Optimize system parameters based on the adjusted data retrieval cycle of the collected data source to correspond to the retrieval execution effect.

10. A computer program product, said computer program product being stored in a computer-readable storage medium, characterized in that, When the computer program product is executed by a processor, it is used to implement at least the data acquisition method of claim 9.

Images