Data acquisition method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data acquisition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining current collection data item information through a target interface; generating collection configuration information based on the current collection data item information and a preset template, accessing an endpoint of the data agent collection tool according to the collection configuration information, and obtaining an index name and label data in a target format through the endpoint; the data agent acquisition tool determines a data provider by reading the acquisition configuration information, acquires to-be-processed data from the data provider, screens out target data from the to-be-processed data based on the acquisition configuration information, and converts the target data into an index name and label data in a target format; and sending the index name of the target format and the label data to a target database to complete data acquisition. By adopting the method, the efficiency of data acquisition configuration can be improved, so that the flexibility and expandability of data acquisition configuration are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data acquisition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] As the business grows, customer monitoring needs become more diverse, and personalized business operations become increasingly prominent. Customized monitoring scenarios pose significant challenges to monitoring data collection and storage. For example, the amount of data collected for monitoring increases with the continuous growth of business volume, resulting in high storage pressure. Furthermore, customized monitoring for customers often requires a relatively long delivery cycle, including requirement scheduling, customized development, and deployment, as well as significant manpower costs, leading to low efficiency in data collection and configuration. Summary of the Invention

[0003] Therefore, it is necessary to provide a data acquisition method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of data acquisition configuration, thereby enhancing the flexibility and scalability of data acquisition configuration, in response to the aforementioned technical problems.

[0004] Firstly, this application provides a data acquisition method, including:

[0005] The target interface is used to obtain the information of the currently collected data item. The target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured collection rules.

[0006] The data acquisition tool generates acquisition configuration information based on the current data item information and preset templates. It then accesses the endpoint of the data agent acquisition tool according to the acquisition configuration information and obtains the indicator name and tag data in the target format through the endpoint. The data agent acquisition tool determines the data provider by reading the acquisition configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the acquisition configuration information, converting the target data into indicator name and tag data in the target format.

[0007] Send the target format indicator name and label data to the target database to complete the data collection.

[0008] In one embodiment, the process involves generating collection configuration information based on the currently collected data item information and a preset template, accessing the endpoint of the data proxy collection tool according to the collection configuration information, and obtaining the target format indicator name and tag data through the endpoint. This includes: determining each content to be filled in the preset template; obtaining target data item information matching each content to be filled from the currently collected data item information; substituting the target data item information into the preset template to obtain the collection configuration information; obtaining the storage path corresponding to the collection configuration information; constructing an access request based on the storage path; sending the access request to the data proxy collection tool so that the data proxy collection tool obtains the collection configuration information according to the storage path in the access request, and obtains the target format indicator name and tag data based on the collection configuration information; and extracting the target format indicator name and tag data from the endpoint of the data proxy collection tool.

[0009] In one embodiment, the target data item information includes the access address of the data provider. Substituting the target data item information into a preset template to obtain collection configuration information includes: substituting the access address of the data provider into the preset template to obtain collection configuration information; after substituting the target data item information into the preset template to obtain collection configuration information, the method includes: accessing the storage path in the access request through a data proxy collection tool to obtain collection configuration information, and obtaining the access address of the data provider from the collection configuration information, and accessing the access address of the data provider to obtain the indicator name and tag data in the target format.

[0010] In one embodiment, the process of filtering target data from the data to be processed based on the collection configuration information and converting the target data into indicator names and tag data in the target format includes: reading the proxy collection configuration file through the data proxy collection tool, obtaining the pending indicator names from the proxy collection configuration file, determining the target indicator names from the pending indicator names based on the data to be processed, obtaining the target tags corresponding to the target indicator names from the proxy collection configuration file, determining the structured data corresponding to the target indicator names from the data to be processed, and determining the data corresponding to each target tag from the structured data, so as to convert the target data into indicator names and tag data in the target format.

[0011] In one embodiment, the currently collected data item information includes at least one of the following: customer name, domain name, effective time, and alarm rules.

[0012] In one embodiment, the method further includes: obtaining the alarm rule corresponding to the information of the currently collected data item through an alarm tool, sending a query request to the target database to query the indicator data that matches the alarm rule from the target database, and generating alarm information if the indicator data is not within the preset range corresponding to the alarm rule.

[0013] Secondly, this application also provides a data acquisition device, comprising:

[0014] The acquisition module is used to acquire information about the currently collected data item through the target interface. The target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured acquisition rules.

[0015] The data acquisition module is used to generate acquisition configuration information based on the current data item information and preset templates using a data crawling tool. It then accesses the endpoint of the data agent acquisition tool according to the acquisition configuration information and obtains the indicator name and tag data in the target format through the endpoint. The data agent acquisition tool determines the data provider by reading the acquisition configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the acquisition configuration information, converting the target data into indicator name and tag data in the target format.

[0016] The communication module is used to send the target format indicator name and label data to the target database to complete the data collection.

[0017] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0018] The target interface is used to obtain the information of the currently collected data item. The target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured collection rules.

[0019] The data acquisition tool generates acquisition configuration information based on the current data item information and preset templates. It then accesses the endpoint of the data agent acquisition tool according to the acquisition configuration information and obtains the indicator name and tag data in the target format through the endpoint. The data agent acquisition tool determines the data provider by reading the acquisition configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the acquisition configuration information, converting the target data into indicator name and tag data in the target format.

[0020] Send the target format indicator name and label data to the target database to complete the data collection.

[0021] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0022] The target interface is used to obtain the information of the currently collected data item. The target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured collection rules.

[0023] The data acquisition tool generates acquisition configuration information based on the current data item information and preset templates. It then accesses the endpoint of the data agent acquisition tool according to the acquisition configuration information and obtains the indicator name and tag data in the target format through the endpoint. The data agent acquisition tool determines the data provider by reading the acquisition configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the acquisition configuration information, converting the target data into indicator name and tag data in the target format.

[0024] Send the target format indicator name and label data to the target database to complete the data collection.

[0025] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0026] The target interface is used to obtain the information of the currently collected data item. The target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured collection rules.

[0027] The data acquisition tool generates acquisition configuration information based on the current data item information and preset templates. It then accesses the endpoint of the data agent acquisition tool according to the acquisition configuration information and obtains the indicator name and tag data in the target format through the endpoint. The data agent acquisition tool determines the data provider by reading the acquisition configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the acquisition configuration information, converting the target data into indicator name and tag data in the target format.

[0028] Send the target format indicator name and label data to the target database to complete the data collection.

[0029] The aforementioned data acquisition methods, devices, computer equipment, computer-readable storage media, and computer program products, through data scraping tools, acquire information about currently collected data items based on a target interface, transforming the trigger point for data acquisition from "full" to "on-demand." The target interface is an intelligent filter based on user-configured acquisition rules. The information about currently collected data items is determined based on the currently effective acquisition rules configured by the user. Therefore, it can filter the data to be collected from the source, thereby improving the scalability and flexibility of data acquisition configuration. Furthermore, after user configuration, the data scraping tool substitutes the acquired information about currently collected data items into a preset template to generate complete acquisition configuration information. Then, a data proxy acquisition tool reads this acquisition configuration information to perform relevant data acquisition, reducing manual configuration costs and enabling rapid onboarding of new customers. In addition, since the data scraping tool collects data based on the currently acquired data item information, it ensures that acquisition resources are dynamically adjusted according to actual needs, thereby dynamically expanding or shrinking the acquisition scope. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a diagram illustrating the application environment of a data acquisition method in one embodiment.

[0032] Figure 2 This is a flowchart illustrating a data acquisition method in one embodiment;

[0033] Figure 3 This is a schematic diagram of the architecture of a data acquisition system in one embodiment;

[0034] Figure 4 This is a flowchart illustrating the data acquisition method in another embodiment;

[0035] Figure 5 This is a flowchart illustrating the data acquisition method in yet another embodiment;

[0036] Figure 6 This is a structural block diagram of a data acquisition device in one embodiment;

[0037] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] With business development, customer monitoring needs have diversified, and personalized business operations have become increasingly prominent. Customized monitoring scenarios pose significant challenges to monitoring data collection and storage, necessitating a flexible, configurable, and rapidly deployable technology and solution. Traditional data collection methods typically employ full-volume access or a one-customer-one-plan or multi-customer-one-plan approach. While full-volume access to all customer data provides rapid response, it also incurs substantial storage pressure. One-customer-one-plan or multi-customer-one-plan approaches often involve lengthy delivery cycles, including requirement scheduling, customized development, and deployment, resulting in significant manpower costs. Furthermore, traditional data collection methods almost universally face the problem that data access expands incrementally only with business needs, making it difficult to achieve on-demand collection, on-demand monitoring, and dynamic changes in the collection scope, ultimately leading to wasted resources in collection and storage. Therefore, to address these technical issues, this application proposes at least the following embodiments.

[0040] The data acquisition method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system stores the data that server 104 needs to process. This data storage system can be integrated onto server 104 or placed on a cloud or other network server. Users can configure collection rules (e.g., configuring the data items to be collected, the effective time of the data item collection, etc.) through the user system deployed on terminal 102. Then, terminal 102 sends the user-configured collection rules to server 104. Server 104 has a target interface corresponding to the user system. The data scraping tool deployed on server 104 can obtain the information of the currently collected data items through this target interface. Next, based on the information of the currently collected data items and a preset template, collection configuration information is generated. The endpoint of the data proxy collection tool is accessed according to the collection configuration information, and the target format indicator name and tag data are obtained through the endpoint. Specifically, the data proxy collection tool deployed on server 104 determines the data provider by reading the collection configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the collection configuration information, converting the target data into the target format indicator name and tag data. Afterward, the data scraping tool sends the target formatted indicator names and label data to the target database to complete the data collection.

[0041] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0042] In one exemplary embodiment, such as Figure 2 As shown, a data acquisition method is provided, which can be applied to... Figure 1 The following explanation uses a data scraping tool as an example, including the following steps S202 to S206. Wherein:

[0043] Step S202: Obtain the information of the currently collected data item through the target interface.

[0044] The target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured collection rules.

[0045] The target interface refers to the interface of the user system, which is used to filter out the data items that need to be collected based on the collection rules set by the user.

[0046] For example, when setting data collection rules, in addition to specifying the data items to be collected (e.g., customers, domain names, link addresses, etc.), users can also set the effective time and alarm rules corresponding to the collection rule. In this way, the target interface can filter out the currently effective collection rules from multiple user-defined collection rules, allowing the data scraping tool to obtain the information of the currently collected data items through this interface. For example, the current time can be obtained, compared with the effective time of each collection rule, and the information of the currently collected data item can be determined based on the data item information corresponding to the collection rule with the effective time including the current time.

[0047] In some embodiments, the user system includes a configuration website. Users can log in to this website and configure information such as clients, domains, and alarm policies on the configuration page according to the actual monitoring scenario. Furthermore, the configuration page supports operations such as setting and deleting rule effective times to facilitate dynamic access and data collection and cleanup tasks. This configuration page uses a horizontally scalable external configuration interface.

[0048] In some embodiments, the target interface can be configured based on the output information of the data provider. For example, based on the data provider's interface, it can be divided into types such as customer accid, customerid, and domain, as well as the amount of data of the corresponding type returned in a single batch. The data collection configuration can then be optimized according to the amount of data the data provider can handle in a single request and the acceptable concurrency.

[0049] Step S204: Generate collection configuration information based on the current collected data item information and preset template, access the endpoint of the data agent collection tool according to the collection configuration information, and obtain the indicator name and tag data in the target format through the endpoint; the data agent collection tool determines the data provider by reading the collection configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the collection configuration information, and converts the target data into indicator name and tag data in the target format.

[0050] Step S206: Send the indicator name and label data in the target format to the target database to complete the data collection.

[0051] In this system, a data scraping tool is a component responsible for actively scraping (pulling) metric data based on configuration information; for example, the vmagent component. A data proxy collection tool is an agent program that doesn't generate data itself but can be accessed by the data scraping tool. It's responsible for obtaining raw data from other data sources, converting it to a specific format, and providing it to the data scraping tool. An example is json-exporter. An endpoint is a network access address (usually a URL) exposed by the data proxy collection tool, which the data scraping tool accesses to obtain data. If multiple data proxy collection tools exist, the data scraping tool can concurrently access the endpoints of multiple data proxy collection tools to extract data, thus achieving batch data collection. The target format refers to a standardized metric data format, typically the Prometheus text format, i.e., the form metric_name{"label_name"="label_value"}. The target database is a time-series database used to store the finally collected metric data, such as Prometheus or VictoriaMetrics. The data provider refers to the source of the raw data, such as an API service that provides JSON formatted status information.

[0052] For example, a management service on the server (or the data scraping tool itself) calls an internal API (i.e., the target interface), passing in the user identifier. This target interface queries a rules database and, based on the user's preset collection rules (such as "collect only customer A" or "collect only production environment"), returns a list of data items to be collected (e.g., a list containing customer names, domain names, and API addresses). The data scraping tool then obtains this list and, combined with a preset configuration template (such as a YAML file template), dynamically generates a complete collection configuration file. This collection configuration file defines the target to be scraped (i.e., the endpoint address of the data proxy collection tool). Next, the data scraping tool, based on the generated configuration, initiates an access request to the endpoint of the data proxy collection tool. Upon receiving this access request, the data proxy collection tool reads the collection configuration information passed to it by the data scraping tool or loaded by the data proxy collection tool itself, determining which data provider to access (e.g., http: / / some-app / api / status). Then, the data proxy collection tool accesses that data provider and obtains the raw data (e.g., a JSON response). Furthermore, based on the configuration information, specific fields are extracted from the raw data, mapped to metric names and label data, and finally formatted into the target format for metric names and label data. Finally, the data scraping tool obtains this formatted metric name and label data from the endpoint of the data agent acquisition tool, and then sends the metric name and label data to the remote target database for storage.

[0053] In the aforementioned data acquisition method, user-defined acquisition rules enable dynamic configuration of data acquisition tasks. Acquisition items can be added or removed based on user rules without manual modification or restarting of the data scraping tool, improving operational efficiency and system flexibility. Furthermore, by introducing a data proxy acquisition tool, complex, non-standard data sources (such as various APIs) can be decoupled from standard monitoring systems. The data scraping tool only needs to focus on standard endpoints, while the complexity of data transformation is handled by the proxy tool. In addition, a new data source is supported; only a new data proxy acquisition tool needs to be developed or configured, while the core acquisition and storage processes remain unchanged, thus improving the scalability of the data acquisition process.

[0054] In some embodiments, a data scraping tool generates collection configuration information based on the currently collected data item information and a preset template. The data scraping tool then accesses the endpoint of a data proxy collection tool according to the collection configuration information and obtains the target format indicator name and tag data through the endpoint. This includes: determining each content to be filled in the preset template using the data scraping tool; obtaining target data item information matching each content to be filled from the currently collected data item information; substituting the target data item information into the preset template to obtain the collection configuration information; obtaining the storage path corresponding to the collection configuration information using the data scraping tool; constructing an access request based on the storage path; sending the access request to the data proxy collection tool so that the data proxy collection tool obtains the collection configuration information according to the storage path in the access request and obtains the target format indicator name and tag data based on the collection configuration information; and extracting the target format indicator name and tag data from the endpoint of the data proxy collection tool.

[0055] For example, the data scraping tool internally maintains a configuration template. After the data scraping tool obtains data item information from the target interface, it uses a mechanism similar to Go templates to populate these values ​​into the placeholders {{...}} in the template, generating a complete, executable YAML configuration file, as shown below.

[0056] - job_name: log-delay

[0057] honor_timestamps: true

[0058] scrape_interval: 300s

[0059] scrape_timeout: 60s

[0060] metrics_path: / probe

[0061] http_sd_configs:

[0062] -url: 'http: / / xxx / v1 / canary / scrape_config / prometheus_http_sd?type=accid&metric=log_delay_bandwidth&batch=4'

[0063] basic_auth:

[0064] username: xxx

[0065] password: xxx

[0066] relabel_configs:

[0067] - source_labels: [__address__]

[0068] target_label: __address__

[0069] replacement: json-exporter:7979

[0070] - target_label: __param_module

[0071] replacement: cdnLogFiveDelayBW

[0072] - target_label: __param_target

[0073] replacement: http: / / xxxx / logapi / api / v3 / statistics

[0074] - source_labels: [client_id]

[0075] target_label: __param_client_id

[0076] - action: labeldrop

[0077] regex: client_id

[0078] Next, the generated configuration file is saved to a storage location accessible to the data scraping tool (such as an HTTP file server or shared storage volume), i.e., the storage path. The data scraping tool sends a request to the endpoint of the data scraping tool. This request can include an HTTP header (such as X-Config-Url: http: / / file-server / configs / customerA.yml) or use URL parameters (such as / metrics?config_path=...) to tell the data scraping tool where to read the configuration it needs for its operation. Upon receiving the request, the data scraping tool resolves the storage path and then downloads or reads the configuration information from that storage path. In other words, based on this collection configuration information, the data scraping tool knows which data source to access, how to parse the data, and ultimately generates metric data to return to the data scraping tool.

[0079] In this embodiment, by standardizing and automating the configuration generation process, the tediousness and potential for errors in manually writing configurations can be reduced, making it suitable for large-scale, multi-tenant scenarios. Furthermore, transmitting configurations via network paths makes configuration updates and distribution highly flexible, allowing for centralized management of the configurations of all data agent acquisition tools without needing to log into each machine to modify files.

[0080] In some embodiments, the target data item information includes the access address of the data provider. Substituting the target data item information into a preset template to obtain collection configuration information includes: substituting the access address of the data provider into the preset template to obtain collection configuration information. After substituting the target data item information into the preset template to obtain collection configuration information, the process includes: accessing the storage path in the access request through a data proxy collection tool to obtain collection configuration information, obtaining the access address of the data provider from the collection configuration information, and accessing the access address of the data provider to obtain the indicator name and tag data in the target format.

[0081] Specifically, the data scraping tool retrieves the data provider's access address (e.g., http: / / user-service / api / health) from the target interface. The scraping tool then populates this specific URL into the target field of a pre-defined template. Next, the data proxy acquisition tool obtains the final generated configuration file through the storage path in the access request sent by the scraping tool. The data proxy acquisition tool opens the configuration file, reads the access address field, and obtains http: / / user-service / api / health. It then sends an HTTP request to this address to retrieve the raw JSON data, which it then transforms and exposes.

[0082] This embodiment enables dynamic configuration across the entire process, from user-defined data collection rules to final data source access. Users only need to configure a data source address on the interface, and the entire system automatically completes the configuration flow from the crawler to the proxy and then to the data source. Users do not need to understand the internal workings of data crawling tools or data proxy collection tools; by providing the core information—the "access address of the data provider"—the system automatically completes the configuration of all technical details. This embodiment achieves automated configuration generation and consistency guarantees by transforming the user's business intent (which address to collect data from) into executable configuration through a template mechanism.

[0083] In some embodiments, the collection configuration information includes a proxy collection configuration file. Based on the collection configuration information, target data is filtered from the data to be processed, and the target data is converted into indicator names and tag data in the target format. This includes: reading the proxy collection configuration file through a data proxy collection tool, obtaining the pending indicator names from the proxy collection configuration file, determining the target indicator names from the pending indicator names based on the data to be processed, obtaining the target tags corresponding to the target indicator names from the proxy collection configuration file, determining the structured data corresponding to the target indicator names from the data to be processed, and determining the data corresponding to each target tag from the structured data, so as to convert the target data into indicator names and tag data in the target format.

[0084] The proxy data collection configuration file is specifically designed to guide the data proxy collection tool on how to operate. It defines which metrics to extract from the raw data and how to extract them. This configuration file may define a series of potential metric names. The data proxy collection tool checks if data corresponding to these metrics exists in the raw data. If it exists, the metric name is determined to be the target metric name and is actually generated. Structured data refers to the raw data, which typically has a hierarchical structure, such as JSON or XML data.

[0085] For example, the proxy collection configuration file for json-exporter might look like this:

[0086] cdnDelayBW:

[0087] metrics:

[0088] - name: log_delay

[0089] type: object

[0090] help: cdn log of 5min delay flow

[0091] epochTimestamp: "{ .timestamp}"

[0092] labelStrategy: cdnlog-statistics

[0093] path: "{.data[*]}"

[0094] labels:

[0095] accid: "{ .client_id}"

[0096] cust: "{ .client_id}"

[0097] host: "{ .channel}"

[0098] isp: "{ .isp_code}"

[0099] province: "{ .province}"

[0100] values:

[0101] bandwidth: "{ .flow}"

[0102] body:

[0103] templatize: true

[0104] content: '{ "start_time": "", "end_time": "", "interval": "5m","client_id":[{{index .client_id 0}}], "group_by": [ "province", "isp", "client_id", "channel" ], "metric": [ "flow" ]}'

[0105] strategy: cdnlog-statistics

[0106] headers:

[0107] content-type: application / json

[0108] strategy_params:

[0109] start_offset: 20

[0110] end_offset: 5

[0111] floor: 5

[0112] In this context, `name: log_delay` represents one of the pending metric names, `path: "{.data[*]}"` indicates that the value of the `data` field from the root path of the JSON is taken as the metric value, and `accid: "{ .client_id}"` under the `labels` tag indicates that the value of the `client_id` field from the root path of the JSON is taken as the `accid` tag. For explanations of the `cust`, `host`, `isp`, and `province` tags, please refer to the explanation of the `accid` tag.

[0113] Next, json-exporter reads this configuration file and obtains multiple pending metric names. json-exporter accesses the data provider to obtain the raw JSON data. When the raw JSON data contains structured data corresponding to a specific pending metric name, that pending metric name is determined to be the target metric name. Then, the values ​​of each tag are extracted from the same JSON to obtain the corresponding tag data. Finally, Prometheus formatted output is generated.

[0114] In this embodiment, the data from the data provider can be filtered through the data collection configuration file, accurately mapping raw data of any structure to a standard indicator model. The data transformation logic is controlled through the configuration file. Fields defined in the data collection configuration file and actually existing in the raw data are converted into indicators, avoiding the generation of useless or incorrect indicators. In addition, through a tagging mechanism, contextual information (such as environment, version, instance ID, etc.) from the raw data is attached to the indicators, making subsequent data queries, alerts, and visualizations richer and more accurate.

[0115] In some embodiments, such as Figure 3 As shown, the collected data can also be analyzed to generate alarm information in a timely manner when data is abnormal. The method also includes: obtaining the alarm rules corresponding to the information of the currently collected data items through the alarm tool, sending a query request to the target database to query the indicator data that matches the alarm rules from the target database, and generating alarm information if the indicator data is not within the preset range corresponding to the alarm rules.

[0116] Among them, the alarm tool is a standalone monitoring and alarm component (such as...) Figure 3 (canary-alert in the context of alerting). The alerting tool is responsible for evaluating alerting rules and sending alert notifications when a rule is triggered. Alerting rules define "when to trigger an alert" and can be a logical expression, usually based on the PromQL query language. For example, cpu_usage{customer="AcmeCorp"} > 90. Preset ranges refer to the thresholds or conditions defined in the alerting rules, such as > 90, == 0, or outside[5m], etc.

[0117] For example, a management service or alerting tool periodically calls a target interface or accesses a rules database to obtain all active "currently collected data item information" and extracts the `alert_rule` field. The alerting tool loads these obtained alert rules into memory. These rules may be PromQL expressions obtained directly from the interface, or they may be dynamically generated based on data item information (such as customer name). The alerting tool then sends alerts to the target database (such as...) at a set frequency (e.g., every 30 seconds). Figure 3 The system sends a query request to vmstorage-0~vmstorage-n. The corresponding query request is executed via vmselect. The request content is a PromQL expression from the loaded alerting rule; for example, it queries the value of cpu_usage{customer="Acme Corp"} within the last 5 minutes. The target database returns the query results. The alerting tool compares the results to a preset range (e.g., > 90). If the condition is met (e.g., CPU usage consistently above 90%), the alerting tool determines that an alert has been triggered. The alerting tool generates an alert message containing the alert name, severity level, associated customer, current metric value, etc., and sends it via a notification tool (e.g., ...). Figure 3 The canary-notify tool in the software sends the notification to the relevant personnel based on a pre-configured channel (such as email).

[0118] In this embodiment, alarm rules are dynamically associated with the collected data items. When a new customer and its collection task are added, the corresponding alarm rules are automatically loaded and evaluated, realizing dynamic expansion of the alarm strategy. Since the collected indicator data contains rich tags (such as customer name and domain name), alarm rules can pinpoint problems very accurately, such as triggering alarms only for specific services of specific customers, reducing alarm noise.

[0119] In one exemplary embodiment, combined with Figures 3 to 5 As shown, if a user wants to monitor certain data items, they can do so through the corresponding configuration webpage of their system (e.g., ...). Figure 3The configuration of data items for collection is done via the canary-web interface. During configuration, users can configure data such as customers, domains, and alarm policies according to their actual monitoring scenarios. Furthermore, in response to user input of data item information (e.g., customer, domain), a set of optional alarm rules corresponding to the input data item information is displayed. In response to user selection of a target alarm rule from the optional alarm rule set, the target alarm rule is bound to the aforementioned data item information. Additionally, to enable on-demand data collection or monitoring, in response to user input of collection configuration information (e.g., customer, domain), an optional effective time corresponding to the input data item information is displayed. In response to user selection of a target effective time from the optional effective times, the target effective time is bound to the aforementioned data item information. Thus, the target interface can filter the currently effective data item information based on the target effective time corresponding to each data item information and the current time. It is understandable that when a user deletes a data item from the database or the time of a data item expires, the target interface will not filter the expired data item into the current data item during the filtering process, thus facilitating dynamic access and cleanup of data collection tasks.

[0120] In other words, for existing monitoring configurations, new time-based activation policies will automatically increase the returned results starting from the canary interface side, thus achieving incremental data collection across the entire chain. For existing monitoring configurations, new time-based expiration policies will automatically decrease the returned results starting from the canary interface side, thus achieving reduced data collection across the entire chain. For newly added monitoring configurations, new customers and new activation policies will automatically increase the returned results starting from the canary interface side, thus achieving incremental data collection across the entire chain.

[0121] Next, the data scraping tool obtains information about the currently collected data items through the target interface. The target interface can customize the return parameter types and supports multiple parameter types as input parameters. Depending on the data provider's interface, parameter types can include customer, domain, etc. (e.g., accid, clientid, domain). Furthermore, the data collection configuration can be optimized based on the data provider's capacity for processing data in a single request and their acceptable concurrency levels. Specifically, the amount of data of the corresponding type returned by the target interface in a single batch can be determined based on the data provider's capacity for processing data in a single request and their acceptable concurrency levels. (Refer to...) Figure 4 and Figure 5For example, the input parameters of the target interface can be: Type (collection dimension) = accid (accountid customer identifier) ​​& metric (collection metric) = log_delay_bandwidth (log bandwidth delay) & batch = 4 (number of batches, each batch contains a maximum of 4 clientids). The interface data structure of the target interface can be represented as follows, as you can see that based on the input parameters, each batch of the target interface contains a maximum of 4 clientids. [

[0123] {

[0124] "targets": [

[0126] "1.1.1.1"

[0127] ],

[0128] "labels":

[0129] {

[0130] "client_id": "client_id1,client_id2,client_id3,client_id4"

[0131] }

[0132] },

[0133] {

[0134] "targets": [

[0136] "1.1.1.1"

[0137] ],

[0138] "labels":

[0139] {

[0140] "accid": "client_id5,client_id6"

[0141] }

[0142] } ]

[0144] Next, based on the current data item information and the preset template, collection configuration information is generated. The endpoint of the data agent collection tool is accessed according to the collection configuration information, and the target format indicator name and tag data are obtained through the endpoint. The data agent collection tool determines the data provider by reading the collection configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the collection configuration information. The target data is converted into the target format indicator name and tag data. The target format indicator name and tag data are sent to the target database to complete the data collection.

[0145] During the startup process of the data scraping tool, the target storage path is obtained. This path is then written into the subsequent data scraping configuration information generated based on the currently collected data item information. Additionally, the database address of the target database also needs to be configured when configuring the data scraping tool. This allows the data scraping tool to extract the target format indicator names and tag data from the data proxy collection tool and write this data to the target database corresponding to the specified database address. A preset template is pre-set to generate the data scraping configuration information, enabling the data scraping tool to interface with the target interface and the data proxy collection tool. For example, after extracting the current data item information from the target interface, the preset template is filled in based on this information to generate data scraping configuration information that matches the current data item information. Subsequently, the data proxy collection tool can also use this configuration information to perform data transformation and data collection. When a data scraping tool performs a data collection task, it can send an access request to a data proxy collection tool. This allows the data proxy collection tool to obtain the collection configuration information according to the storage path specified in the access request. It then successfully accesses the data provider through the access address stored in the collection configuration information and converts the raw data obtained from the data provider into target format indicator names and label data according to the data parsing method specified in the collection configuration information. The access request sent by the data proxy collection tool to the data provider can be found below:

[0146] curl --location 'http: / / xxxx / logapi / api / v3 / statistics' \

[0147] --header 'Content-Type: application / json' \

[0148] --data '{

[0149] "start_time": "1723620300",

[0150] "end_time": "1723620600",

[0151] "interval": "5m",

[0152] "client_id": [

[0153] client_id1,client_id2,client_id3,client_id4

[0154] ],

[0155] "group_by": [

[0156] "province",

[0157] "isp",

[0158] "client_id",

[0159] "channel"

[0160] ],

[0161] "metric": [

[0162] "flow" ]

[0164] }'

[0165] It is understood that, in this embodiment, in combination with Figure 5 As shown, if it is necessary to expand the collection of new monitoring types, the following configuration can be performed: For the data proxy collection tool, perform the new configuration operation. Configure the input parameters, output parameters, metric names, and tags of the access component corresponding to the new monitoring type; for the data crawling tool, perform the collection configuration operation, confirming the input parameter type and collection target according to the data proxy collection tool configuration; define the metric name, tag information, and configure monitoring rules according to the data proxy collection tool, and provide them to users. In other words, for the access of new monitoring types, the above three configuration steps are sufficient, which can greatly shorten the iterative development and collection scheduling cycle.

[0166] Through the above steps, the system focuses on data collection based on monitoring needs, enabling on-demand access and storage, and dynamically expanding or shrinking the collection scope. Regarding data access, users define monitoring objects and configure monitoring rules according to customer requirements. The interface provides external access for querying monitored customer and target data based on monitoring metrics, achieving dynamic access, collection, storage, and alerting. In terms of data cleanup, the system has automatic dynamic data cleanup capabilities; after user monitoring policies are cleaned or expire, the corresponding collection targets will be automatically removed.

[0167] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0168] Based on the same inventive concept, this application also provides a data acquisition device for implementing the data acquisition method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data acquisition device embodiments provided below can be found in the limitations of the data acquisition method described above, and will not be repeated here.

[0169] In one exemplary embodiment, such as Figure 6 As shown, a data acquisition device 600 is provided, including: an acquisition module 601, a data collection module 602, and a communication module 603, wherein:

[0170] The acquisition module 601 is used to acquire information of the currently collected data item through the target interface. The target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured acquisition rules.

[0171] The acquisition module 602 is used to generate acquisition configuration information based on the current data item information and preset template through the data capture tool, access the endpoint of the data agent acquisition tool according to the acquisition configuration information, and obtain the indicator name and tag data in the target format through the endpoint; the data agent acquisition tool determines the data provider by reading the acquisition configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the acquisition configuration information, and converts the target data into indicator name and tag data in the target format.

[0172] The communication module 603 is used to send the target format indicator name and label data to the target database to complete the data acquisition.

[0173] In some embodiments, in generating collection configuration information based on the current collected data item information and a preset template, accessing the endpoint of the data proxy collection tool according to the collection configuration information, and obtaining the target format indicator name and tag data through the endpoint, the collection module 602 is specifically used to: determine each content to be filled in the preset template, obtain target data item information matching each content to be filled from the current collected data item information, substitute the target data item information into the preset template to obtain collection configuration information; obtain the storage path corresponding to the collection configuration information, construct an access request based on the storage path, send the access request to the data proxy collection tool, so that the data proxy collection tool obtains the collection configuration information according to the storage path in the access request, and obtains the target format indicator name and tag data based on the collection configuration information; and extract the target format indicator name and tag data from the endpoint of the data proxy collection tool.

[0174] In some embodiments, when the target data item information includes the access address of the data provider, and the target data item information is substituted into a preset template to obtain the collection configuration information, the collection module 602 is specifically used to: substitute the access address of the data provider into the preset template to obtain the collection configuration information; after substituting the target data item information into the preset template to obtain the collection configuration information, the collection module 602 is specifically used to access the storage path in the access request through the data proxy collection tool to obtain the collection configuration information, and obtain the access address of the data provider from the collection configuration information, and access the access address of the data provider to obtain the indicator name and tag data in the target format.

[0175] In some embodiments, in terms of filtering target data from the data to be processed based on the collection configuration information and converting the target data into indicator names and tag data in the target format, the collection module 602 is specifically used to read the proxy collection configuration file through the data proxy collection tool, obtain the pending indicator names from the proxy collection configuration file, determine the target indicator names from the pending indicator names based on the data to be processed, obtain the target tags corresponding to the target indicator names from the proxy collection configuration file, determine the structured data corresponding to the target indicator names from the data to be processed, and determine the data corresponding to each target tag from the structured data, so as to convert the target data into indicator names and tag data in the target format.

[0176] In some embodiments, the data acquisition device 600 is further configured to: obtain the alarm rule corresponding to the information of the currently collected data item through an alarm tool, send a query request to the target database to query the indicator data that matches the alarm rule from the target database, and generate alarm information if the indicator data is not within the preset range corresponding to the alarm rule.

[0177] Each module in the aforementioned data acquisition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0178] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores data related to data acquisition, such as acquisition configuration rules, target format indicator names, and tag data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data acquisition method.

[0179] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0186] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data acquisition method, characterized in that, The method includes: The target interface is used to obtain the information of the currently collected data item, which is used to filter the information of the currently collected data item from the information of at least one configured data item based on the user-configured collection rules. Based on the current collected data item information and the preset template, collection configuration information is generated. The endpoint of the data proxy collection tool is accessed according to the collection configuration information, and the target format indicator name and tag data are obtained through the endpoint. The data proxy collection tool determines the data provider by reading the collection configuration information, collects the data to be processed from the data provider, and filters the target data from the data to be processed based on the collection configuration information, and converts the target data into the target format indicator name and tag data. The target format indicator name and label data are sent to the target database to complete the data collection.

2. The method according to claim 1, characterized in that, The process of generating collection configuration information based on the currently collected data item information and a preset template, accessing the endpoint of the data proxy collection tool according to the collection configuration information, and obtaining the target format indicator name and tag data through the endpoint includes: Each content to be filled in the preset template is determined, target data item information matching each content to be filled is obtained from the current data item information, and the target data item information is substituted into the preset template to obtain the collection configuration information; Obtain the storage path corresponding to the collection configuration information, construct an access request based on the storage path, and send the access request to the data proxy collection tool so that the data proxy collection tool obtains the collection configuration information according to the storage path in the access request, and obtains the indicator name and tag data in the target format based on the collection configuration information; Extract the target format indicator name and label data from the endpoint of the data agent acquisition tool.

3. The method according to claim 2, characterized in that, The target data item information includes the access address of the data provider. Substituting the target data item information into the preset template to obtain the collection configuration information includes: Substitute the access address of the data provider into the preset template to obtain the collection configuration information; After substituting the target data item information into the preset template to obtain the collection configuration information, the following steps are included: The data acquisition tool obtains the acquisition configuration information by accessing the storage path in the access request, and obtains the access address of the data provider from the acquisition configuration information. The access address of the data provider is then accessed to obtain the indicator name and tag data in the target format.

4. The method according to claim 1, characterized in that, The step of filtering target data from the data to be processed based on the collection configuration information, and converting the target data into indicator names and tag data in the target format, includes: The data acquisition tool reads the proxy acquisition configuration file, obtains the pending indicator name from the proxy acquisition configuration file, determines the target indicator name from the pending indicator name based on the data to be processed, obtains the target label corresponding to the target indicator name from the proxy acquisition configuration file, determines the structured data corresponding to the target indicator name from the data to be processed, and determines the data corresponding to each target label from the structured data, so as to convert the target data into indicator name and label data in the target format.

5. The method according to any one of claims 1 to 2, characterized in that, The currently collected data items include at least one of the following: customer name, domain name, effective time, and alarm rules.

6. The method according to claim 1, characterized in that, The method further includes: The alarm tool obtains the alarm rule corresponding to the information of the currently collected data item, sends a query request to the target database to query the indicator data that matches the alarm rule from the target database, and generates alarm information if the indicator data is not within the preset range corresponding to the alarm rule.

7. A data acquisition device, characterized in that, The device includes: The acquisition module is used to acquire information of the currently collected data item through a target interface, wherein the target interface is used to filter the information of the currently collected data item from the information of at least one configured data item based on user-configured acquisition rules. The data acquisition module is used to generate acquisition configuration information based on the current data item information and a preset template using a data crawling tool, access the endpoint of the data proxy acquisition tool according to the acquisition configuration information, and obtain the indicator name and tag data in the target format through the endpoint; the data proxy acquisition tool determines the data provider by reading the acquisition configuration information, acquires the data to be processed from the data provider, filters the target data from the data to be processed based on the acquisition configuration information, and converts the target data into indicator name and tag data in the target format; The communication module is used to send the target format indicator name and label data to the target database to complete the data collection.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images