Index data acquisition method, device and equipment
By using a key-value pair storage and grouping aggregation method in a multi-cloud environment to transfer indicator data, the security and scalability issues of cross-environment indicator data collection and aggregation are solved, data isolation and system expansion are achieved, and the efficiency of monitoring indicator aggregation in a hybrid cloud environment is improved.
Patent Information
- Application Number
- CN202511451228.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-11
- Publication Date
- 2026-01-13
AI Technical Summary
In multi-cloud environments, existing technologies struggle to efficiently and securely collect and aggregate cross-environment indicator data, especially in large-scale scenarios where they suffer from insufficient security, limited performance, or poor adaptability.
The system receives monitoring indicator data through the indicator data relay module and stores it as key-value pairs. It generates a key list using tenant personalized identifiers, groups and aggregates the indicators to generate an indicator aggregation and pull configuration file. The monitoring data acquisition module pulls data from the relay module according to the configuration file. It adopts a stateless design to support scalability.
It achieves data isolation and enhanced security in multi-cloud environments, reduces port scanning and traversal pressure, supports horizontal system scaling, and improves security and scalability in large-scale scenarios.
Smart Images
Figure CN121333972A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically, to a method, apparatus, and equipment for acquiring indicator data. Background Technology
[0002] With the continuous development of cloud computing technology, enterprise IT architectures are gradually evolving towards multi-cloud environments, achieving flexible deployment and expansion of business by integrating public, private, and hybrid cloud resources. In this context, real-time and accurate monitoring of metrics data generated by servers, network devices, and applications across various cloud environments has become crucial for ensuring stable system operation, optimizing resource allocation, and quickly troubleshooting. However, due to differences in the technical architecture, data formats, and access mechanisms of different cloud platforms, efficiently collecting and aggregating cross-environment metrics data remains a pressing technical challenge for the industry. Currently, there are three main approaches to collecting and aggregating metrics data in multi-cloud environments: First, the data is typically pushed proactively from the monitored endpoint to the monitoring server. This approach requires the server to expose its data receiving port to meet the push requirements. Second, the data collection end directly pulls data from the monitored endpoint using a proxy or network penetration technology. However, penetration technology itself has security vulnerabilities, and when the number of monitored endpoints is large, the server's traversal and data retrieval method can put significant performance pressure on the system. Third, an aggregation tool is deployed at the network edge, which proxies the server to complete the metrics collection, and then the server retrieves the data uniformly from the aggregation end. This approach is the most widely used due to its strong applicability, but it places extremely high demands on the collection performance and reliability of the edge aggregation tool, making it difficult for existing products to achieve ideal application results in large-scale hybrid cloud systems. Summary of the Invention
[0003] To overcome the shortcomings of existing indicator data collection and aggregation technologies, such as insufficient security, limited performance, or poor adaptability to large-scale scenarios, this invention proposes the following technical solution: Firstly, this invention proposes a method for acquiring indicator data, comprising: The indicator data relay module receives monitoring indicator data pushed by the data acquisition server, constructs key-value pairs according to preset key-value pair storage rules, and stores the monitoring indicator data as the value; wherein, the key of the key-value pair includes tenant personalized identification data. The indicator aggregation and retrieval configuration dynamic generation module generates a key list using the identity data of the tenant's personalized data, and uses the identity data of each key and other identification data of the tenant's personalized data to group and aggregate each key in the key list to generate the indicator aggregation and retrieval configuration file. The monitoring data acquisition module retrieves the configuration file based on the aggregated indicators and pulls the corresponding monitoring indicator data from the indicator data transfer module.
[0004] As a preferred technical solution, the tenant personalized identification data includes at least a tenant prefix identifier, an indicator type identifier, or an instance identifier; the received and pushed monitoring indicator data is configured into key-value pairs according to a preset key-value pair storage rule, and the monitoring indicator data is stored as the value, including: The indicator data relay module receives monitoring indicator data pushed by the edge service, extracts the tenant prefix identifier, indicator type identifier, and instance identifier corresponding to the monitoring indicator data, and generates a key by combining them in the order of "tenant prefix identifier + indicator type identifier + instance identifier". The monitoring indicator data is used as a value, and a key-value pair is formed with the key, and stored in a preset storage system.
[0005] As a preferred technical solution, the storage system is an external database that supports key-value pair structures. The key-value pairs are set with an expiration time when they are created, and the expiration time is refreshed when the key-value pairs are updated. Unupdated key-value pairs are automatically deleted after the expiration time is reached.
[0006] As a preferred technical solution, after constructing key-value pairs according to preset key-value pair storage rules and storing monitoring indicator data as values, the method further includes: The key-value pairs are written to an external Redis database. The external Redis database uses a short connection method, and the connection is closed immediately after each write. A uniform expiration time is set for the key during the write, and the expiration time is refreshed when the same key-value pair is written again in a subsequent write.
[0007] As a preferred technical solution, the indicator aggregation and retrieval configuration dynamic generation module generates a key list using the identity data of the tenant's personalized data, including: The indicator data transfer module uses the key scanning interface provided by the external Redis database, with the tenant prefix identifier as the scanning prefix, to read all keys that match the scanning prefix in batches, and accumulates the keys returned by each scan to form a complete key list.
[0008] As a preferred technical solution, the step of grouping and aggregating each key in the key list using the identity data of each key and other identifier data of the tenant personalized identifier data to generate an indicator aggregation and retrieval configuration file includes: For each key in the key list, extract the tenant prefix identifier, metric type identifier, and instance identifier, and then concatenate the tenant prefix identifier and the metric type identifier to form a grouping key; Using the grouping key as an index, the instance identifiers with the same grouping key are grouped into the same set to obtain the grouping result; Iterate through the grouping results, extract each grouping key and its corresponding set of instance identifiers in turn, use the extracted grouping key as the file name, and write all instance identifiers in the corresponding set of instance identifiers in a fixed format to form a separate configuration file. Write the paths of all the individual configuration files into the main configuration template in sequence to generate the metric aggregation pull configuration file.
[0009] As a preferred technical solution, the monitoring data acquisition module retrieves the corresponding monitoring indicator data from the indicator data transfer module according to the indicator aggregation and retrieval configuration file, including: The monitoring system loads the aggregated pull configuration file of the metrics, parses the main configuration template to obtain the paths of all individual configuration files, and reads the tenant prefix identifier, metric type identifier and instance identifier recorded in each individual configuration file; For each instance identifier, an HTTP GET request carrying the tenant prefix identifier, indicator type identifier, and instance identifier is sent to the indicator data relay module according to the access path recorded in the individual configuration file. After receiving a request, the indicator data relay module uses the tenant prefix identifier, indicator type identifier, and instance identifier carried in the request as a key to read the monitoring indicator data corresponding to the key from the external Redis database and return it to the monitoring system.
[0010] Secondly, the present invention also proposes an indicator data acquisition device, applied in the indicator data acquisition method as described in any embodiment of the first aspect, comprising: The indicator data relay module is used to receive monitoring indicator data pushed by the data acquisition server, construct key-value pairs according to preset key-value pair storage rules, and store the monitoring indicator data as values; wherein, the key of the key-value pair includes tenant personalized identification data. The indicator aggregation and retrieval configuration dynamic generation module is used to generate a key list using the identity data of the tenant personalized data, and to group and aggregate each key in the key list using the identity data of each key and other identification data of the tenant personalized data to generate an indicator aggregation and retrieval configuration file. The monitoring data acquisition module is used to retrieve the corresponding monitoring indicator data from the indicator data transfer module based on the indicator aggregation and pull configuration file.
[0011] Thirdly, the present invention also proposes an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the operations performed by the indicator data acquisition method as described in any of the embodiments of the first aspect.
[0012] Fourthly, the present invention also proposes a method for acquiring indicator data in a hybrid cloud environment. The method for acquiring indicator data in a hybrid cloud environment is based on the indicator data acquisition method described in any scheme of the first aspect to realize the transit and aggregation of monitoring indicator data of edge nodes in different cloud regions.
[0013] The beneficial effects of the present invention include at least the following: (1) The indicator data acquisition method proposed in this invention receives monitoring indicator data pushed by the data acquisition server through the indicator data transfer module, and stores the monitoring indicator data according to the preset key-value pair storage rules with keys containing tenant personalized identification data; then the indicator aggregation and retrieval configuration dynamic generation module generates a key list using the identity identification data of tenant personalized data, and groups and aggregates each key in the key list to generate an indicator aggregation and retrieval configuration file. Finally, the monitoring data acquisition module retrieves the corresponding monitoring indicator data from the indicator data transfer module according to the configuration file. This can avoid exposing the write port to the outside, reduce the pressure of port scanning and traversal, realize data isolation between tenant and business dimensions within the key, and, together with the stateless design, enable the system to be horizontally scaled in a multi-cloud environment, avoid data conflicts and state inconsistencies, thereby improving the security and scalability in large-scale scenarios.
[0014] (2) The method for acquiring indicator data in a hybrid cloud environment proposed in this invention deploys indicator data transfer modules in each cloud region of the hybrid cloud. After the data acquisition server pushes the monitoring indicator data to the local transfer module, it stores the data uniformly with the tenant's personalized identification data as the key. The central side pulls the cross-cloud region read-only based on the generated indicator aggregation configuration file, without exposing the write port in each cloud region, eliminating cloud penetration and repeated traversal. Cross-cloud data isolation is achieved at the tenant dimension within the key. The stateless design supports the expansion of transfer instances as needed, thereby improving the security, scalability and operation and maintenance efficiency of monitoring indicator aggregation in the hybrid cloud environment. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating the indicator data acquisition method provided in an embodiment of the present invention.
[0016] Figure 2 This is a deployment architecture diagram of the indicator data acquisition method provided in the embodiments of the present invention.
[0017] Figure 3 This is a system architecture diagram of the indicator data acquisition method provided in an embodiment of the present invention.
[0018] Figure 4 This is a timing diagram of the indicator data acquisition method provided in an embodiment of the present invention.
[0019] Figure 5 This is an architectural diagram of the indicator data acquisition device provided in an embodiment of the present invention.
[0020] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation
[0021] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention and not for limiting the scope of protection of the present invention.
[0022] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0023] Example 1 This invention proposes a method for obtaining indicator data, such as... Figure 1 and Figure 2 As shown, Figure 1 This is a flowchart illustrating the indicator data acquisition method provided in an embodiment of the present invention. Figure 2 This is a deployment architecture diagram of the indicator data acquisition method provided in this embodiment of the invention. The method includes the following steps: S1: The indicator data transfer module receives the monitoring indicator data pushed by the data acquisition server, constructs key-value pairs according to the preset key-value pair storage rules, and stores the monitoring indicator data as the value; wherein, the key of the key-value pair includes the tenant personalized identification data.
[0024] In this embodiment, the data acquisition server includes an edge server.
[0025] In some implementation scenarios, the identity identifier data of a user's tenant personalized data is the tenant prefix identifier.
[0026] S2: The indicator aggregation and retrieval configuration dynamic generation module generates a key list using the identity data of the tenant's personalized data, and uses the identity data of each key and other identification data of the tenant's personalized data to group and aggregate each key in the key list to generate the indicator aggregation and retrieval configuration file.
[0027] S3: The monitoring data acquisition module retrieves the configuration file based on the index aggregation and pulls the corresponding monitoring index data from the index data transfer module.
[0028] like Figure 3 As shown, the system architecture of the indicator data acquisition method of this invention is as follows: (1) API Interface Layer: Based on the Gin framework, an HTTP service is built to receive external POST and GET requests. Through the route registration mechanism, the path [ / metrics / job / :job / instance / :instance] of the POST request is mapped to the data processing function [SetHandler] for pushing metric data (where job corresponds to the metric type and instance corresponds to the target instance parameter). The path [ / metrics:module={job}&target={instance}] of the GET request is mapped to the data processing function [GetHandler] for querying the stored metric data (where module and target correspond to the formal parameters of job and instance, respectively).
[0029] (2) Data processing layer: includes context manager and request handler. The context manager encapsulates the request context of the Gin framework and system configuration parameters into a unified processing context. The request handler calls [SetHandler] and [GetHandler] respectively according to the request type. [SetHandler] deserializes the data message of the POST request, extracts the [job], [instance] in the URL and the monitoring metric data in the body and passes it to the storage management layer. [GetHandler] parses the [module] and [target] parameters of the corresponding [job] and [instance] from the URL of the GET request, extracts the corresponding data from the storage management layer and serializes it into the output format and returns it to the requesting end.
[0030] (3) Storage management layer: Based on the Redis client library, it adopts short connection mode for data access (a new Redis connection is created for each data processing request and closed immediately after processing is completed); the key-value design adopts the format "[prefix][job]__[instance]" ([prefix] is a multi-tenant unique identifier configuration item, and [job] and [instance] are passed in by the data processing layer). The monitoring indicator data is written to the storage in the form of key-value pairs. When the same key exists, it is overwritten and when it does not exist, it is directly inserted. Each key is set with an expiration time when it is created, and the expiration time is refreshed when it is updated. It is automatically deleted after expiration.
[0031] (4) Configuration Management Layer: Based on the Viper configuration management library, the system manages the necessary configuration parameters (including service listening address, self-monitoring address, tenant prefix identifier, Redis connection parameters, key expiration time, log configuration, etc.) through YAML format configuration files. The configuration files are parsed and configuration items are generated when the system starts up, and the system runs according to the startup parameters.
[0032] In the specific implementation process, the edge service first captures the monitoring indicator data of this node and pushes it to the indicator data relay module via an HTTP POST request; after receiving the data, the indicator data relay module... Key-value pairs are constructed according to preset key-value pair storage rules. Monitoring metric data is written as values to an external Redis database, and the write command carries the preset expiration time of the configuration file. The keys of the key-value pairs include tenant personalized identification data. Subsequently, the dynamic generation module for metric aggregation and retrieval configuration traverses all keys containing tenant personalized identification data through the Redis key scanning interface, accumulates a key list, and uses the identity identification data of each key and other identification data of the tenant personalized identification data to group and aggregate each key in the key list to generate a metric aggregation and retrieval configuration file. Finally, the monitoring system loads the configuration file and sends an HTTP GET request to the metric data transfer module according to the access path recorded in the individual configuration file. The metric data transfer module uses the tenant prefix identifier, metric type identifier, and instance identifier in the request as the key, reads the monitoring metric data from the external Redis database, and returns it, completing the unified aggregation of metric data.
[0033] Understandably, the system receives monitoring indicator data pushed by the data acquisition server through the indicator data relay module, stores the monitoring indicator data according to preset key-value pair storage rules, with keys containing tenant personalized identification data; then, the indicator aggregation and retrieval configuration dynamic generation module generates a key list using the identity identification data of tenant personalized data, and groups and aggregates each key in the key list to generate an indicator aggregation and retrieval configuration file. Finally, the monitoring data acquisition module retrieves the corresponding monitoring indicator data from the indicator data relay module according to the configuration file. This avoids exposing write ports to the outside, reduces port scanning and traversal pressure, achieves data isolation between tenants and business dimensions within keys, and, together with the stateless design, enables the system to scale horizontally in multi-cloud environments, avoids data conflicts and state inconsistencies, thereby improving security and scalability in large-scale scenarios.
[0034] Example 2 This invention improves upon the indicator data acquisition method proposed in Example 1, such as... Figure 4 As shown, Figure 4 This is a timing diagram of the indicator data acquisition method provided in an embodiment of the present invention.
[0035] In this embodiment, the tenant personalized identification data includes at least a tenant prefix identifier, an indicator type identifier, or an instance identifier. The received and pushed monitoring indicator data is constructed into key-value pairs according to a preset key-value pair storage rule, and the monitoring indicator data is stored as the value, including: The indicator data relay module receives monitoring indicator data pushed by the pushbot module through the [PushHandler] function, extracts the tenant prefix identifier, indicator type identifier, and instance identifier corresponding to the monitoring indicator data, and generates a key by combining them in the order of "tenant prefix identifier + indicator type identifier + instance identifier". The pushbot module, based on the Viper configuration management library, configures system parameters through a YAML configuration file. Configuration items include the mps server address, the address of the pulled monitoring indicator (supporting multiple addresses), the pull timeout, the tenant prefix identifier, and log configuration. It calls the [GetHandler] and [PushHandler] functions to pull and push monitoring indicators from the target server. [GetHandler] sends an HTTP GET request to pull the monitoring indicators using the service address of the pulled end parsed in the configuration management and temporarily stores them before passing them to the [PushHandler] function to push them to the indicator data relay module.
[0036] The monitoring indicator data is used as a value, and a key-value pair is formed with the key, and stored in a preset storage system.
[0037] As an example, the indicator data relay module receives monitoring indicator data pushed by the edge service, performs deserialization processing on the received request message, extracts the indicator type identifier (job), instance identifier (instance) from the URL, and the monitoring indicator data from the message; combines the configured tenant prefix identifier (prefix) and generates a key according to the format "[prefix][job]__[instance]"; uses the deserialized monitoring indicator data as the value, forms a key-value pair with the generated key, and stores it in the preset storage system.
[0038] In this embodiment, the storage system is an external database that supports key-value pair structures. The storage system uses Redis as the data storage server and supports multiple connection methods such as single-node Redis, sentinel mode, and cluster mode. By setting a maximum memory limit on the amount of content used, it prevents the server from being out of memory (OOM). The key-value pairs have an expiration time set when they are created, and the expiration time is refreshed when the key-value pairs are updated. Unupdated key-value pairs are automatically deleted after the expiration time is reached.
[0039] As an example, the storage system is an external database implemented based on the Redis client library, employing a short-connection mode for data access. This means that a new Redis connection is created for each data processing request, and the connection is closed immediately after processing. The keys in the key-value pairs follow the format "[prefix][job]__[instance]", where [prefix] is a configuration item serving as a unique identifier for multi-tenants, and [job] and [instance] are parameters passed in after data processing. When storing key-value pairs, if a duplicate key exists, it is overwritten; otherwise, a new key is inserted directly. Each key has an expiration time set upon creation, and this expiration time is refreshed when the key-value pair is updated. Unupdated key-value pairs are automatically deleted after their expiration time to prevent invalid keys from occupying storage space.
[0040] In this embodiment, after constructing key-value pairs according to preset key-value pair storage rules and storing monitoring indicator data as values, the method further includes: Write key-value pairs to an external Redis database. The external Redis database uses a short connection method, and closes the connection immediately after each write. A uniform expiration time is set for the key during the write, and the expiration time is refreshed when the same key-value pair is written again in a subsequent write.
[0041] As an example, when writing key-value pairs, the external Redis database uses the format "[prefix][job]__[instance]", where [prefix] is a configuration item uniquely identified by the multi-tenant, [job] corresponds to the metric type parameter, and [instance] corresponds to the target instance parameter. Short-connection mode is used for data writing; a new Redis connection is created for each data write request, and the connection is closed immediately after the write is complete. An expiration time is set for the key during writing; subsequent writes to the same key refresh the expiration time, and the key is automatically deleted after expiration to avoid invalid keys occupying storage space.
[0042] In this embodiment, the indicator aggregation and retrieval configuration dynamic generation module generates a key list using the identity data of the tenant's personalized data, including: The metrics data transfer module uses a key scanning interface provided by an external Redis database. Using the tenant prefix as the scanning prefix, it reads all keys matching the prefix in batches, accumulating the keys returned from each scan to form a complete key list. This process relies on the mpsbot module (a dynamic generation module for metrics aggregation and retrieval configuration). The mpsbot module uses Redis's keyspace scanning service discovery mechanism, employing the Redis SCAN command to traverse all keys stored in Redis by the mps module that begin with the tenant prefix, ultimately obtaining a complete key list, which is then used as the objects to be pulled. Furthermore, the mpsbot module's configuration management is based on the Viper configuration management library, using YAML format configuration files to configure system parameters, including the mps server address, parsing task configuration (service discovery interval, configuration file output path, tenant tags), and log configuration.
[0043] As an example, the metrics data transfer module uses the key scanning interface provided by the external Redis database, with the tenant prefix identifier as the scanning prefix, to read all keys that match the prefix in batches, and accumulates the keys returned by each scan to form a complete key list; for example, if the tenant prefix identifier is "tenant_001", then it will read all keys that start with "tenant_001" in batches, and accumulate the keys obtained from each scan to form a complete key list corresponding to that tenant.
[0044] In this embodiment, the step of grouping and aggregating each key in the key list using the identity data of each key and other identifier data of the tenant's personalized identifier data to generate an indicator aggregation and retrieval configuration file includes: For each key in the key list, extract the tenant prefix identifier, metric type identifier, and instance identifier, and concatenate the tenant prefix identifier and metric type identifier to form a grouping key; Using the grouping key as an index, instance identifiers with the same grouping key are grouped into the same set to obtain the grouping result. During this grouping aggregation process, the mpsbot module logically groups the pulled objects according to the job name, with each group representing a metric type and containing data from multiple edge instances. It also aggregates and nests jobs according to the tenant prefix identifier, with each tenant containing multiple jobs, ultimately grouping and aggregating them into a "tenant[job[instance]]" structure. Simultaneously, when pushing metrics, the pushbot module pushes them to different mps module interface addresses according to the current metric type. After receiving the metric data, the mps module groups the data according to the job corresponding to the interface address, with the grouping rule being: metric type = job name. The mpsbot module then aggregates the scanned list according to the configuration items in its own configuration file (tenant, project specifying the tenant and project), generating the data with the above structure, used to generate the full configuration. Iterate through the grouping results, extract each grouping key and its corresponding set of instance identifiers in turn, use the grouping key as the file name, and write all the instance identifiers in the set in a fixed format to form a separate configuration file. The paths of all individual configuration files are sequentially written into the main configuration template to generate the aggregated metric pull configuration file. When generating the full configuration, the structured configuration data content is obtained through dynamic service discovery and group aggregation. The configuration file is generated according to the configuration template (taking VictoriaMetrics as an example of the monitoring system, the final configuration file aggregation folder is stored in / etc / prometheus / , containing the generated configuration files: / etc / Prometheus / vmagent.yaml, / etc / prometheus / file_sd_configs / {{prefix}}_{{job}}.yaml, etc., where prefix is the tenant prefix identifier and job is the metric type).
[0045] As an example, if the key list contains keys “tenant_001_cpu__host_01”, “tenant_001_cpu__host_02”, and “tenant_001_mem__host_03”, extract the tenant prefix identifier, metric type identifier, and instance identifier for each key in the key list. Concatenate the tenant prefix identifier “tenant_001” with the metric type identifier “cpu” to form the group key “tenant_001_cpu”. Using this group key as an index, find the instance identifiers “host_01”, “host_001”, “host_001”, “host_001”, “host_001”, and “host_001” with the same group key. 2” are grouped into the same set; the grouping results are traversed, and the grouping key “tenant_001_cpu” and its corresponding instance identifier set {“host_01”, “host_02”} are retrieved in sequence. “tenant_001_cpu” is used as the file name, and all instance identifiers in the set are written in a fixed format to form a separate configuration file; the same operation of generating a separate configuration file is performed on the grouping key “tenant_001_mem” and its corresponding instance identifier “host_03”; then the paths of all the separate configuration files are written sequentially into the main configuration template to generate the indicator aggregation and pull configuration file.
[0046] The monitoring data acquisition module retrieves the corresponding monitoring indicator data from the indicator data transfer module based on the indicator aggregation and retrieval configuration file, including: The monitoring system loads the aggregated metric pull configuration file, parses the main configuration template to obtain the paths of all individual configuration files, and reads the tenant prefix identifier, metric type identifier, and instance identifier recorded in each individual configuration file. For each instance identifier, an HTTP GET request carrying the tenant prefix identifier, indicator type identifier, and instance identifier is sent to the indicator data transfer module according to the access path recorded in the separate configuration file. After receiving a request, the indicator data relay module uses the tenant prefix identifier, indicator type identifier, and instance identifier carried in the request as a key to read the monitoring indicator data corresponding to the key from the external Redis database and return it to the monitoring system.
[0047] The monitoring system in this embodiment supports Prometheus and VictoriaMetrics as monitoring system servers, and either one can be selected for implementation; the generated configuration file is a general configuration file, which can be directly imported into Prometheus or VictoriaMetrics. By implementing the system according to the above steps and deployment architecture, the system can be realized.
[0048] Example 3 like Figure 5As shown in the figure, this embodiment of the invention proposes an indicator data acquisition device, which is applied to the indicator data acquisition method described in the above embodiment, including: an indicator data transfer module 100, an indicator aggregation and retrieval configuration dynamic generation module 200, and a monitoring data acquisition module 300.
[0049] The indicator data transfer module 100 receives monitoring indicator data pushed by the data acquisition server, constructs key-value pairs according to preset key-value pair storage rules, and stores the monitoring indicator data as values. The keys of the key-value pairs include tenant personalized identification data. The indicator aggregation and retrieval configuration dynamic generation module 200 generates a key list using the identity identification data of the tenant personalized data, and groups and aggregates each key in the key list using the identity identification data of each key and other identification data of the tenant personalized identification data to generate an indicator aggregation and retrieval configuration file. The monitoring data acquisition module 300 retrieves the corresponding monitoring indicator data from the indicator data transfer module according to the indicator aggregation and retrieval configuration file.
[0050] It should be noted that the foregoing explanation of the embodiment of the indicator data acquisition method also applies to the indicator data acquisition device of this embodiment, and will not be repeated here.
[0051] Example 4 Figure 6 This is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of this application. The electronic device 400 includes: a memory 401, a processor 402, and a computer program stored in the memory 401 and executable on the processor 402.
[0052] When the processor 402 executes the program, it implements the lightweight high-precision map preparation method provided in the above embodiments.
[0053] Furthermore, the electronic device 400 also includes a communication interface 403 for communication between the memory 401 and the processor 402.
[0054] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.
[0055] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0056] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.
[0057] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.
[0058] Example 5 This embodiment proposes a method for acquiring indicator data in a hybrid cloud environment. The method for acquiring indicator data in a hybrid cloud environment is based on the indicator data acquisition method described in the above embodiment to realize the transfer and aggregation of monitoring indicator data of edge nodes in different cloud regions.
[0059] In its implementation, the pushbot module, based on the Viper configuration management library, loads system parameters such as the MPS server address and the address of the monitoring metrics being pulled through a YAML configuration file. It then calls the `[GetHandler]` function to send HTTP GET requests to the service addresses of the pulled endpoints on different cloud region edge nodes to retrieve the monitoring metrics. After temporary storage, the metrics are pushed to the metric data relay module via the `[PushHandler]` function. Upon receiving the pushed monitoring metric data, the metric data relay module extracts the tenant prefix identifier, metric type identifier, and instance identifier. Combining this with the configured tenant prefix identifier, it generates a key in the format "[prefix][job]__[instance]". The deserialized monitoring metric data is used as the value and stored in a Redis database in short-connection mode. Each key has its expiration time set and refreshed upon creation and update to prevent invalid keys from occupying storage space. Next, the mpsbot module, based on Redis's keyspace scanning service discovery mechanism, uses Redis's SCAN command to traverse all keys in Redis that begin with a tenant prefix identifier, obtaining a complete list of keys. It then groups and aggregates the keys according to the tenant prefix identifier and job name, forming structured data in "tenant[job[instance]]" format. Based on this data and a configuration template, a general configuration file is generated. Finally, the monitoring system imports this general configuration file to achieve the relay, aggregation, and monitoring of monitoring metrics data from edge nodes in different cloud regions.
[0060] Understandably, by deploying metric data relay modules in various cloud regions of the hybrid cloud, edge nodes push monitoring metric data to the local relay module, and then store it using tenant prefixes, metric type identifiers, and instance identifiers as keys. The central side pulls the generated metric aggregation configuration file across cloud regions in read-only mode, without needing to expose write ports in each cloud region, eliminating cloud penetration and repeated traversal. Cross-cloud data isolation is achieved at the tenant level within the key, and the stateless design supports on-demand expansion of relay instances, thereby improving the security, scalability, and operational efficiency of monitoring metric aggregation in the hybrid cloud environment.
[0061] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0062] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0063] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0064] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.
[0065] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.
[0066] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. An index data acquisition method characterized by comprising: The method comprises the following steps: The index data conversion module receives the monitoring index data pushed by the data collection server, and constructs a key-value pair according to a preset key-value pair storage rule, and stores the monitoring index data as a value. The index aggregation pull configuration dynamic generation module generates a key list by using the identity data of the tenant personalized data, and performs grouping and aggregation on each key of the key list by using the identity data of each key and other identity data of the tenant personalized identity data, and generates an index aggregation pull configuration file. The monitoring data acquisition module pulls the corresponding monitoring index data from the index data conversion module according to the index aggregation pull configuration file.
2. The indicator data acquisition method of claim 1, wherein, The tenant personalized identity data at least includes a tenant prefix identifier, an index type identifier or an instance identifier. The index data conversion module receives the monitoring index data pushed by the edge service, extracts the tenant prefix identifier, the index type identifier and the instance identifier corresponding to the monitoring index data, and generates a key in the order of "tenant prefix identifier + index type identifier + instance identifier". The monitoring index data is stored as a value and the key constitutes a key-value pair, and is stored in a preset storage system.
3. The method of claim 2, wherein, The storage system is an external database supporting a key-value pair structure, the key-value pair is set with an expiration time when created, and the expiration time is refreshed when the key-value is updated, and the key-value that is not updated is automatically deleted after reaching the expiration time.
4. The method of claim 2, wherein, After constructing the key-value pair according to the preset key-value pair storage rule and storing the monitoring index data as a value, the method further comprises: The key-value pair is written into an external Redis database; the external Redis database adopts a short connection mode, the connection is closed immediately after each writing is completed, and a uniform expiration time is set for the key during writing, and the expiration time is refreshed again when the same key-value pair is written again subsequently.
5. The method of claim 2, wherein, The index aggregation pull configuration dynamic generation module generates a key list by using the identity data of the tenant personalized data, comprising: The index data conversion module reads all keys that meet the scan prefix by taking the tenant prefix identifier as the scan prefix through the key scanning interface provided by the external Redis database, and accumulates the keys returned by each scan to form a complete key list.
6. The method of claim 2, wherein, The index aggregation pull configuration dynamic generation module generates a key list by using the identity data of the tenant personalized data, comprising: The tenant prefix identifier and the index type identifier are spliced as a grouping key for each key in the key list. The instance identifiers with the same grouping key are grouped into the same set by taking the grouping key as an index, and a grouping result is obtained. Traverse the grouping result, take out each grouping key and its corresponding instance identifier set in turn, take the grouping key as a file name, and write all instance identifiers in the corresponding instance identifier set in a fixed format to form a separate configuration file; Write the paths of all the formed separate configuration files in sequence to the main configuration template to generate an index aggregation pull configuration file.
7. The indicator data acquisition method of claim 6, wherein, The monitoring data acquisition module pulls corresponding monitoring index data from the index data transfer module according to the index aggregation pull configuration file, including: The monitoring system loads the index aggregation pull configuration file, parses the main configuration template to obtain all separate configuration file paths, and reads the tenant prefix identifier, index type identifier and instance identifier recorded in each separate configuration file; For each instance identifier, an HTTP GET request carrying the tenant prefix identifier, index type identifier and instance identifier is sent to the index data transfer module according to the access path recorded in the separate configuration file; After receiving the request, the index data transfer module splices the tenant prefix identifier, index type identifier and instance identifier carried by the request as a key, reads the monitoring index data corresponding to the key from the external Redis database and returns it to the monitoring system.
8. An index data acquisition apparatus characterized by comprising: Comprise: The index data transfer module is used for receiving the monitoring index data pushed by the data acquisition server, and constructing a key-value pair according to a preset key-value storage rule, and storing the monitoring index data as a value; wherein the key of the key-value pair comprises tenant personalized identification data; The index aggregation pull configuration dynamic generation module is used for generating a key list using the identity data of the tenant personalized data, and performing grouping aggregation on each key of the key list using the identity data of each key and other identification data of the tenant personalized identification data, to generate an index aggregation pull configuration file; The monitoring data acquisition module is used for pulling corresponding monitoring index data from the index data transfer module according to the index aggregation pull configuration file.
9. An electronic device, comprising: The control device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the operations performed by the index data acquisition method of any one of claims 1 to 7. 10.A method for acquiring index data in a hybrid cloud environment, the method comprising: The index data acquisition method in the hybrid cloud environment is based on the index data acquisition method of any one of claims 1 to 7 to realize the transfer and aggregation of monitoring index data of different cloud region edge nodes.