An index time series data access method, system, device and storage medium

Abstract

The present disclosure relates to the field of data access, and discloses an index time series data access method, system, device and storage medium. The method comprises: sending an HTTP request by business software to push index time series data to an identity authentication component; performing identity authentication and standard verification on the HTTP request by the identity authentication component, and forwarding the HTTP request to a shunting component after verification; checking the load of each push gateway by the check notification component, and generating a shunting strategy according to the load and sending it to the shunting component; forwarding the HTTP request to the target push gateway based on the shunting strategy by the shunting component; and periodically pulling the index time series data of the target push gateway by the Prometheus server. The present disclosure combines identity authentication and load shunting, has high security, reliability and flexibility, and can meet various complex monitoring requirements.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data access, in particular to an index time series data access method, system, device and storage medium. BACKGROUND

[0002] In the civil aviation industry, index time series data is a key component of real-time data recording flight status, equipment monitoring, weather information, etc. With the rapid development of civil aviation business, the processing requirements for index time series data are becoming higher and higher, and efficient access, accurate storage and fast query of data need to be realized. The traditional data processing method often appears to be inadequate when dealing with a large amount of high-frequency index time series data. Time series database (TSDB) as a database system specially used for processing time series data has attracted much attention due to its efficient processing of time series data. Among them, Prometheus, as an open-source index time series database and monitoring and alarm tool, has been widely used in the civil aviation industry due to its simplicity, powerful functions, and active community.

[0003] Prometheus provides two main data access methods: pull and push. In the pull mode, the Prometheus server actively pulls data from the target service; while in the push mode, the target service actively pushes data to the Prometheus server. For civil aviation business, the push access method has significant advantages. First, the push access method can realize real-time data pushing. In civil aviation business, index time series data often needs to be delivered to the monitoring system in a timely manner for analysis and processing. Through the push method, the data source can actively push data to the Prometheus server without waiting for the pull request of Prometheus, thereby realizing real-time updating of data. Second, the push access method has higher flexibility. Different civil aviation business systems may have different data formats and transmission protocols. Through the push method, the data source can customize the data pushing method and content according to its own needs, thereby better adapting to different business scenarios. In addition, the push access method can also reduce the load of the Prometheus server. In the pull mode, Prometheus needs to periodically pull data from a large number of target services, which may cause the server load to be too high. After adopting the push method, the data source can actively push data to the Prometheus server, reducing the number of server pull requests and thus reducing the load of the server.

[0004] However, although the push access mode has many advantages, it still faces some challenges in practical application. How to ensure the accuracy and security of data pushing and how to guarantee the efficiency of data processing need to be considered when designing the index time series data access method. SUMMARY

[0005] Embodiments of the present disclosure provide an index time series data access method, system, device and storage medium to solve or alleviate one or more of the above technical problems in the prior art.

[0006] According to an aspect of the present disclosure, an index time series data access method is provided, comprising:

[0007] sending an http request by a business software to push index time series data to an identity authentication component;

[0008] performing identity authentication and standard verification on the http request by the identity authentication component, and forwarding the http request to a shunting component after the verification is passed;

[0009] checking the load condition of each push gateway by a checking notification component, and generating a shunting strategy according to the load condition and sending it to the shunting component;

[0010] forwarding the http request to a target push gateway based on the shunting strategy by the shunting component;

[0011] pulling the index time series data of the target push gateway by a Prometheus server at a regular time.

[0012] In a possible implementation, before checking the load condition of each push gateway by the checking notification component and generating a shunting strategy according to the load condition and sending it to the shunting component, it comprises:

[0013] The shunting component forwards the http request to a specified push gateway based on a preset shunting strategy.

[0014] In a possible implementation, checking the load condition of each push gateway by the checking notification component and generating a shunting strategy according to the load condition and sending it to the shunting component specifically comprises:

[0015] setting a saturation rate threshold;

[0016] checking the saturation rate of the push gateway at a regular time;

[0017] when the saturation rate of the push gateway is less than the saturation rate threshold, forwarding the http request to a specified push gateway based on a preset shunting strategy;

[0018] When the saturation rate of the push gateway is not less than the saturation rate threshold, the number of data is counted according to the business software name and sorted in ascending order.

[0019] The push gateway corresponding to the business software in the top n is switched.

[0020] In a possible implementation, the push gateway corresponding to the business software in the top n is switched, and the method specifically comprises the following steps:

[0021] It is judged whether there is a push gateway with a saturation rate less than a saturation rate threshold;

[0022] When the push gateway with the saturation rate less than the saturation rate threshold exists, the push gateway corresponding to the business software in the top n is switched to the push gateway with the saturation rate less than the saturation rate threshold.

[0023] Otherwise, a standby push gateway is deployed, and the push gateway corresponding to the business software in the top n is switched to the standby push gateway.

[0024] According to one aspect of the present disclosure, an index time series data access system is provided, comprising:

[0025] Business software, configured to push index time series data to an identity authentication component by sending an http request;

[0026] Identity authentication component, configured to perform identity authentication and standard verification on the http request, and forward the http request to a shunting component after the verification is passed;

[0027] Check notification component, configured to check the load of each push gateway at a time, and generate a shunting strategy according to the load and send the shunting strategy to the shunting component;

[0028] Shunting component, configured to forward the http request to a target push gateway based on the shunting strategy;

[0029] Prometheus server, configured to pull index time series data of the target push gateway at a time.

[0030] In a possible implementation, the shunting component is further configured to forward the http request to a specified push gateway based on a preset shunting strategy before checking the load of each push gateway at a time.

[0031] In a possible implementation, the check notification component specifically comprises:

[0032] Setting unit, configured to set a saturation rate threshold;

[0033] Check unit, configured to check the saturation rate of the push gateway at a time;

[0034] The forwarding unit is configured to forward the HTTP request to a specified push gateway based on a preset shunting strategy when a saturation rate of the push gateway is less than a saturation rate threshold.

[0035] The statistical unit is configured to count data items according to the business software names and perform ascending sequence sorting when the saturation rate of the push gateway is not less than the saturation rate threshold.

[0036] The switching unit is configured to switch the push gateway corresponding to the n business software with the highest ranking.

[0037] In a possible implementation, the switching unit specifically includes:

[0038] The judging module is configured to judge whether there is a push gateway with a saturation rate less than a saturation rate threshold.

[0039] The switching module is configured to switch the push gateway corresponding to the n business software with the highest ranking to the push gateway with the saturation rate less than the saturation rate threshold when there is the push gateway with the saturation rate less than the saturation rate threshold.

[0040] The deploying module is configured to deploy a backup push gateway when there is no push gateway with a saturation rate less than a saturation rate threshold.

[0041] The second switching module is configured to switch the push gateway corresponding to the n business software with the highest ranking to the backup push gateway.

[0042] According to an aspect of the present disclosure, an apparatus is provided, comprising:

[0043] a processor and a memory;

[0044] The memory is configured to store a computer program, and the processor is configured to invoke the computer program stored in the memory to execute the index time series data access method according to any one of the preceding aspects.

[0045] According to an aspect of the present disclosure, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the processor can execute the index time series data access method according to any one of the preceding aspects.

[0046] The present disclosure has the following beneficial effects: first, by introducing identity authentication, Pushgateway ensures the security and privacy of data. Only users who have been authenticated and obtained corresponding permissions can push access to monitoring data. This effectively prevents unauthorized users from maliciously accessing or tampering with data, ensuring the integrity and confidentiality of data; second, combined with load balancing technology, Pushgateway can automatically adjust resource allocation according to the load of the system. When the amount of monitoring data increases, the system can automatically expand resources to improve data processing capacity, which not only improves the availability and reliability of the system, but also significantly reduces operating costs; in summary, the scheme of Pushgateway combining identity authentication and load balancing has high security, reliability and flexibility, and can better meet various complex monitoring needs, providing strong and efficient protection for monitoring systems of various sizes.

[0047] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features and advantages of the application will become apparent from the description that follows. It should be understood that the foregoing general description and the following detailed description are merely examples and are not restrictive of the application. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure. It is readily apparent to one skilled in the art that the following figures in the description are merely some embodiments of the present disclosure, and other figures can be obtained from these figures without creative labor.

[0049] Figure 1 is a flowchart of an index time series data access method of the present exemplary embodiment;

[0050] Figure 2 is a system component diagram of the present exemplary embodiment;

[0051] Figure 3 is a block diagram of an index time series data access system of the present exemplary embodiment;

[0052] Figure 4 is a structural schematic diagram of a device of the present exemplary embodiment. DETAILED DESCRIPTION

[0053] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0054] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware units or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0055] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0056] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the application described herein can, for example, be implemented in an order other than that illustrated or described herein.

[0057] In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or submodules is not necessarily limited to those steps or submodules explicitly listed, but may include other steps or submodules not explicitly listed or inherent to such process, method, product or apparatus.

[0058] Explanation of terms:

[0059] Time series database: A specially designed and optimized database management system specifically for efficiently storing, indexing, querying, and analyzing time series data.

[0060] Prometheus: An open-source monitoring and alerting system based on time-series databases.

[0061] Prometheus server: A Prometheus server used for data collection and storage.

[0062] Pushgateway: An important component in the Prometheus ecosystem that allows temporary or batch jobs to actively push metric data to Prometheus, rather than being pulled by the Prometheus server.

[0063] Load balancing: A computer network technology whose core goal is to evenly distribute a large number of network requests or computing tasks to multiple servers or other resources to improve the overall response capability, availability, and reliability of the system.

[0064] Figure 1 A flowchart of a method for accessing metric time-series data according to an example embodiment of the present disclosure is shown in FIG. 1. Figure 1 An example embodiment of the present disclosure provides a method for accessing metric time-series data, which includes:

[0065] Sending an HTTP request to push metric time-series data to an identity authentication component through business software;

[0066] Performing identity authentication and standard verification on the HTTP request through the identity authentication component, and forwarding the HTTP request to a load balancing component after verification;

[0067] Checking the load of each push gateway through a notification component, and generating a load balancing strategy based on the load and sending it to the load balancing component;

[0068] Forwarding the HTTP request to a target push gateway based on the load balancing strategy through the load balancing component;

[0069] Pulling metric time-series data from the target push gateway through a Prometheus server at regular intervals.

[0070] The time-series data access method proposed in this embodiment realizes the unified access and efficient management of time-series data in different business fields by classifying and standardizing civil aviation business. This embodiment optimizes the data pushing mechanism and improves the data processing efficiency and accuracy.

[0071] The embodiment proposes a practical method for accessing the timing database of civil aviation business monitoring data. The method is based on identity authentication components, (real-time) inspection notification components, (switching) shunting components, Pushgateway services and Prometheus server. It can achieve identity authentication and load shunting according to civil aviation business software, and ensure the security of data and high availability of the system.

[0072] As shown in Figure 2 The embodiment is composed of identity authentication components, shunting components, inspection notification components, Pushgateway services and Prometheus server.

[0073] Identity authentication components: civil aviation business software accesses by sending http requests, and checks the account and password of the request to achieve identity authentication. After identity authentication, the parameters in the request string are checked to see if they meet the software classification standard to prevent malicious access or dirty data into the database.

[0074] Inspection notification components: regularly check the usage rate of each Pushgateway. When the load is found to be too large, a new shunting strategy is calculated and notified.

[0075] (Switching) shunting components: according to the data volume of each software request access, shunt to different Pushgateway according to software to ensure access performance and system high availability. Receive the notification of the "inspection notification component", and switch according to the new shunting strategy.

[0076] Pushgateway services and Prometheus server: each software will be sent to Pushgateway, which is stored by Prometheus server for monitoring and alarming of index data.

[0077] Specifically, the load of each push gateway is checked by the inspection notification component at regular intervals, and a shunting strategy is generated according to the load and sent to the shunting component. Before that, it includes:

[0078] The shunting component forwards the http request to the specified push gateway based on the preset shunting strategy.

[0079] The embodiment proposes a practical method for accessing the civil aviation business monitoring data time series database, and the specific principle is: for civil aviation software with access requirements for Prometheus time series database, SDK is provided for the software, accounts and access standards are divided according to the software, each software pushes the index time series data (a series of data sets formed by arranging index data in chronological order) to the identity authentication component by sending an HTTP request (Hypertext Transfer Protocol Request), and accepts the request after passing the check; the request is sent to the shunting component for shunting forwarding (according to the default strategy set in advance) to the specified Pushgateway; the Prometheus server is configured to pull at a fixed time, and the index data in the Pushgateway is pulled to the Prometheus for storage. In order to ensure the high availability of the service, the load condition of the Pushgateway is monitored and checked at a fixed time, a new shunting strategy is generated according to the load condition, and the shunting component is notified to switch.

[0080] The embodiment mainly has four main processing procedures: identity authentication procedure, check and shunting strategy generation procedure, load shunting procedure, standby Pushgateway service check procedure and data warehousing procedure. The following will be introduced in detail.

[0081] Identity authentication procedure:

[0082] 1. Each software sends an HTTP request;

[0083] 2. The request is intercepted, and identity authentication and standard check are performed;

[0084] 3. After verification, the request is forwarded to the shunting component.

[0085] Check and shunting strategy generation procedure:

[0086] 1. Check the load condition of each Pushgateway at a fixed time;

[0087] 2. Calculate the load condition, hereinafter referred to as the saturation rate: current data volume / 5w*100% (verified by testing, the maximum load of Pushgateway is set to 5w);

[0088] 3. Determine whether a new shunting strategy needs to be generated according to the load condition;

[0089] 3.1 If the saturation rate is less than 90%, no switching is needed, and the next check is waited;

[0090] 3.2 If the saturation rate is > 90%, count the number of data items according to the name of the business software, sort in ascending order, and obtain the first n software names (the data amount of the first n software is not less than 5%). The n software needs to switch the Pushgateway;

[0091] 3.3 Select the Pushgateway node with a saturation rate < 90%. If there is no selectable node, notify the deployment of a standby Pushgateway service;

[0092] 3.4 Generate a shunting strategy according to the above information, such as (software name 1, original Pushgateway, new Pushgateway);

[0093] 4. Notify the shunting component of the check results and the new shunting strategy (if any).

[0094] Load shunting process:

[0095] 1. Accept the request and receive the check results and new shunting strategy of the "check and shunting strategy generation process";

[0096] 2. According to the check results, if there is no change, forward the message according to the initialization strategy (initialization configuration);

[0097] If a change is required, read the new shunting strategy and perform the following operations (if the new shunting strategy is <software A, Pushgateway1, Pushgateway3>, which means that the request of software A is originally distributed to Pushgateway1, and now needs to be changed to Pushgateway3):

[0098] (1) Request interception, store the received software A request data in the cache;

[0099] (2) Take the number from Pushgateway1, store it in the cache, and merge it with the cache in the previous step;

[0100] (3) Wait for confirmation that the data in Pushgateway1 has been pulled by Prometheus (at least 2 / 3 of the pull has been completed);

[0101] (4) Clean up Pushgateway1;

[0102] (5) Perform switching according to the new shunting strategy.

[0103] Standby Pushgateway service deployment check process:

[0104] 1. Check if there is a notification to deploy a standby Pushgateway service (check and shunting strategy generation process);

[0105] 2. If yes, execute the Pushgateway deployment script.

[0106] Data warehousing process:

[0107] 1. Configure the pull configuration of the Pushgateway on each Prometheus Server, including the pull frequency, etc.

[0108] 2. Pull the index data of the Pushgateway into Prometheus at a fixed time.

[0109] Specifically, the load of each push gateway is checked by the notification component at a fixed time, and a shunting strategy is generated according to the load and sent to the shunting component, which specifically includes:

[0110] Set a saturation rate threshold;

[0111] Check the saturation rate of the push gateway at a fixed time;

[0112] When the saturation rate of the push gateway is less than the saturation rate threshold, the http request is shunted and forwarded to the specified push gateway based on the preset shunting strategy;

[0113] When the saturation rate of the push gateway is not less than the saturation rate threshold, the number of data is counted according to the business software name and sorted in ascending order;

[0114] Switch the push gateway corresponding to the top n business software.

[0115] Specifically, the push gateway corresponding to the top n business software is switched, which specifically includes:

[0116] Determine whether there is a push gateway with a saturation rate less than the saturation rate threshold;

[0117] When it exists, the push gateway corresponding to the top n business software is switched to the push gateway with a saturation rate less than the saturation rate threshold;

[0118] Otherwise, deploy a backup push gateway, and switch the push gateway corresponding to the top n business software to the backup push gateway.

[0119] When accessing time series data, in order to ensure the security of the data and the high availability of the access service, the present application mainly involves five specific implementation links of identity authentication, check and shunting strategy generation, shunting forwarding, backup Pushgateway service deployment, and data warehousing, which are as follows:

[0120] Identity authentication:

[0121] 1. Before pushing timing data, the business personnel stores the corresponding account password and data pushing URI standard of the business software in the identity information table. The identity information table data is shown in Table 1:

[0122] Table 1: Identity information table

[0123]

[0124] 2. The http request format of the timing data pushed by the business software is shown in Table 2:

[0125] Table 2: http request format of timing data pushed by business software

[0126]

[0127] When receiving the timing data request pushed by the business software, the following authentication process needs to be completed:

[0128] (1) According to the software name information contained in the headers in this request, obtain the data pushing URI standard from the identity information table and perform regular matching with the URI of this request. If it cannot be matched, directly return the 401 error code. If it can be matched, go to step (2).

[0129] (2) According to the software name information contained in the headers in this request, obtain the account and password information from the identity information table and compare with the account and password information of this request. If the comparison fails, directly return the 401 error code. If the comparison succeeds, go to the shunting forwarding link.

[0130] Check and shunting strategy generation:

[0131] The business personnel stores all the information of the current Pushgateway service in the Pushgateway service information table, sets the service address state of the Pushgateway service that has been deployed and is available to enabled, and sets other to disabled. The Pushgateway service information table data is shown in Table 3:

[0132] Table 3: Pushgateway service information table

[0133]

[0134] The following logic is executed every 1 minute:

[0135] (1) Obtain the addresses of all the Pushgateway services in the enabled state from the Pushgateway service information table.

[0136] (2) According to the address of all Pushgateway services obtained in step (1), the time series data pushed into the Pushgateway service is queried in turn, and the response data is parsed line by line. If the line exists job label, the software name in the job label value is obtained by regular matching, and the line is counted as the data number of the software name; if the line does not exist job label, skip the line statistics. The http request format for querying the time series data pushed to the Pushgateway service is as shown in Table 4:

[0137] Table 4 http request format table of time series data pushed to Pushgateway service:

[0138]

[0139]

[0140] (3) The saturation of T time series data pushed into each Pushgateway service is counted in units of Pushgateway service addresses, and the calculation rule is as follows

[0141] The saturation of Pushgateway service = sum (the data number of each software name pushed into the Pushgateway service) / 50000*100%.

[0142] (4) According to the saturation of each Pushgateway service calculated in step (3), sort from large to small, judge whether there is a Pushgateway service with saturation <=90%, if there is, record the address, saturation and other information of the Pushgateway service as the shunt Pushgateway service information for the shunt strategy generation in step (6); if not, skip this shunt strategy generation and send a notification to the standby Pushgateway service deployment link to deploy a new backup Pushgateway service.

[0143] (5) According to the saturation of each Pushgateway service calculated in step (3), it is judged whether the saturation of the Pushgateway service is > 90%. If the saturation of the Pushgateway service is > 90%, the saturation corresponding to each software name under the Pushgateway service is calculated (the saturation corresponding to each software name = the number of data corresponding to the software name / 50000 * 100), and then the saturation corresponding to each software name is accumulated in order from small to large. When the accumulated saturation is > 5%, the accumulation is terminated, and the software name, the corresponding saturation, the original push Pushgateway service address and other information participating in the accumulation are recorded as software shunting strategy information for the generation of shunting strategy in step (6); if the saturation of the Pushgateway service is <= 90%, the shunting strategy generation of the Pushgateway service is skipped.

[0144] (6) The shuntable Pushgateway service information recorded in step (4) and the software shunting strategy information recorded in (5) are obtained, and the saturation corresponding to the software name is added to the saturation corresponding to the Pushgateway service. If the saturation of the software name and the saturation corresponding to all Pushgateway services do not exist <= 90% condition, the software name is recorded as the shunted software information; if it exists, the addition logic is terminated, the Pushgateway service address added this time is added to the shunting strategy information as a new push Pushgateway service address for the shunting forwarding link, and the saturation corresponding to the Pushgateway service is updated to the sum of the saturation corresponding to the software name and the saturation corresponding to the Pushgateway service. Finally, if the shunted software is not empty, a notification is sent to the standby Pushgateway service deployment link to notify it to deploy a new backup Pushgateway service.

[0145] Shunting forwarding:

[0146] All received time series data requests pushed by business software are intercepted, and the following shunting forwarding logic is completed.

[0147] (1) According to the software name information contained in the headers in this request, it is judged whether the shunting strategy corresponding to the software name exists in the shunting strategy information generated in the previous link. If it does not exist, an address of a Pushgateway service in an enabled state is randomly obtained from the Pushgateway service information table, and step (3) is entered; if it exists, step (2) is entered.

[0148] (2) The business personnel record the addresses of all currently deployed Prometheus services as Prometheus service information, record the current time as the reference time, and query the Prometheus service information according to the software name in the shunting strategy information to obtain the situation of all Prometheus services scraping the Pushgateway service pushing data. The query request format is as shown in Table 5:

[0149] Table 5 Query request format table:

[0150]

[0151]

[0152] The response data is parsed. It is determined whether the value of the scrapeUrl field is consistent with the original Pushgateway service address in the shunting strategy information. If it is consistent, it is determined whether the value of the lastScrape field is later than the reference time. If it is later, the Prometheus service is counted as having scraped the software name pushing data. When all Prometheus services scraping the software name pushing data are queried, it is determined whether the number of Prometheus services that have scraped the software name pushing data / the number of all Prometheus services > 2 / 3. If it is not, step (2) is continued to be executed. If it is, the original Pushgateway service address of the software name in the shunting strategy information is used to query the pushed data. The query method refers to the http request format for querying the time series data that has been pushed to the Pushgateway service in the checking and shunting strategy generation link. The response data is filtered by the software name to only retain the pushed data of the software name. Then, the pushed data in the job field value and the instance field value are used to delete the pushed data on the Pushgateway service. The Pushgateway deletion data http request format is as shown in Table 6:

[0153] Table 6 Pushgateway deletion data http request format table:

[0154]

[0155]

[0156] Then, the original Pushgateway service address in the shunting strategy information is updated to the new Pushgateway service address. Finally, the time series data pushed by the business software in this request is grouped according to the software name to record the cache data information for deduplication, which is used for message forwarding in step (3).

[0157] (3) According to the software name information contained in the headers in this request, check whether the corresponding cache data exists in the cache data information. If not, directly forward the current request to the Pushgateway service address; if it exists, after deduplication of the current request and the cache data, forward to the original push Pushgateway service address corresponding to the software name in the shunt strategy information.

[0158] Backup Pushgateway service deployment:

[0159] When receiving the notification of deploying a new backup Pushgateway service, randomly obtain a Pushgateway service address with the status of not enabled from the Pushgateway service information table, and execute the Pushgateway deployment script. After successful deployment, update the Pushgateway service information table to update the status of the Pushgateway service address to enabled. If no Pushgateway service address with the status of not enabled is obtained from the Pushgateway service information table or the deployment fails, a notification of backup Pushgateway service deployment failure is sent out.

[0160] Data warehousing:

[0161] Configure the interface information of accessing the Pushgateway service to Prometheus.yaml to realize the storage of data grabbed by Prometheus at regular intervals. The configuration information is as follows:

[0162] {job_name: Pushgateway / / The monitoring task name is Pushgateway

[0163] scrape_interval: 10s / / The time interval for grabbing index data is 10s

[0164] honor_labels: true / / Configure basic authentication related information for username and password authentication for access

[0165] basic_auth:

[0166] username: ***** / / Specific username

[0167] password: ***** / / Specific password

[0168] static_configs:

[0169] targets: ***** / / Specific target to grab index, which can be an IP address, domain name, etc.}

[0170] Access Prometheus through the page to check whether the configured Pushgateway has taken effect.

[0171] Access Prometheus through the page and make a data query to check the data stored.

[0172] In summary, the target service of the embodiment actively pushes data to the Prometheus server through the push mode, realizes real-time pushing of data, reduces the number of pulling requests of the server, and thus reduces the load of the server; at the same time, through the pre-identity authentication, the safety and privacy of the data are ensured; through the redundant deployment and the load balancing switching of the nodes, the high availability of the service is ensured.

[0173] Figure 3 is a block diagram of a metric time series data access system according to an example embodiment, as shown in Figure 3 The example embodiment of the present disclosure provides a metric time series data access system, which comprises:

[0174] Business software, configured to push metric time series data to an identity authentication component by sending an http request;

[0175] An identity authentication component configured to perform identity authentication and standard verification on the http request, and forward the http request to a shunting component after the verification is passed;

[0176] An inspection notification component configured to check the load of each push gateway at regular intervals, and generate a shunting strategy according to the load and send it to the shunting component;

[0177] A shunting component configured to forward the http request to a target push gateway based on the shunting strategy;

[0178] A Prometheus server configured to pull metric time series data of the target push gateway at regular intervals.

[0179] Specifically, the shunting component is further configured to forward the http request to a specified push gateway based on a preset shunting strategy before checking the load of each push gateway at regular intervals.

[0180] Specifically, the inspection notification component comprises:

[0181] A setting unit configured to set a saturation rate threshold;

[0182] An inspection unit configured to check the saturation rate of the push gateway at regular intervals;

[0183] A forwarding unit, configured to forward the HTTP request to a designated push gateway based on a preset diversion strategy when the saturation rate of the push gateway is less than a saturation rate threshold;

[0184] The statistics unit is used to collect the number of statistical data items by business software name and sort them in ascending order when the saturation rate of the push gateway is not less than the saturation rate threshold;

[0185] The switching unit is used to switch the push gateway corresponding to the top n business software in the ranking.

[0186] Specifically, the switching unit includes:

[0187] A judgment module, used to judge whether there is a push gateway whose saturation rate is less than a saturation rate threshold;

[0188] A switching module is used to switch the push gateways corresponding to the top n business software to the push gateway with a saturation rate less than the saturation rate threshold when there is a push gateway with a saturation rate less than the saturation rate threshold;

[0189] A deployment module, configured to deploy a backup push gateway when there is no push gateway with a saturation rate less than a saturation rate threshold;

[0190] The second switching module is used to switch the push gateway corresponding to the top n business software to the backup push gateway.

[0191] Figure 4 FIG. 1 is a schematic diagram of the structure of a device of this exemplary embodiment. Figure 4 As shown, corresponding to the indicator time series data access method provided above, the present disclosure also provides a device. Since the embodiment of the device is similar to the above method embodiment, the description is relatively simple. Please refer to the description of the above method embodiment for relevant details. The device described below is only schematic. The device may include: a processor (processor) 1, a memory (memory) 2 and a communication bus (i.e., the above-mentioned device bus) and a search engine, wherein the processor 1 and the memory 2 communicate with each other through the communication bus and communicate with the outside through the communication interface. The processor 1 can call the logic instructions in the memory 2 to execute the indicator time series data access method.

[0192] In addition, the logic instructions in the memory 2 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present disclosure. The aforementioned storage medium includes: a storage chip, a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0193] In another aspect, the embodiments of the present disclosure also provide a processor readable storage medium, and the processor readable storage medium stores a computer program 3. When the computer program 3 is executed by the processor 1, the index time series data access method provided by the above-mentioned embodiments is implemented.

[0194] The processor readable storage medium can be any available medium or data storage device that the processor 1 can access, including but not limited to a magnetic storage (such as a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc.), an optical storage (such as a CD, a DVD, a BD, a HVD, etc.), and a semiconductor memory (such as a ROM, an EPROM, an EEPROM, a non-volatile memory (NAND FLASH), a solid state disk (SSD)), etc.

[0195] The above is only the preferred embodiment of the present disclosure, and the protection scope of the present disclosure is not limited to the above-mentioned embodiments. Any technical solution that belongs to the technical solutions of the present disclosure should be considered as the protection scope of the present disclosure. It should be noted that for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present disclosure should be considered as the protection scope of the present disclosure.

Claims

1. A method for accessing indicator time series data, characterized in that: include: Send HTTP requests through business software to push indicator time series data to the identity authentication component; The HTTP request is authenticated and verified by the identity authentication component, and after passing the verification, the HTTP request is forwarded to the diversion component; Regularly check the load of each push gateway through the inspection notification component, and generate a diversion strategy based on the load and send it to the diversion component; Forwarding the http request to the target push gateway based on the diversion strategy through the diversion component; Periodically pull the indicator time series data of the target push gateway through the Prometheus server; The notification component periodically checks the load of each push gateway and generates a diversion strategy based on the load and sends it to the diversion component. This includes: The diversion component diverts and forwards the http request to the designated push gateway based on the preset diversion strategy; The notification component periodically checks the load of each push gateway and generates a diversion strategy based on the load and sends it to the diversion component, specifically including: Set the saturation rate threshold; Regularly check the saturation rate of the push gateway; When the saturation rate of the push gateway is less than the saturation rate threshold, the HTTP request is forwarded to the designated push gateway based on the preset diversion strategy; When the saturation rate of the push gateway is not less than the saturation rate threshold, the statistical data items are sorted in ascending order according to the business software name; Switch the push gateway corresponding to the top n business software.

2. The method for accessing indicator time series data according to claim 1, characterized in that: Switch the push gateways corresponding to the top n business software, including: Determine whether there is a push gateway whose saturation rate is less than the saturation rate threshold; If there is one, the push gateway corresponding to the top n business software in the ranking is switched to the push gateway whose saturation rate is less than the saturation rate threshold; Otherwise, a backup push gateway is deployed, and the push gateways corresponding to the top n business software are switched to the backup push gateway.

3. An indicator time series data access system, characterized in that: include: Business software, used to push indicator time series data to the identity authentication component by sending HTTP requests; The identity authentication component is used to perform identity authentication and standard verification on the http request, and after passing the verification, forward the http request to the diversion component; The inspection notification component is used to regularly check the load of each push gateway and generate a diversion strategy based on the load and send it to the diversion component; A diversion component is used to forward the http request to the target push gateway based on the diversion strategy; Prometheus server, used to regularly pull the indicator time series data of the target push gateway; The diversion component is also used to divert and forward HTTP requests to the designated push gateway based on the preset diversion strategy before regularly checking the load status of each push gateway; The inspection notification components specifically include: A setting unit, used for setting a saturation rate threshold; The checking unit is used to periodically check the saturation rate of the push gateway; A forwarding unit, configured to forward the HTTP request to a designated push gateway based on a preset diversion strategy when the saturation rate of the push gateway is less than a saturation rate threshold; The statistics unit is used to collect the number of statistical data items by business software name and sort them in ascending order when the saturation rate of the push gateway is not less than the saturation rate threshold; The switching unit is used to switch the push gateway corresponding to the top n business software in the ranking.

4. The indicator time series data access system according to claim 3, characterized in that: The switching unit specifically includes: A judgment module, used to judge whether there is a push gateway whose saturation rate is less than a saturation rate threshold; A switching module is used to switch the push gateways corresponding to the top n business software to the push gateway with a saturation rate less than the saturation rate threshold when there is a push gateway with a saturation rate less than the saturation rate threshold; A deployment module, configured to deploy a backup push gateway when there is no push gateway with a saturation rate less than a saturation rate threshold; The second switching module is used to switch the push gateway corresponding to the top n business software to the backup push gateway.

5. A device, characterized in that include: processor and memory; The memory is used to store a computer program, and the processor calls the computer program stored in the memory to execute the indicator time series data access method according to any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is enabled to execute the indicator time series data access method according to any one of claims 1 to 2.

Images