Business traffic data early warning method, system, device, storage medium and program product

Abstract

The application discloses a service traffic data early warning method, system, device, medium and program product, and the method comprises the steps of: monitoring service traffic data in real time and sending the service traffic data to a distributed Redis cluster; processing the service traffic data in real time by using the distributed Redis cluster to obtain real-time interface call quantity in multiple dimensions; setting early warning threshold values and corresponding early warning lines of multiple gears based on service rules; and performing early warning judgment on the real-time interface call quantity by using the early warning lines of the multiple gears, and performing early warning notification on the real-time interface call quantity exceeding the early warning lines. The application realizes traffic statistics of API call quantity, multidimensional statistics of API interface traffic and accurate early warning of data traffic anomalies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a business traffic data early warning method, system, device, storage medium and program product. BACKGROUND

[0002] With the large-scale use of API, the monitoring and early warning of business data traffic are increasingly in demand. In the field of API monitoring and early warning, the existing technology mainly focuses on API call records, interface traffic statistics and traffic early warning detection.

[0003] However, the traditional traffic early warning system is mostly based on simple threshold setting and counter mechanism, and triggers a warning when the traffic exceeds the preset threshold. Therefore, this method is not up to the task when facing complex and variable traffic patterns, often leading to frequent false positives and false negatives, causing unnecessary losses to enterprises.

[0004] Therefore, there is an urgent need for a new business traffic data early warning method to realize traffic statistics of API call volume, multidimensional statistics of API interface traffic and accurate early warning of data traffic anomalies, and to provide enterprises with a more real-time, accurate, flexible and cost-effective business data traffic early warning solution. SUMMARY

[0005] The present application aims to provide a business traffic data early warning method, system, device, storage medium and program product, which realizes traffic statistics of API call volume, multidimensional statistics of API interface traffic and accurate early warning of data traffic anomalies.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a business traffic data early warning method, which comprises:

[0008] Real-time monitoring of business traffic data and sending to a distributed Redis cluster;

[0009] Real-time processing of business traffic data using a distributed Redis cluster to obtain multidimensional real-time interface call volume;

[0010] Based on business rules, setting multiple gear pre-warning thresholds and their corresponding pre-warning lines;

[0011] Using multiple gear pre-warning lines to make early warning judgments on real-time interface call volume and making early warning notifications on real-time interface call volume exceeding the pre-warning lines.

[0012] In a second aspect, the present application provides a business traffic data early warning system, which comprises:

[0013] A data collection module is configured to listen to service traffic data in real time and send the data to a distributed Redis cluster.

[0014] The distributed Redis cluster is configured to perform traffic calculation on the service traffic data and obtain multi-dimensional real-time interface call volume.

[0015] A configuration module is configured to set warning thresholds and corresponding warning lines of multiple gears based on service rules.

[0016] A warning notification module is configured to perform warning judgment on the real-time interface call volume using the warning lines of multiple gears and perform warning notification on the real-time interface call volume exceeding the warning lines.

[0017] In a third aspect, an electronic device is also provided, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program or instructions to implement the aforementioned service traffic data warning method.

[0018] In a fourth aspect, a computer storage medium is also provided, which stores a computer program or instructions, and the computer program or instructions are executed by a processor to implement the aforementioned service traffic data warning method.

[0019] In a fifth aspect, a computer program product is also provided, which includes a computer program or instructions, and the computer program or instructions are executed by a processor to implement the aforementioned service traffic data warning method.

[0020] The technical effects and advantages of the present application are as follows: The present application introduces a distributed Redis cluster, uses Redis in-memory database and high-concurrency write characteristics to ensure real-time collection and storage of service data traffic, provides real-time traffic monitoring and warning, introduces a distributed Redis cluster, which can also support horizontal expansion and dynamic configuration, and is suitable for distributed systems of different scales and needs, uses efficient data structures and query operations provided by the distributed Redis cluster, and the present application system can realize million-level data statistics and analysis, accurately detect abnormal traffic, and perform real-time processing, each node in the distributed Redis cluster can work cooperatively to complete large-scale data processing and storage tasks, and the performance and availability of the warning system are improved, the present application can monitor Redis service data traffic in real time, has low performance overhead and network delay, can provide accurate traffic data and trigger warning, and in addition, the present application system supports flexible configuration of thresholds and rules, can be adjusted according to actual business scenarios and needs, has adaptability and scalability, and can adapt to the needs of different business scenarios.

[0021] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0023] Figure 1 A flow chart of a service traffic data early warning method according to an embodiment of the present application;

[0024] Figure 2 A structural schematic diagram of a service traffic data early warning system according to an embodiment of the present application;

[0025] Figure 3 A structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0027] To solve the problems in the prior art, an embodiment of the present application discloses a service traffic data early warning method, as shown in Figure 1 The method comprises the following steps:

[0028] Step S1: Real-time monitoring of service traffic data and sending to a distributed Redis cluster;

[0029] Step S2: Real-time processing of the service traffic data by using the distributed Redis cluster to obtain multi-dimensional real-time interface call quantity;

[0030] Step S3: Based on service rules, setting early warning thresholds and corresponding early warning lines of multiple gears;

[0031] Step S4: Using the early warning lines of multiple gears to make early warning judgments on the real-time interface call quantity, and making early warning notifications on the real-time interface call quantity exceeding the early warning lines.

[0032] In some specific embodiments, the step S1 of monitoring the service traffic data in real time and sending to the distributed Redis cluster comprises:

[0033] The step S11 comprises monitoring the change of the traffic log file corresponding to the service server in real time by setting a monitoring mechanism for the plurality of groups of service servers; when the traffic log file is changed, it indicates that the service server generates new service traffic data.

[0034] The step S12 comprises generating a corresponding specified rule file based on the generated service traffic data and sending it to the data center server when the service server generates service traffic data; specifically comprising:

[0035] (1) When each group of service servers generates service traffic data, the application on the service server generates specified rule data according to the generated service traffic data according to the agreement, and stores it in the traffic log file corresponding to the service traffic data to form a specified rule file;

[0036] The service traffic data includes API call volume and interface traffic data.

[0037] The specified rule data indicates the service traffic data conforming to the specified parameter format; for example: [ENGINERESFONSE]:,[DATE]:,[USER]:,[SERVICEVE]:,[SERVICEVERSION]:,[SATERWICE]:,[SUCCESS]:false;

[0038] (2) Configure the storage directory of the data center server file, and then establish a connection with the service server in the data center server according to the configuration information (including IP, port, etc.) recorded in the configuration file of the data center server;

[0039] The log of the data center server records whether the login is successful (i.e. whether the connection between the data center server and the service server is successful), and if it fails, the log records detailed failure information, which is convenient for subsequent inspection by maintenance personnel.

[0040] (3) The service server transmits the specified rule file to a group of data center servers for aggregation through the SFPT transmission protocol, and needs to comply with the business rule protocol in the transmission process;

[0041] The business rule protocol includes transmission rules, filtering traffic rules and server blocking rules.

[0042] The transmission rules include: transmission in a specified format, i.e. transmission in only enginAng.log.xxxx format;

[0043] The filtering traffic rule comprises: filtering the service traffic data in the specified rule file, and screening out useless data.

[0044] The server blocking rule comprises: setting the specified IP and switch of the service server, and blocking the service server when the service server generates large-scale malicious traffic.

[0045] Step S13: The data center server reads and filters the service traffic data in the specified rule file, and sends the filtered service traffic data to the distributed Redis cluster for aggregated storage with the historical service traffic data; specifically comprising:

[0046] The data center server stores the directory of the specified rule file, and listens to the data server. When the specified rule file is transmitted, the data center server reads the service traffic data in the specified rule file by using the BufferedReader character stream mode with reduced I / O operation (i.e. transmission rule); then the data center server filters the read service traffic data by using the regular method to obtain the data conforming to the rule format, i.e. the filtered service traffic data (i.e. filtering traffic rule).

[0047] The data center server sends the filtered service traffic data to the distributed Redis cluster by using the PipeLine pipeline technology. That is, the data center server stores the filtered service traffic data in the PipeLine pipeline, and then distributes the filtered service traffic data to the distributed Redis cluster for integration, which can realize efficient transmission of a large amount of traffic data.

[0048] Among them, the distributed Redis cluster is introduced as follows:

[0049] The distributed Redis cluster comprises a plurality of Redis nodes, and each node exchanges data and cooperates with each other through a specific communication protocol; each node in the distributed Redis cluster has its own memory and data storage space, and can independently process and store data.

[0050] At the same time, the distributed Redis cluster supports the master-slave replication mode, can copy data from one node to multiple slave nodes, and improves the reliability and performance of the overall system; and the distributed Redis cluster has rich data storage formats, wherein the Redis hash is a mapping table of string type field and value, and the hash is suitable for storing and filtering objects, similar to Map<String, Object> in Java.

[0051] In some specific embodiments, step S2: real-time processing of business traffic data using a distributed Redis cluster to obtain multi-dimensional real-time interface call volume, including:

[0052] Setting a statistical dimension configuration for business traffic data; wherein the statistical dimension configuration includes: APIs of different interface functions, APIs of different users, APIs of different types (such as read interfaces and write interfaces), and charging steps.

[0053] Based on the statistical dimension configuration, the effective set data structure of the distributed Redis cluster is used to perform traffic statistical calculation on the business traffic data to obtain multi-dimensional real-time interface call volume.

[0054] In some specific embodiments, step S3: based on business rules, setting the pre-warning threshold and its corresponding pre-warning line of multiple gears, including:

[0055] Step S31: obtaining historical interface call volume within a predetermined period, and calculating the monthly average of the historical interface call volume as a benchmark reference value of the historical interface call volume;

[0056] For example, the flight volume of the previous three months is 9999, 10011 and 9990 respectively, and then the monthly average of the flight volume of the previous three months is 10000, which is taken as the benchmark reference value of the flight volume (i.e. 10000).

[0057] Step S32: setting business rules;

[0058] Wherein, setting business rules includes: customizing upper and lower limits of pre-warning threshold, classifying and setting 4 gears according to the interface type (i.e. read interface and write interface) of real-time interface call volume transmission.

[0059] Step S33: calculating a standard parameter according to the benchmark reference value and a predetermined coefficient, and calculating the pre-warning threshold of each gear and setting the corresponding pre-warning line based on the standard parameter and the business rules;

[0060] Wherein, the predetermined coefficient can be adjusted according to actual conditions, assuming that the predetermined coefficient is 1.0, the standard parameter = the benchmark reference value x the predetermined coefficient = 10000 x 1.0 = 10000;

[0061] The pre-warning threshold of each gear = the standard parameter x the over-flow pre-warning parameter;

[0062] The pre-warning line of each gear = n x (the pre-warning threshold of this gear / the number of days in the month), where n represents the nth day of the month.

[0063] Therefore, the pre-warning threshold of each gear and the corresponding pre-warning line are shown in Table 1:

[0064] Table 1 Warning threshold and its warning line table

[0065]

[0066] According to the table, the inventor sets four warning lines for the real-time interface call quantity, i.e., two for reading and two for writing.

[0067] In some specific embodiments, step S4: using the warning lines of multiple gears to make a warning judgment on the real-time interface call quantity, and making a warning notification on the real-time interface call quantity exceeding the warning line; includes:

[0068] Step S41: defining the warning access authority, warning method and warning content for different user roles, including:

[0069] Different access authorities and access levels are defined for different user roles (e.g., administrator and ordinary user, etc.); for example, the administrator role can have higher authority and can access and modify the warning settings, while the ordinary user can only receive the warning notification.

[0070] Different warning methods are set for different user roles, and by providing multiple warning methods, the needs of different users can be met; for example, the administrator can choose the warning methods of email, SMS or system message, while the ordinary user can choose to receive only the system message or the warning method of mobile phone SMS.

[0071] Different warning contents are set for different user roles; for example, the administrator can receive detailed warning information, while the ordinary user can receive warning information directly related to them.

[0072] Step S42: constructing a warning model, using the warning model to determine whether the real-time interface call quantity exceeds the warning line by threshold comparison and other methods, triggering a warning when the real-time interface call quantity data is found to be abnormal or exceeds the warning line, and recording the warning content in the log and then sending the warning content to the relevant user role for processing.

[0073] For example, the traffic data on the 19th of a certain month is n19, then the actual call quantity from the beginning of the month to the 19th is calculated as n1+n2+...n; the number of call days in the month so far is 19 days; then according to the corresponding interface keyword, it is found out whether the corresponding configuration is read or write data, and according to the algorithm, the warning is sorted from large to small, and it is checked whether the real-time interface call quantity of the month exceeds the second-gear warning line of the corresponding interface; if it exceeds the second-gear warning line, the warning notification is automatically triggered, and the call quantity is sent to the business representative mailbox; if it does not exceed the second-gear warning line, it is checked whether it exceeds the first-gear warning line of the corresponding interface, and if it exceeds the first-gear warning line, the warning notification is still automatically triggered.

[0074] The embodiment of the present application discloses a service traffic data early warning system, as shown in the figure, the system comprises: Figure 2

[0075] A data collection module is configured to listen to service traffic data in real time and send the data to a distributed Redis cluster.

[0076] The distributed Redis cluster is configured to process the service traffic data in real time and obtain multi-dimensional real-time interface call volume.

[0077] A configuration module is configured to set the early warning thresholds and corresponding early warning lines of multiple gears based on service rules.

[0078] An early warning notification module is configured to use the early warning lines of multiple gears to make early warning judgments on the real-time interface call volume and make early warning notifications on the real-time interface call volume exceeding the early warning lines.

[0079] In some specific embodiments, the data collection module comprises a listening unit, a service server and a data center server.

[0080] The listening unit is configured to listen to the service server in real time.

[0081] The service server is configured to generate service traffic data and generate corresponding specified rule files based on the generated service traffic data and send the files to the data center server.

[0082] The data center server is configured to read and filter the service traffic data in the specified rule files and send the filtered service traffic data to the distributed Redis cluster for aggregated storage with historical service traffic data.

[0083] The service traffic data comprises API call volume and interface traffic data.

[0084] As to the system in the above embodiment, the specific manner in which each unit module performs operations has been described in detail in the embodiment related to the method, and will not be described in detail here.

[0085] Based on the same inventive concept, the embodiment of the present application also provides an electronic device, as shown in the figure, comprising a memory, a processor and a computer program stored in the memory, and the processor executes the computer program or instructions to realize the service traffic data early warning method. Figure 3

[0086] ​​Based on the same inventive concept, the embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores computer programs or instructions, and the computer programs or instructions are executed by a processor to realize the business flow data early warning method.

[0087] Based on the same inventive concept, the embodiment of the present application further provides a computer program product, comprising computer programs or instructions, and the computer programs or instructions are executed by a processor to realize the business flow data early warning method.

[0088] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it still can be modified to the technical scheme recorded in the foregoing embodiments, or to the equivalent replacement of some technical features, within the spirit and principles of the present application, any modification, equivalent replacement, improvement, etc., should be included in the protection scope of the present application.

Claims

1. A service traffic data pre-warning method, characterized in that, The method comprises: Real-time monitoring of business traffic data and sending to a distributed Redis cluster; Real-time processing of business traffic data using a distributed Redis cluster to obtain multi-dimensional real-time interface call volume; Based on business rules, set the warning threshold and its corresponding warning line of multiple gears; Use multiple gear warning lines to make early warning judgments on real-time interface call volume and make early warning notifications on real-time interface call volume that exceeds the warning line; Wherein, Real-time monitoring of business traffic data and sending to a distributed Redis cluster comprises: Real-time monitoring of business servers; when the business server generates business traffic data, generate the corresponding specified rule file based on the generated business traffic data and send it to the data center server; the data center server reads and filters the business traffic data in the specified rule file, and sends the filtered business traffic data to the distributed Redis cluster for aggregated storage with historical business traffic data; wherein the business traffic data includes API call volume and interface traffic data; Real-time processing of business traffic data using a distributed Redis cluster to obtain multi-dimensional real-time interface call volume comprises: Set the statistical dimension configuration for business traffic data; based on the statistical dimension configuration, use the effective set data structure of the distributed Redis cluster to perform traffic statistical calculation on the business traffic data to obtain multi-dimensional real-time interface call volume; wherein the statistical dimension configuration includes: different interface functions API, different user APIs, different types of APIs, and charging ladder.

2. The method of claim 1, wherein, When the business server generates business traffic data, generate the corresponding specified rule file based on the generated business traffic data and send it to the data center server, comprising: Generate specified rule data based on the generated business traffic data, and store it in the traffic log file corresponding to the business traffic data to form a specified rule file; The business server transmits the specified rule file to the data center server through the SFPT transmission protocol; Wherein, the specified rule data indicates business traffic data conforming to the specified parameter format.

3. The service traffic data pre-warning method according to claim 1 or 2, characterized in that, The data center server reads and filters the business traffic data in the specified rule file, and sends the filtered business traffic data to the distributed Redis cluster, comprising: The data center server reads the business traffic data in the specified rule file using the BufferedReader character stream method to reduce I / O operations; The data center server filters the read business traffic data through a regular method to obtain data conforming to the rule format, i.e. filtered business traffic data; The data center server sends the filtered business traffic data to the distributed Redis cluster using the PipeLine pipeline technology.

4. The method of claim 2, wherein, The business server transmits the specified rule file to the data center server through the SFPT transmission protocol, comprising: The business server needs to comply with the business rule protocol during the transmission process of sending the specified rule file to the data center server; The service rule protocol comprises a transmission rule, a filtering traffic rule and a server blocking rule. The transmission rule comprises transmission in a specified format. The filtering traffic rule comprises filtering traffic data in a specified rule file to filter out useless data. The server blocking rule comprises setting a specified IP and switch of the service server, and blocking the service server when the service server generates large-scale malicious traffic.

5. The method of claim 1, wherein, Based on the service rule, a plurality of warning threshold values and corresponding warning lines of the plurality of gears are set, comprising: obtaining historical interface call quantity within a predetermined period, and calculating a monthly average of the historical interface call quantity as a benchmark reference value of the historical interface call quantity; setting a service rule; wherein setting the service rule comprises customizing upper and lower limits of a warning threshold value, classifying and setting a plurality of gears according to an interface type of real-time interface call quantity transmission; calculating a standard parameter according to the benchmark reference value and a predetermined coefficient, and calculating a warning threshold value of each gear based on the standard parameter and the service rule and setting a corresponding warning line; wherein the warning threshold value of each gear = standard parameter x traffic overload warning parameter; the warning line of each gear = n x (warning threshold value of the gear / number of days in the month), wherein n represents the nth day of the month.

6. A service traffic data early warning system characterized by, The system comprises: a data collection module for real-time monitoring of traffic data and sending to a distributed Redis cluster; a distributed Redis cluster for traffic calculation of traffic data to obtain multi-dimensional real-time interface call quantity; a configuration module for setting a plurality of warning threshold values and corresponding warning lines of the plurality of gears based on a service rule; a warning notification module for warning judgment of real-time interface call quantity using a plurality of warning lines of the plurality of gears, and warning notification of real-time interface call quantity exceeding the warning line; wherein the data collection module comprises a listening unit, a service server and a data center server; the listening unit is used for real-time monitoring of the service server; the service server is used for generating traffic data, and generating a corresponding specified rule file based on the generated traffic data and sending it to the data center server; wherein the traffic data comprises API call quantity and interface traffic data; the data center server is used for data reading and data filtering of the traffic data in the specified rule file, and sending the filtered traffic data to the distributed Redis cluster for aggregated storage with historical traffic data; real-time processing of traffic data using a distributed Redis cluster to obtain multi-dimensional real-time interface call quantity, comprising: setting a statistical dimension configuration for traffic data; based on the statistical dimension configuration, using the effective set data structure of the distributed Redis cluster to perform traffic statistical calculation on the traffic data to obtain multi-dimensional real-time interface call quantity; wherein the statistical dimension configuration comprises APIs of different interface functions, APIs of different users, APIs of different types, and charging steps.

7. An electronic device, comprising: The computer readable storage medium stores a computer program or instruction, and the computer program or instruction is executed by the processor to realize the business flow data early warning method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program or instruction, and the computer program or instruction is executed by the processor to realize the business flow data early warning method in any one of claims 1-5.

9. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instruction is executed by the processor to realize the business flow data early warning method in any one of claims 1-5.

Images