Service flow data early warning method, system and device, storage medium and program product

Abstract

The invention discloses a service flow data early warning method, system and device, a medium and a program product. The method comprises the following steps: monitoring service flow data in real time and sending the service flow data to a distributed Redis cluster; performing real-time processing on the service flow data by using a distributed Redis cluster to obtain a multi-dimensional real-time interface call amount; setting early warning thresholds of a plurality of gears and corresponding early warning lines based on the business rules; and performing early warning judgment on the real-time interface calling amount by utilizing the early warning lines of the multiple gears, and performing early warning notification on the real-time interface calling amount exceeding the early warning lines. According to the method and the device, the flow statistics of the API calling amount, the multi-dimensional statistics of the API interface flow and the accurate early warning of the data flow abnormity are realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a business flow data early warning method, system, device, storage medium and program product. Background Art

[0002] With the large-scale use of APIs, the demand for monitoring and early warning of business data traffic is growing. In the field of API monitoring and early warning, existing technologies mainly focus on API call records, interface traffic statistics, and traffic early warning detection.

[0003] However, most traditional traffic warning systems are based on simple threshold settings and counter mechanisms, triggering warnings when traffic exceeds the preset threshold. Therefore, this method is powerless when faced with complex and changing traffic patterns, often leading to frequent false alarms and missed alarms, causing unnecessary losses to enterprises.

[0004] Therefore, a new business traffic data warning method is urgently needed to realize traffic statistics of API call volume, multi-dimensional statistics of API interface traffic, and accurate warning of data traffic anomalies, so as to provide enterprises with more real-time, accurate, flexible and cost-effective business data traffic warning solutions. Summary of the invention

[0005] The purpose of the present invention is to provide a business traffic data early warning method, system, device, storage medium and program product to achieve traffic statistics of API call volume, multi-dimensional statistics of API interface traffic and accurate early warning of data traffic anomalies.

[0006] To achieve the above object, the present invention provides the following technical solutions: In a first aspect, an embodiment of the present invention provides a service flow data early warning method, the method comprising: Monitor business traffic data in real time and send it to the distributed Redis cluster; Use distributed Redis clusters to process business traffic data in real time and obtain multi-dimensional real-time interface call volumes; Based on business rules, set warning thresholds for multiple gears and their corresponding warning lines; Use multiple warning lines to make early warning judgments on the real-time interface call volume, and issue early warning notifications for real-time interface call volume that exceeds the warning line.

[0007] In a second aspect, an embodiment of the present invention provides a service flow data early warning system, the system comprising: Data collection module, used to monitor business traffic data in real time and send it to the distributed Redis cluster; Distributed Redis cluster, used to calculate business traffic data and obtain multi-dimensional real-time interface call volume; Configuration module, used to set warning thresholds and corresponding warning lines for multiple gears based on business rules; The early warning notification module is used to use the early warning lines of multiple gears to make early warning judgments on the real-time interface call volume, and to issue early warning notifications for the real-time interface call volume that exceeds the early warning line.

[0008] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program or instructions to implement the aforementioned service flow data early warning method.

[0009] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the aforementioned service flow data warning method is implemented.

[0010] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned service flow data early warning method.

[0011] Technical effects and advantages of the present invention: The present invention introduces a distributed Redis cluster, adopts a Redis memory database and high concurrent writing characteristics, ensures real-time collection and storage of business data traffic, and provides real-time traffic monitoring and early warning; the introduction of a distributed Redis cluster can also support horizontal expansion and dynamic configuration, and adapt to distributed systems of different scales and needs; using the efficient data structure and query operation provided by the distributed Redis cluster, the system of the present invention can achieve million-level data statistics and analysis, accurately detect abnormal traffic, and process it in real time; each node in the distributed Redis cluster can work together to complete large-scale data processing and storage tasks, thereby improving the performance and availability of the early warning system; the present invention can monitor Redis business data traffic in real time, has low performance overhead and network latency, and can provide accurate traffic data and trigger early warnings; in addition, the system of the present invention supports flexible configuration of thresholds and rules, which can be adjusted according to actual business scenarios and needs, has adaptability and scalability, and can adapt to the needs of different business scenarios.

[0012] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 This is a flow chart of a service flow data early warning method according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a business flow data early warning system according to an embodiment of the present invention; Figure 3 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] To address the deficiencies of the prior art, the present invention discloses a service flow data early warning method. Figure 1 As shown, the following steps are included: Step S1: Monitor business traffic data in real time and send it to the distributed Redis cluster; Step S2: Use the distributed Redis cluster to process the business traffic data in real time to obtain multi-dimensional real-time interface call volume; Step S3: Based on business rules, set warning thresholds for multiple gears and their corresponding warning lines; Step S4: using the warning lines of multiple gears to make a warning judgment on the real-time interface call volume, and issuing a warning notification for the real-time interface call volume exceeding the warning line.

[0017] In some specific embodiments, step S1: monitoring service traffic data in real time and sending it to a distributed Redis cluster includes: Step S11: Setting a monitoring mechanism for multiple groups of business servers to perform real-time monitoring to monitor changes in the traffic log files corresponding to the business servers; when the traffic log files become larger, it indicates that the business servers have new business traffic data generated.

[0018] Step S12: When the business server generates business flow data, a corresponding designated rule file is generated based on the generated business flow data and sent to the data center server; specifically including: (1) When each group of business servers generates business traffic data, the application on the business server generates specified rule data according to the agreement based on the generated business traffic data, and stores it in the traffic log file corresponding to the business traffic data to form a specified rule file; The business traffic data includes API call volume and interface traffic data; The specified rule data represents the service flow data that conforms to the specified parameter format; for example: [ENGINERESFONSE]:, [DATE]:, [USER]:, [SERVICEVE]:, [SERVICEVERSION]:, [SATERWICE]:, [SUCCESS]: false; (2) Configure the storage directory of the data center server files, and then establish a connection between the data center server and the business server based on the configuration information (including IP, port, etc.) recorded in the configuration file of the data center server; The data center server log records whether the login is successful (i.e., whether the connection between the data center server and the business server is successful). If it fails, the log records detailed failure information to facilitate subsequent inspection by maintenance personnel.

[0019] (3) The business server transmits the specified rule file to a group of data center servers for aggregation through the SFPT transmission protocol, and the business rule protocol needs to be followed during the transmission process; Wherein, the business rule protocol includes transmission rules, traffic filtering rules and server blocking rules; The transmission rules include: using a specified format for transmission, that is, only using the enginAng.log.xxxx format for transmission; The filtering traffic rules include: filtering the business traffic data in the specified rule file to filter out useless data; The server blocking rule includes: setting a designated IP and switch of the business server, and blocking the business server when the business server generates large-scale malicious traffic.

[0020] Step S13: The data center server reads and filters the service flow data in the specified rule file, and sends the filtered service flow data to the distributed Redis cluster for aggregation and storage with the historical service flow data; specifically including: Configure the directory where the data center server stores the specified rule files and monitor the data server. When a specified rule file is transmitted, the data center server uses the BufferedReader character stream method to reduce I / O operations to read the business traffic data in the specified rule file (i.e., transmission rules); then the data center server filters the read business traffic data through a regular method to obtain data that conforms to the rule format, i.e., obtains the filtered business traffic data (i.e., filtering traffic rules).

[0021] The data center server uses PipeLine pipeline technology to send the filtered business flow data to the distributed Redis cluster. That is, the data center server stores the filtered business flow data in the PipeLine pipeline, and then distributes the filtered business flow data to the distributed Redis cluster for integration, which can achieve efficient transmission of a large amount of flow data.

[0022] Among them, the distributed Redis cluster is introduced as follows: The distributed Redis cluster includes multiple Redis nodes, and each node exchanges data and works together through a specific communication protocol; in the distributed Redis cluster, each node has its own memory and data storage space, and can independently process and store data.

[0023] At the same time, the distributed Redis cluster supports the master-slave replication mode, which can replicate data from one node to multiple slave nodes, improving the reliability and performance of the overall system; and the distributed Redis cluster has a rich data storage format, among which Redis hash is a mapping table of string type field and value, and hash is suitable for storing and filtering objects, similar to Map in Java.<String,Object> .

[0024] In some specific embodiments, step S2: using a distributed Redis cluster to process service traffic data in real time to obtain multi-dimensional real-time interface call volume, including: Set the statistical dimension configuration for business traffic data; wherein, the statistical dimension configuration includes: APIs of different interface functions, APIs of different users, different types of APIs (such as read interfaces and write interfaces), and charging tiers.

[0025] Based on the statistical dimension configuration, the effective set data structure of the distributed Redis cluster is used to perform traffic statistics calculation on the business traffic data to obtain multi-dimensional real-time interface call volume.

[0026] In some specific embodiments, step S3: setting warning thresholds and corresponding warning lines for multiple gears based on business rules includes: Step S31: Obtain the historical interface call volume within a predetermined period, and calculate the monthly average of the historical interface call volume as a benchmark reference value of the historical interface call volume; Exemplarily, the flight volumes in the first three months are obtained as 9999, 10011 and 9990 respectively, and then the monthly average of the flight volumes in the first three months is calculated as 10000, which is used as the flight volume benchmark reference value (ie, 10000).

[0027] Step S32: Setting business rules; Among them, setting business rules includes: customizing the upper and lower limits of warning thresholds, classifying according to the interface type (i.e. read interface and write interface) transmitted by the real-time interface call volume, and setting 4 gears.

[0028] Step S33: calculating standard parameters according to the benchmark reference value and the predetermined coefficient, calculating the warning threshold of each gear position based on the standard parameters and business rules, and setting the corresponding warning line; The predetermined coefficient can be adjusted according to actual conditions. Assuming that the predetermined coefficient is 1.0, the standard parameter = the reference value × predetermined coefficient = 10000 × 1.0 = 10000; The warning threshold of each gear = standard parameter × over-flow warning parameter; The warning line for each gear = n × (warning threshold of the gear / number of days in the month), where n represents the nth day of the month.

[0029] Therefore, the warning threshold for each gear position and the corresponding warning line are set as shown in Table 1: Table 1 Warning thresholds and warning lines

[0030] As shown in the above table, the inventor has set four call warning lines according to the real-time interface call volume, namely, two for each of the read and write interfaces.

[0031] In some specific embodiments, step S4: using the warning lines of multiple gears to make a warning judgment on the real-time interface call volume, and issuing a warning notification when the real-time interface call volume exceeds the warning line; includes: Step S41: define warning access rights, warning methods and warning contents for different user roles, including: Define different access rights and access levels for different user roles (such as administrators and ordinary users, etc.); for example, the administrator role can have higher permissions and can access and modify warning settings, while ordinary users can only receive warning notifications.

[0032] Different warning methods are set for different user roles. By providing multiple warning methods, the needs of different users can be met; for example, administrators can choose warning methods such as email, SMS or system messages, while ordinary users can choose to only receive warning methods such as system messages or mobile phone SMS.

[0033] Different warning contents are set for different user roles; for example, administrators can receive detailed warning information, while ordinary users can receive warning information directly related to them.

[0034] Step S42: Construct an early warning model, and use the early warning model to adopt threshold comparison and other methods to determine whether the real-time interface call volume exceeds the warning line. When the real-time interface call volume data is found to be abnormal or exceeds the warning line, an early warning is triggered, and the warning content is recorded in the log and then sent to the relevant user role for processing.

[0035] For example, if the traffic data on the 19th of a certain month is n19, then it is calculated that there are a total of n1+n2+...n actual call volumes from the beginning of the month to the 19th; so far this month, there have been 19 call days; then, based on the corresponding interface keyword, find out whether the corresponding configuration is read-write data, and sort from large to small according to the algorithm for early warning, to see whether the real-time interface call volume of the month exceeds the second-level early warning line of the corresponding interface; if it exceeds the second-level early warning line, an early warning notification is automatically triggered, and the call volume situation is sent to the business representative's mailbox; if it does not exceed the second-level early warning line, check whether it exceeds the first-level early warning line of the corresponding interface. If it exceeds the first-level early warning line, an early warning notification is still automatically triggered.

[0036] The embodiment of the present invention discloses a business flow data early warning system, such as Figure 2 As shown, the system comprises: Data collection module, used to monitor business traffic data in real time and send it to the distributed Redis cluster; Distributed Redis cluster, used to process business traffic data in real time and obtain multi-dimensional real-time interface call volume; Configuration module, used to set warning thresholds and corresponding warning lines for multiple gears based on business rules; The early warning notification module is used to use the early warning lines of multiple gears to make early warning judgments on the real-time interface call volume, and to issue early warning notifications for the real-time interface call volume that exceeds the early warning line.

[0037] In some specific embodiments, the data collection module includes a monitoring unit, a business server, and a data center server; The monitoring unit is used to monitor the service server in real time; The business server is used to generate business flow data, and generate corresponding specified rule files based on the generated business flow data and send them to the data center server; The data center server is used to read and filter the business traffic data in the specified rule file, and send the filtered business traffic data to the distributed Redis cluster for aggregation and storage with historical business traffic data; The business traffic data includes API call volume and interface traffic data.

[0038] Regarding the system in the above embodiment, the specific manner in which each unit module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0039] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, whose structure is as follows: Figure 3 As shown, it includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program or instructions to implement the aforementioned business flow data early warning method.

[0040] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the aforementioned service flow data warning method is implemented.

[0041] Based on the same inventive concept, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, which implement the aforementioned service flow data early warning method when executed by a processor.

[0042] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A business flow data early warning method, characterized in that: The method comprises: Monitor business traffic data in real time and send it to the distributed Redis cluster; Use distributed Redis clusters to process business traffic data in real time and obtain multi-dimensional real-time interface call volumes; Based on business rules, set warning thresholds for multiple gears and their corresponding warning lines; Use multiple warning lines to make early warning judgments on the real-time interface call volume, and issue early warning notifications for real-time interface call volume that exceeds the warning line.

2. A business flow data early warning method according to claim 1, characterized in that: Monitor business traffic data in real time and send it to the distributed Redis cluster, including: Monitor the business server in real time; when the business server generates business traffic data, generate the corresponding specified rule file based on the generated business traffic data and send it to the data center server; The data center server reads and filters the business traffic data in the specified rule file, and sends the filtered business traffic data to the distributed Redis cluster for aggregation and storage with historical business traffic data; The business traffic data includes API call volume and interface traffic data.

3. A business flow data early warning method according to claim 2, characterized in that: When the business server generates business traffic data, a corresponding specified rule file is generated based on the generated business traffic data and sent to the data center server, including: Generate specified rule data based on the generated business flow data, and store it in a flow log file corresponding to the business flow data to form a specified rule file; The business server transmits the specified rule file to the data center server via the SFPT transmission protocol; The specified rule data represents the business flow data that conforms to the specified parameter format.

4. A service flow data early warning method according to claim 2 or 3, characterized in that: The data center server reads and filters the business traffic data in the specified rule file, and sends the filtered business traffic data to the distributed Redis cluster, including: The data center server uses a BufferedReader character stream method that reduces I / O operations to read the business flow data in the specified rule file; The data center server filters the read business flow data by a regular method to obtain data that conforms to the regular format, that is, to obtain filtered business flow data; The data center server uses PipeLine pipeline technology to send the filtered business traffic data to the distributed Redis cluster.

5. A service flow data early warning method according to claim 3, characterized in that: The business server transmits the specified rule file to the data center server through the SFPT transmission protocol, including: The business server needs to comply with the business rule protocol during the transmission process of sending the specified rule file to the data center server; Wherein, the business rule protocol includes transmission rules, traffic filtering rules and server blocking rules; The transmission rules include: using a specified format for transmission; The filtering traffic rules include: filtering the business traffic data in the specified rule file to filter out useless data; The server blocking rule includes: setting a designated IP and switch of the business server, and blocking the business server when the business server generates large-scale malicious traffic.

6. A business flow data early warning method according to claim 1, characterized in that: Use the distributed Redis cluster to process business traffic data in real time and obtain multi-dimensional real-time interface call volume, including: Set the statistical dimension configuration for business traffic data; Based on the statistical dimension configuration, the effective set data structure of the distributed Redis cluster is used to perform traffic statistics calculation on the business traffic data to obtain multi-dimensional real-time interface call volume; The statistical dimension configuration includes: APIs of different interface functions, APIs of different users, APIs of different types, and charging tiers.

7. A business flow data early warning method according to claim 1, characterized in that: Based on business rules, set warning thresholds for multiple gears and their corresponding warning lines, including: Obtain the historical interface call volume within a predetermined period, and calculate the monthly average of the historical interface call volume as a benchmark reference value for the historical interface call volume; Setting business rules; wherein setting business rules includes: customizing the upper and lower limits of warning thresholds, classifying the interface types transmitted by the real-time interface call volume and setting multiple gears; Calculating standard parameters according to the benchmark reference value and the predetermined coefficient, calculating the warning threshold of each gear position based on the standard parameters and business rules and setting the corresponding warning line; Among them, the warning threshold of each gear = standard parameter × over-flow warning parameter; The warning line for each gear = n × (warning threshold of the gear / number of days in the month), where n represents the nth day of the month.

8. A business traffic data early warning system, characterized in that: The system comprises: Data collection module, used to monitor business traffic data in real time and send it to the distributed Redis cluster; Distributed Redis cluster, used to calculate business traffic data and obtain multi-dimensional real-time interface call volume; Configuration module, used to set warning thresholds and corresponding warning lines for multiple gears based on business rules; The early warning notification module is used to use the early warning lines of multiple gears to make early warning judgments on the real-time interface call volume, and to issue early warning notifications for the real-time interface call volume that exceeds the early warning line.

9. A business flow data early warning system according to claim 8, characterized in that: The data collection module includes a monitoring unit, a business server and a data center server; The monitoring unit is used to monitor the service server in real time; The business server is used to generate business flow data, and generate corresponding specified rule files based on the generated business flow data and send them to the data center server; The data center server is used to read and filter the business traffic data in the specified rule file, and send the filtered business traffic data to the distributed Redis cluster for aggregation and storage with historical business traffic data; The business traffic data includes API call volume and interface traffic data.

10. An electronic device, characterized in that: It includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program or instructions to implement a business traffic data early warning method as described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed by the processor, a service flow data early warning method as described in any one of claims 1-7 is implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by the processor, a service flow data early warning method as described in any one of claims 1-7 is implemented.

Images