Traffic processing method, system and device based on policy intermediate station, and storage medium

Abstract

The invention provides a flow processing method, system and device based on a strategy platform, and a storage medium, and the method comprises the steps: splitting a Redis cluster, and carrying out the resource isolation of a high-query-rate data source through employing an independent Redis cluster; an Flink asynchronous function is adopted, the concurrent request capacity of an Flink framework for a Redis client side is increased, and an asynchronous timeout mechanism is added; performing mixed matching on a long message based on a Google regularization engine re2j and a Jvm regularization engine; and adjusting the memory model, and dispersing the sub-tasks under the corresponding tasks in different clusters for execution. According to the invention, various characteristics of Kafka and Flink can be utilized to realize millisecond-level real-time strategy analysis of million-level qps data volume, and backlog-free, delay-free and stable operation of data processing.

Description

Background Art

[0002] The strategy center plays a very important role in information security-basic security; its main function is to conduct real-time threat analysis on various types of information security data / logs / traffic, and input security incident alerts to downstream business parties; it is based on the Flink big data stream processing framework, and has currently connected to more than 80 data sources such as hids, nids, waf, netflow, slb traffic, https traffic, etc., with an overall QPS of nearly 3 million; currently it covers more than 500 detection rules (each rule contains an average of more than 5 operators); the security detection scope covers all attack stages of the ATT&CK model.

[0003] Currently, the strategy center processes nearly 3 million data items per second, including large message data (megabytes). Hundreds of millions of matching calculations are required per second, including many complex regular calculations, window count statistics and other time-consuming calculations. This poses great challenges to processing timeliness and stability.

[0004] If processing backlogs, delays, downtime, etc. occur, malicious intrusions may not be caught quickly and in real time, which may put company assets at risk of intrusion; therefore, high requirements are placed on the timeliness and stability of processing.

[0005] Therefore, the present invention provides a traffic processing method, system, device and storage medium based on a policy middle station. Summary of the invention

[0006] In response to the problems in the prior art, the purpose of the present invention is to provide a traffic processing method, system, device and storage medium based on a policy middle platform, which overcomes the difficulties of the prior art, can utilize the various characteristics of Kafka and Flink, and realize millisecond-level real-time policy analysis of millions of qps data volumes, with no backlog, no delay and stable operation of data processing.

[0007] An embodiment of the present invention provides a traffic processing method based on a policy center, comprising the following steps:

[0008] S110, split the Redis cluster and use a separate Redis cluster for resource isolation of high query rate data sources;

[0009] S120, using Flink asynchronous functions, increasing the concurrent request capability of the Flink framework for Redis clients, and adding an asynchronous timeout mechanism;

[0010] S130, performing mixed matching of long messages based on Google regular engine re2j and Jvm regular engine; and

[0011] S140: Adjust the memory model, and distribute the subtasks under the corresponding task to different clusters for execution.

[0012] Preferably, the step S110 includes:

[0013] S111, splitting the Redis cluster, and grouping the data sources according to the query rate based on a preset threshold;

[0014] S112: Use a separate Redis cluster for resource isolation of a high query rate data source whose query rate is greater than a preset threshold, and use the remaining Redis clusters for a low query rate data source whose query rate is less than the preset threshold.

[0015] Preferably, the step S120 includes:

[0016] S121. Use Flink asynchronous functions to increase the concurrent request capability of the Flink framework for Redis clients.

[0017] S122. Add asynchronous timeout mechanism;

[0018] S123. Increase the statistical granularity of the time window from seconds to minutes;

[0019] Preferably, in step S123, the redis cluster access volume is reduced from 700w / s to 100w / s.

[0020] Preferably, the step S130 includes:

[0021] S131, split the super-large concurrent task into several subtasks and distribute them in various yarn clusters.

[0022] S132: Establish a division method for the subtasks and the consumed Kafka data, and perform parallel processing on the super-large concurrent tasks according to the division method.

[0023] An embodiment of the present invention further provides a traffic processing system based on a policy middle station, which is used to implement the above-mentioned traffic processing method based on a policy middle station, and the traffic processing system based on a policy middle station includes:

[0024] The cluster splitting module splits the Redis cluster and uses a separate Redis cluster to isolate high-query rate data sources;

[0025] The concurrent request module uses Flink asynchronous functions to increase the concurrent request capability of the Flink framework for the Redis client and add an asynchronous timeout mechanism.

[0026] A hybrid matching module that performs hybrid matching of long messages based on Google's regular engine re2j and Jvm's regular engine; and

[0027] The task dispersion module adjusts the memory model and distributes the subtasks under the corresponding task to different clusters for execution.

[0028] An embodiment of the present invention further provides a traffic processing device based on a policy center, including:

[0029] processor;

[0030] a memory storing executable instructions of the processor;

[0031] Among them, the processor is configured to execute the steps of the above-mentioned traffic processing method based on the policy middle station by executing the executable instructions.

[0032] An embodiment of the present invention also provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-mentioned traffic processing method based on the policy middle station.

[0033] The purpose of the present invention is to provide a traffic processing method, system, device and storage medium based on a policy middle platform, which can utilize the various characteristics of Kafka and Flink to achieve millisecond-level real-time policy analysis of millions of qps data volumes, and ensure stable operation without backlog or delay in data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0035] Figure 1 It is a flow chart of the traffic processing method based on the policy middle station of the present invention.

[0036] Figure 2 It is a structural diagram of the traffic processing system based on the policy middle station of the present invention.

[0037] Figure 3 It is a structural diagram of the traffic processing device based on the policy middle station of the present invention.

[0038] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present application. The present application can also be implemented or applied through other different specific implementation methods, and the details in the present application can also be modified or changed in various ways according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0040] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in a variety of different forms and is not limited to the embodiments described herein.

[0041] In the representations of the present application, the representations with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the represented specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples represented in the present application and the features of the different embodiments or examples, unless they contradict each other.

[0042] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the representation of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0043] In order to clearly describe the present application, components irrelevant to the description are omitted, and the same reference symbols are given to the same or similar components throughout the specification.

[0044] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0045] When a device is said to be "on" another device, it may be directly on the other device, but there may also be other devices between it. In contrast, when a device is said to be "directly" on another device, there are no other devices between it.

[0046] Although the terms first, second, etc. are used to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate the existence of features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

[0047] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present application. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0048] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with the relevant technical literature and the contents of the current disclosure, and unless defined, shall not be overly interpreted as ideal or very formal meanings.

[0049] Figure 1 Flow chart of the traffic processing method based on the policy middle station of the present invention. Figure 1 As shown, the traffic processing method based on the policy center of the present invention includes:

[0050] S110. Split the Redis cluster and use a separate Redis cluster for resource isolation of a high query rate data source.

[0051] S120, using Flink asynchronous functions, increasing the concurrent request capability of the Flink framework for Redis clients, and adding an asynchronous timeout mechanism. Among them, Flink is a distributed processing engine and framework for state calculation of bounded and unbounded data streams. In layman's terms, Flink is a stream computing framework that is mainly used to process streaming data.

[0052] S130. Mix and match long messages based on Google regular engine re2j and Jvm regular engine. Among them, the regular engine re2j provides an efficient and deterministic regular expression matching algorithm, avoiding the infinite loop problem that may be caused by traditional PCRE (Perl) compatible regular expressions). It is designed to be fast, safe and easy to understand, making it an ideal choice for projects with high requirements for performance and stability. JVM is the abbreviation of Java Virtual Machine. JVM is a specification for computing devices. It is a fictional computer that is implemented by simulating various computer functions on an actual computer.

[0053] S140: Adjust the memory model, and distribute the subtasks under the corresponding task to different clusters for execution.

[0054] The performance design of the strategy middleware in the present invention is a complete system engineering, and the goal cannot be achieved by solving a single point. First, according to the running status of each Flink node, the overall processing link process and the challenges faced are analyzed, among which the following three key points are listed in order of impact: (1) Window counting statistical performance bottleneck (Redis implementation) (2) Complex regular matching long message performance issues (3) Stability and performance issues of Flink super-large clusters (thousands of concurrent connections).

[0055] In a preferred embodiment, step S110 includes:

[0056] S111. Split the Redis cluster, and group the data sources according to the query rate based on a preset threshold.

[0057] S112: Use a separate Redis cluster for resource isolation for a high query rate data source whose query rate is greater than a preset threshold, and use the remaining Redis clusters for a low query rate data source whose query rate is less than the preset threshold, but the present invention is not limited thereto.

[0058] In a preferred embodiment, step S120 includes:

[0059] S121. Use Flink asynchronous functions to increase the concurrent request capability of the Flink framework for Redis clients.

[0060] S122. Add an asynchronous timeout mechanism.

[0061] S123. Increase the statistical granularity of the time window from seconds to minutes, but not limited thereto.

[0062] In a preferred embodiment, in step S123, the access volume of the redis cluster is reduced from 700w / s to 100w / s, but not limited thereto.

[0063] In a preferred embodiment, step S130 includes:

[0064] S131. Decompose the super-large concurrent task into several subtasks and distribute them in various YARN clusters.

[0065] S132. Establish a partitioning method for subtasks and consuming Kafka data, and process super-large concurrent tasks in parallel according to the partitioning method. Kafka is an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the website. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern network. This data is usually solved by processing logs and log aggregation due to throughput requirements. This is a viable solution for log data and offline analysis systems like Hadoop, but with the limitation of real-time processing. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time messages through clusters.

[0066] The present invention targets the performance bottleneck of window counting statistics: 1) increasing the concurrency capability of the Redis Client inside Flink through Flink Async Function; 2) reducing the processing pressure of the Redis Cluster by optimizing the counting algorithm. AsyncFunction is the core interface of Flink asynchronous I / O, which defines how to convert synchronous I / O operations into asynchronous operations. The AsyncFunction interface contains the following two methods: asyncinvoke: receives input elements, performs asynchronous I / O operations, and passes the results to the callback function. timeout: specifies the timeout of the asynchronous operation, which will trigger an exception after the timeout.

[0067] To address the performance issues of long messages with complex regular expressions: 1) Introduce a regular expression engine with higher performance for mixed use; 2) Split subtasks and distribute them to various yarn clusters to avoid processing blockage.

[0068] Stability and performance issues of Flink super-large clusters (thousands of concurrent users): 1) Using Flink features to split large tasks, the processing tasks of the same data source can be distributed to different yarn clusters to achieve true distributed operation. Yarn is a resource management and task scheduling platform that can be used to process MapReduce jobs, stream processing jobs, and other types of distributed processing jobs.

[0069] The present invention improves the processing performance of the policy center, with MTTD (security incident detection time) reaching seconds and SLA reaching 999.

[0070] The present invention cleverly utilizes various features of Kafka and Flink to solve a series of performance and stability challenges faced by the strategy middle platform, and realizes real-time strategy analysis of millions of qps data in milliseconds; data processing has no backlog, no delay, and stable operation; it provides the company with real-time detection and analysis such as intrusion detection, threat analysis, and data leakage detection 24 hours a day throughout the year, and plays an important role in the Panshi and network protection operations over the years;

[0071] Embodiments of the present invention are as follows:

[0072] Regarding the performance bottleneck of window count statistics: According to experience, the biggest performance bottleneck of this type of stream processing system is usually the interaction with external systems. The window count statistics function is implemented through Redis. Although Redis's performance is at the first-tier level in the industry, it still cannot meet the performance requirements in the strategy middleware scenario; therefore, first (1) split the Redis cluster and use a separate Redis cluster for resource isolation of high qps data sources (2) Flink is a single-threaded model, so the Async Function (a new feature contributed by Alibaba to Apache Flink) mechanism is used to increase Flink's concurrent request capabilities for Redis, and add an Async timeout mechanism (3) increase the time window statistical granularity from seconds to minutes (the number of visits to the Redis cluster dropped from 700w / s to 100w / s)

[0073] Regarding the performance issue of long message matching with complex regular expressions: We solved this problem from two aspects: (1) Regular expression execution. We introduced a hybrid of Google’s regular expression engine re2j and Jvm’s regular expression engine, taking advantage of their respective strengths and improving the matching efficiency by more than 30%. (2) Flink layer optimization. Since the processing of complex regular expressions on individual nodes takes a long time, the upstream node LocalBuffer, NetworkBuffer, etc. will be full, which will cause the entire Flink cluster to be blocked. In order to solve this problem, we adjusted the data distribution strategy to the Rescala method, adjusted the memory model, and distributed the subtasks under the corresponding Task in different clusters for execution. This achieved fault isolation and maximized the use of computing resources.

[0074] Stability and performance issues of Flink super-large clusters (thousands of concurrent users): The HTTPS data source qps reaches more than 300,000, and the number of complex rules processed is nearly 50, and the corresponding Flink concurrent nodes reach thousands. At this time, we are faced with a problem that such a large Flink task cannot be completely placed in a Yarn cluster, and the startup and restart of such a large task will be a disaster. Therefore, we studied the internal mechanism of Flink Kafka Connector and found that Flink consumes Kafka through Assign (Kafka rebalance is impossible), so we decompose the super-large concurrent tasks into subtasks and distribute them in various Yarn clusters. The subtasks consume Kafka data by specifying fixed partitions, and jointly complete the processing tasks of the same data source. The above design uses a relatively simple and clean way to perfectly resolve this problem and greatly improve stability and performance.

[0075] The above specific implementation methods do not constitute a limitation on the protection scope of the present invention. Any technician familiar with the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present application without departing from the scope of the technical solution of the present application, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application still falls within the scope of protection of the technical solution of the present application.

[0076] Figure 2 Schematic diagram of the structure of the traffic processing system based on the policy center of the present invention. Figure 2 As shown, an embodiment of the present invention further provides a traffic processing system based on a policy middle station, which is used to implement the above-mentioned traffic processing method based on a policy middle station. The traffic processing system 5 based on a policy middle station includes:

[0077] The cluster splitting module 51 splits the Redis cluster and uses a separate Redis cluster for resource isolation of a high query rate data source.

[0078] The concurrent request module 52 adopts the Flink asynchronous function, increases the concurrent request capability of the Flink framework for the Redis client, and adds an asynchronous timeout mechanism.

[0079] The hybrid matching module 53 performs hybrid matching of long messages based on the Google regular engine re2j and the Jvm regular engine.

[0080] The task dispersing module 54 adjusts the memory model and distributes the subtasks under the corresponding task to different clusters for execution.

[0081] In a preferred embodiment, the cluster splitting module 51 is configured to split the Redis cluster and group the data sources based on the query rate based on a preset threshold; a high query rate data source with a query rate greater than the preset threshold is isolated using a separate Redis cluster for resource isolation, and a low query rate data source with a query rate less than the preset threshold is isolated using the remaining Redis clusters.

[0082] In a preferred embodiment, the concurrent request module 52 is configured to use Flink asynchronous functions to increase the concurrent request capability of the Flink framework for the Redis client; add an asynchronous timeout mechanism; increase the statistical granularity of the time window from seconds to minutes;

[0083] In a preferred embodiment, the concurrent request module 52 is also configured to reduce the redis cluster access volume from 700w / s to 100w / s.

[0084] In a preferred embodiment, the hybrid matching module 53 is configured to decompose the super-large concurrent task into several subtasks distributed in various YARN clusters, establish a segmentation method for subtasks and consumed Kafka data, and perform parallel processing on the super-large concurrent tasks according to the segmentation method.

[0085] The traffic processing system based on the policy middle platform of the present invention can utilize various characteristics of Kafka and Flink to realize millisecond-level real-time policy analysis of millions of qps data volumes, with no backlog, no delay, and stable operation of data processing.

[0086] The embodiment of the present invention also provides a traffic processing device based on a policy middle station, including a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of the traffic processing method based on the policy middle station via the executable instructions.

[0087] As shown above, the traffic processing device based on the policy middle platform of this embodiment of the present invention can utilize various characteristics of Kafka and Flink to realize millisecond-level real-time policy analysis of millions of qps data volumes, with no backlog, no delay, and stable operation of data processing.

[0088] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0089] Figure 3 This is a schematic diagram of the structure of the traffic processing device based on the policy center of the present invention. Figure 3 The electronic device 600 according to this embodiment of the present invention is described. Figure 3 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0090] like Figure 3 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0091] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0092] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0093] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0094] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0095] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0096] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the traffic processing method based on the policy middle station are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above method section of this specification.

[0097] As shown above, the traffic processing system based on the policy middle platform of this embodiment of the present invention can utilize various characteristics of Kafka and Flink to realize millisecond-level real-time policy analysis of millions of qps data volumes, with no backlog, no delay, and stable operation of data processing.

[0098] Figure 4 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Figure 4 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0099] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0100] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0101] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0102] In summary, the purpose of the present invention is to provide a traffic processing method, system, device and storage medium based on a policy middle platform, which can utilize the various characteristics of Kafka and Flink to achieve millisecond-level real-time policy analysis of millions of qps data volumes, and ensure stable operation without backlog or delay in data processing.

[0103] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A traffic processing method based on a policy center, characterized in that: The following steps are involved: S110, split the Redis cluster and use a separate Redis cluster for resource isolation of high query rate data sources; S120, using Flink asynchronous functions, increasing the concurrent request capability of the Flink framework for Redis clients, and adding an asynchronous timeout mechanism; S130, performing mixed matching of long messages based on Google regular engine re2j and Jvm regular engine; as well as S140: Adjust the memory model, and distribute the subtasks under the corresponding task to different clusters for execution.

2. The traffic processing method based on the policy middle station according to claim 1 is characterized in that: The step S110 includes: S111, splitting the Redis cluster, and grouping the data sources according to the query rate based on a preset threshold; S112: Use a separate Redis cluster for resource isolation of a high query rate data source whose query rate is greater than a preset threshold, and use the remaining Redis clusters for a low query rate data source whose query rate is less than the preset threshold.

3. The traffic processing method based on the policy middle station according to claim 1 is characterized in that: The step S120 includes: S121. Use Flink asynchronous functions to increase the concurrent request capability of the Flink framework for Redis clients. S122. Add asynchronous timeout mechanism; S123. Increase the statistical granularity of the time window from seconds to minutes.

4. The traffic processing method based on the policy middle station as claimed in claim 3 is characterized in that: In step S123, the access volume of the redis cluster is reduced from 700w / s to 100w / s.

5. The traffic processing method based on the policy middle station as claimed in claim 3 is characterized in that: The step S130 includes: S131, split the super-large concurrent task into several subtasks and distribute them in various yarn clusters. S132: Establish a division method for the subtasks and the consumed Kafka data, and perform parallel processing on the super-large concurrent tasks according to the division method.

6. A traffic processing system based on a policy middle station, used to implement the traffic processing method based on a policy middle station according to claim 1, characterized in that: include: The cluster splitting module splits the Redis cluster and uses a separate Redis cluster to isolate high-query rate data sources; The concurrent request module uses Flink asynchronous functions to increase the concurrent request capability of the Flink framework for the Redis client and add an asynchronous timeout mechanism. The hybrid matching module is based on the Google regular engine re2j and the Jvm regular engine to perform hybrid matching of long messages; as well as The task dispersion module adjusts the memory model and distributes the subtasks under the corresponding task to different clusters for execution.

7. A traffic processing device based on a policy center, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to execute the steps of the traffic processing method based on the policy middle station according to any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by the processor, the steps of the traffic processing method based on the policy middle station as described in any one of claims 1 to 5 are implemented.

Images