A real-time enriched streaming data processing system

Abstract

The application discloses a kind of real-time enrichment streaming data processing techniques, including overall architecture, overall architecture includes reverse search process, reverse search process includes the following steps: user access data is entered into EventTopic by kafka, is preprocessed to the information collected by Yan-resource management and Spark / storm, and each type of data is identified, whether it is network security data is judged, if it is login event, event enters IP reverse search logic configuration and judges whether to open;If IP reverse search logic configuration is judged to open, it will enter according to ip and site id, to redis to see whether there is enrichment information judgment;If IP reverse search logic configuration is not judged to open, directly enter ES, then end;If there is enrichment information, information will be processed by IP reverse search enrichment, and the expiration time of key is extended, then enter ES, then end.Kafka data backlog problem no longer occurs, and the processing delay of stream processing program is reduced from hours to seconds, reducing page response time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a real-time enrichment stream data processing system. BACKGROUND

[0002] Data processing is a basic link of system engineering and automatic control. Data processing runs through various fields of social production and social life. The development of data processing technology and the breadth and depth of its application have greatly influenced the process of human social development.

[0003] Chinese patent number CN109074447A provides techniques for metadata processing of any number of security policies that can be used to encode code running on a processor. Metadata can be added to each word in the system and can use a metadata processing unit that works in parallel with the data stream to enforce any set of policies. In one aspect, the metadata can be characterized as unrestricted and software programmable to be applicable to a variety of metadata processing policies. The techniques and policies have various uses, including security, safety, and synchronization, for example. Additionally, aspects and techniques are described regarding metadata processing in RISC-V architecture-based embodiments.

[0004] (1) The existing similar data integration method one: through the api call, the related business data is associated and displayed, the page response delay is high;

[0005] (2) The existing similar data integration method two: through direct association query to obtain business data from the business library, which causes great pressure on the business library, and seriously affects the processing efficiency of stream processing, which easily leads to data accumulation and data processing delay;

[0006] (3) The existing similar data integration method three: build an offline data warehouse, build a data mart, complete the construction of a large wide table, and provide it for upper-layer application. The data response of this method is T+1, and under the background of high real-time requirements of network communication and security response, it cannot provide timely response to network security events. SUMMARY

[0007] (I) Technical problems to be solved

[0008] In view of the deficiencies of the prior art, the purpose of the present application is to provide a real-time enrichment stream data processing system for improving page response efficiency and reducing response delay.

[0009] (II) Technical solutions

[0010] In order to achieve the above-mentioned purpose, the technical scheme of a real-time enrichment flow data processing system provided by the present application is as follows: the system comprises an overall framework, the overall framework comprises a reverse search process, asset library information and internal network IP library information, the reverse search process comprises the following steps:

[0011] Enter into Event through kafka, and then processed through Yarn-resource management and

[0012] Spark / storm, and determine whether it is a login event through the system;

[0013] 1.1), if it is a login event, enter the event into IP reverse search logic configuration to determine whether to open;

[0014] 1.1.1), if the IP reverse search logic configuration determines to open, enter into judging whether there is enrichment information according to the IP and site id in redis;

[0015] 1.1.2), if the IP reverse search logic configuration determines not to open, directly enter into ES, and then end;

[0016] 1.1.1.1), if there is enrichment information, the information will be processed through IP reverse search enrichment, and the expiration time of the key will be extended, then enter into ES, and then end;

[0017] 1.1.1.2), if there is no enrichment information, the information will be queried through API to enrich the information, and the enriched information is written into redis cache, and the login event data of the enrichment processing is entered into RS, then enter into ES, and the data is entered into Redis;

[0018] 1.2), if it is not a login event, other event enrichment and robustness processing are carried out, then enter into ES, and then end.

[0019] As a preferred scheme, the system for judging whether there is enrichment information according to the IP and site id in redis in the reverse search process is in communication with Redis, and MongoDB enters into redis by buffering hot IP information.

[0020] As a preferred scheme, the overall framework comprises API, DM, MongoDB, Yarn-logic processing, Redis, ES, Spark cluster, Alarms, Yarn-logic processing, data replication, Hive, Spark / storm, Spark / storm cluster, Event, Logstash and Kafka, and the Kafka comprises internal loop event, communication pair, suspicious file, Event, artificial intelligence / other and log.

[0021] As a preferred solution, the inner loop event, communication pair, suspicious file, Event and artificial intelligence / other enter the Spark / storm cluster processing, the log enters the Spark / storm cluster in turn through Logstash and Event, and then the data is processed through Yarn-logical processing and enters the Alarms and ES.

[0022] As a preferred solution, the Yarn-logical processing enters the Yarn-logical processing through the Spark cluster after entering the Alarms, and then the warning data after processing enters the ES, and if there is asset discovery information, it will enter the DM, and then enter the API.

[0023] As a preferred solution, the MongoDB data enters the API, and enters the Redis through the warning rule and the homepage index.

[0024] As a preferred solution, the data in the DM enters the Redis through the configuration information and asset attribute, and the data in the Redis enters the Yarn-logical processing, and enters the Yarn-logical processing through enrichment and robustness, and the data in the Redis enters the API.

[0025] As a preferred solution, the data in the ES enters the API, and the data in the ES enters the Hive through data replication for processing.

[0026] As a preferred solution, the data in the ES enters the Spark / storm through the file, message and communication pair, and the data in the Spark / storm enters the ES through the submission state and report result update.

[0027] As a preferred solution, the asset library information includes company, unit, site, security partition and device name, and the internal network ip library information includes ip type, company, unit, site, security partition, device name, longitude and latitude.

[0028] Compared with the prior art, the application provides a real-time enrichment stream data processing system, which has the following beneficial effects:

[0029] 1. Through big data technology, business data enrichment can be realized in the stream processing stage in advance, which can greatly reduce the time required for upper application self-association calculation, and improve the page response efficiency.

[0030] 2. Through hot data caching, the performance pressure on the business library can be greatly reduced, the problem of kafka data backlog and stream processing response delay can be reduced, and the efficiency of data real-time warehousing can be improved.

[0031] 3、Compared with the asset library and the intranet library information directly associated with the stream processing program, the stream processing enrichment + redis can greatly improve the processing efficiency of the stream processing and reduce the occurrence of delay problems on the basis of ensuring data enrichment. BRIEF DESCRIPTION OF DRAWINGS

[0032] Fig. 1 is the framework of the present application;

[0033] Fig. 2 is the reverse lookup flowchart of the present application. DETAILED DESCRIPTION

[0034] The present application will be further described and explained with reference to specific embodiments and the accompanying drawings:

[0035] Please refer to Figs. 1-2 The present application includes an overall framework, the overall framework includes a reverse lookup flow, asset library information and intranet IP library information, and the reverse lookup flow includes the following steps:

[0036] Enter Event through kafka, and then process through Yarn-resource management and Spark / storm, and judge whether it is a login event through the system:

[0037] 1.1), if it is a login event, enter the event into IP reverse lookup logic configuration to judge whether it is opened; 1.1.1), if the IP reverse lookup logic configuration is judged to be opened, enter according to the ip and site id, and check whether there is enrichment information in redis; 1.1.2), if the IP reverse lookup logic configuration is not judged to be opened, directly enter into ES, and then end; 1.1.1.1), if there is enrichment information, the information will be processed through IP reverse lookup enrichment, and the expiration time of the key will be extended, then enter into ES, and then end; 1.1.1.2), if there is no enrichment information, the information will be queried through PI enrichment information, and the enrichment information will be written into redis cache, and the login event data of enrichment processing will be entered into RS, then enter into ES, and the data will be entered into Redis; 1.2), if it is not a login event, process through other event enrichment and robustness, then enter into ES, and then end.

[0038] As a preferred scheme, the system for judging whether there is enrichment information according to the ip and site id in the reverse lookup flow and Redis are interconnected, and MongoDB enters the hot ip information into redis through buffering.

[0039] As a preferred solution, the overall framework includes API, DM, MongoDB, Yarn-logic processing, Redis, ES, Spark cluster, Alarms, Yarn-logic processing, data replication, Hive, Spark / storm, Spark / storm cluster, Event, Logstash and Kafka, Kafka includes inner loop event, communication pair, suspicious file, Event, artificial intelligence / other and log, inner loop event, communication pair, suspicious file, Event and artificial intelligence / other enter the Spark / storm cluster processing, log enters the Spark / storm cluster through Logstash and Event in turn, then data is processed through Yarn-logic processing and enters Alarms and ES, Yarn-logic processing enters Alarms after entering Spark cluster and then enters Yarn-logic processing, then the processed warning data enters ES, if there is asset discovery information, it will enter DM, then enter API, MongoDB data enters API, at the same time, through warning rules and homepage indicators, it enters Redis, data in DM enters Redis through configuration information and asset attributes, data in Redis enters Yarn-logic processing, at the same time, through enrichment and robustness, it enters Yarn-logic processing, data in Redis enters API, data in ES enters API, data in ES enters Hive through data replication for processing, data in ES enters Spark / storm through file, message and communication pair for inspection, data in Spark / storm enters ES through inspection state and report result update.

[0040] As a preferred solution, the asset library information includes company, unit, site, security partition and device name, and the intranet ip library information includes ip type, company, unit, site, security partition, device name, longitude and latitude.

[0041] The working principle of the application is: through real-time stream data processing architecture transformation, introduce redis hot cache, synchronize asset library and intranet library data to redis cache, form a unique key through site id and device id for stream processing, enrich data in redis, form a large wide table in ES and directly call upper layer application through API, compared with directly associating asset library and intranet library information with stream processing program, stream processing enrichment + redis can greatly improve the processing efficiency of stream processing and reduce the occurrence of delay problem on the basis of realizing data enrichment, transaction changes of asset library data and intranet library data will be changed in real time to redis through a synchronization program, and for data that is not hit when the stream processing program is enriched, the corresponding database will be accessed to enrich and update to redis.

[0042] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the protection scope of the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced without departing from the essence and scope of the technical solutions of the present application.

Claims

1. A real-time enrichment streaming data processing system, comprising an overall architecture, characterized in that the overall architecture comprises a reverse lookup process, asset library information and intranet IP library information, and the reverse lookup process comprises the following steps: Enter into Event through kafka, and then processed through Yarn-resource management and Spark / storm, and whether it is a login event is judged through the system: If it is a login event, the event is entered into IP reverse lookup logic configuration to determine whether to open; 1.1.1), if the IP reverse lookup logic configuration determines to open, it will be entered into the judgment of whether there is enrichment information according to the ip and site id to the redis; 1.1.2), if the IP reverse lookup logic configuration does not determine to open, it is directly entered into ES, and then ended; 1.1.1.1), if there is enrichment information, the information will be processed through IP reverse lookup enrichment, and the expiration time of the key will be extended, and then entered into ES, and then ended; 1.1.1.2), if there is no enrichment information, the information will be queried through the API, enriched, and written to the redis cache, and the enriched login event data will be entered into RS, and then into ES, and the data will be entered into Redis; 1.2), if it is not a login event, it will be enriched and robustly processed through other events, then enter ES, and then end.

2. A real-time enriched streaming data processing system as claimed in claim 1, wherein: In the reverse lookup process, according to the ip and site id, it is determined whether the system and Redis are interconnected by checking whether there is enrichment information in Redis, and MongoDB enters the hot ip information into Redis through buffering.

3. The real-time enriched streaming data processing system of claim 1, wherein: The overall architecture includes API, DM, MongoDB, Yarn-logical processing, Redis, ES, Spark cluster, Alarms, Yarn-logical processing, data replication, Hive, Spark / storm, Spark / storm cluster, Event, Logstash and Kafka, and the Kafka includes internal loop events, communication pairs, suspicious files, Event, artificial intelligence / other and logs.

4. A real-time enriched streaming data processing system as claimed in claim 3, wherein: The internal loop events, communication pairs, suspicious files, Event and artificial intelligence enter the Spark / storm cluster for processing, the logs enter the Spark / storm cluster through Logstash and Event in turn, and then the data is processed through Yarn-logical processing and enters Alarms and ES.

5. A real-time enriched streaming data processing system as claimed in claim 3, wherein: The Yarn-logical processing enters Alarms, then enters Yarn-logical processing through the Spark cluster, and then the warning data enters ES after processing, and if there is asset discovery information, it will enter DM, and then enter API.

6. A real-time enriched streaming data processing system as claimed in claim 3, wherein: The MongoDB data enters API, and enters Redis through warning rules and homepage indicators.

7. A real-time enriched streaming data processing system as claimed in claim 3, wherein: The data in DM enters Redis through configuration information and asset attributes, and the data in Redis enters Yarn-logical processing, and enters Yarn-logical processing through enrichment and robustness, and the data in Redis enters API.

8. A real-time enriched streaming data processing system as claimed in claim 3, wherein: The data in ES enters API, and the data in ES enters Hive for processing through data replication.

9. A real-time enriched streaming data processing system as claimed in claim 3, wherein: The data in ES enters Spark / storm through files, messages and communication pairs, and the data in Spark / storm enters ES through submission status and report result updates.

10. The real-time enriched streaming data processing system of claim 1, wherein: The asset library information includes company, unit, site, security partition and device name, and the internal network ip library information includes ip type, company, unit, site, security partition, device name, longitude and latitude.

Images