Multi-source heterogeneous data processing method and device based on security situation

Abstract

The present invention discloses a multi-source heterogeneous data processing method and device based on security situation, which relates to the field of data processing technology and can also be used in the financial field, wherein the method comprises: obtaining security situation data of at least two security products, and generating an event stream of a real-time computing task corresponding to the security product; creating a configuration stream according to at least two security product rules; connecting the configuration stream with the event stream after broadcasting the configuration stream; reading the at least two security product rules in the configuration stream, and processing the security situation data of the security products in each event stream connected to the configuration stream according to the at least two security product rules, and obtaining the processed security situation data of each security product. The present invention can standardize the format of security situation data of different security products, and solve the problem that heterogeneous data from multiple sources cannot be processed in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology and can also be used in the financial field, and in particular to a multi-source heterogeneous data processing method and device based on security situation. Background Art

[0002] As network security issues become increasingly prominent, the current level of attention paid to network security work is also increasing, and high requirements are also placed on corporate security operations. The People's Bank of China is responsible for financial risk prevention and emergency response, financial infrastructure supervision, financial network security and information guidance, etc. Therefore, the People's Bank of China has built an industry-level security situation awareness platform. In accordance with the requirements of the People's Bank of China's Financial Network Security Situation Awareness Platform (hereinafter referred to as the "Platform"), all financial institutions must report and connect security incident data and share threat intelligence information in accordance with the data access and information sharing standards released by the platform.

[0003] At the same time, the security situation data of various financial institutions are usually heterogeneous, including structured and unstructured data. Therefore, the process of security incident data reporting by various financial institutions has the problem of multi-source heterogeneous data needing to adapt to the standardized structure of the People's Bank of China. However, the processing methods of existing technologies, on the one hand, cannot process heterogeneous data from multiple sources in a timely manner and cannot quickly detect potential threats; on the other hand, customized logic development is usually required to complete data processing; on the other hand, security situation data cannot be merged or suppressed, and at the same time, for more complex data rules, it cannot be combined with Mysql and other storage for processing.

[0004] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section. Summary of the invention

[0005] The embodiments of the present invention provide a multi-source heterogeneous data processing method and device based on security situation, so as to overcome the problem in the prior art that heterogeneous data from multiple sources cannot be processed in a timely manner.

[0006] An embodiment of the present invention provides a multi-source heterogeneous data processing method based on security situation, comprising:

[0007] Acquire security situation data of at least two security products, and generate an event stream of a real-time computing task corresponding to each security product based on the security situation data of the security product;

[0008] Create a configuration flow based on at least two security product rules;

[0009] After broadcasting the configuration stream, connecting the configuration stream with the event stream of the real-time computing task corresponding to each of the security products;

[0010] The at least two security product rules in the configuration flow are read, and according to the at least two security product rules, the security situation data of the security products in each event flow connected to the configuration flow are processed to obtain the processed security situation data of each security product.

[0011] In some embodiments, creating a configuration flow according to at least two security product rules includes:

[0012] Obtain the rule configuration template from the real-time analysis database;

[0013] Parsing the rule configuration template to obtain the at least two security product rules;

[0014] The configuration flow is created by inheriting the at least two security product rules through an abstract class in a distributed stream processing engine.

[0015] In some embodiments, the creating a configuration flow according to at least two security product rules further includes:

[0016] Based on a refresh cycle, the rule configuration template in the real-time analysis database is dynamically refreshed.

[0017] In some embodiments, the security product rules include at least one of the following: field mapping rules, data merging rules, data filtering rules and sampling rules; wherein the field mapping rules include at least one of one-to-one mapping, combined dictionary mapping and combined configuration mapping.

[0018] In some embodiments, the security product rule includes a sampling rule, and the step of reading the at least two security product rules in the configuration stream and processing the security situation data of the security products in each event stream connected to the configuration stream according to the at least two security product rules to obtain the processed security situation data of each security product includes:

[0019] Reading sampling rules in the configuration stream;

[0020] Determine, according to the sampling rule, a sampling ratio and a sampling time of the security situation data of the security product in each of the event streams connected to the configuration stream;

[0021] According to the sampling time of the security situation data of each security product in the event stream, a sampling window is opened to cache the security situation data of the security product until the proportion of the cached security situation data reaches the sampling proportion of the security situation data of the security product, and the sampling window is closed to obtain the processed security situation data of each security product.

[0022] In some embodiments, the security product rule includes a merge rule, and the step of reading the at least two security product rules in the configuration stream and processing the security situation data of the security products in each event stream connected to the configuration stream according to the at least two security product rules to obtain the processed security situation data of each security product includes:

[0023] Reading the merging rules in the configuration stream;

[0024] Determine, according to the merging rule, the merging fields and merging cycles of the security situation data of the security product in each of the event streams connected to the configuration stream;

[0025] According to the merging fields of the security situation data of the security products in each of the event streams, and according to the merging period of the security situation data, the security situation data of the security products in the event stream are grouped to obtain a grouping result;

[0026] Based on the grouping result of the security situation data of each security product in the event stream, the processed security situation data of each security product is obtained.

[0027] In some embodiments, the reading of the at least two security product rules in the configuration stream, and processing the security situation data of the security products in each event stream connected to the configuration stream according to the at least two security product rules to obtain the processed security situation data of each security product, further includes:

[0028] According to the merging field of the security situation data of the security product in each of the event streams, determining whether the current group in the security situation data of the security product in the event stream has been merged in the current merging cycle;

[0029] If the security situation data of the security product in the event stream has been merged in the current merging period for the current group, an alarm corresponding to the current merging period is generated, the alarms corresponding to the current merging period are merged, and the merging counter of the current group is updated.

[0030] In some embodiments, the method further comprises:

[0031] The processed security situation data of at least two security products are sent to the security operation platform for sharing.

[0032] The embodiment of the present invention also provides a multi-source heterogeneous data processing device based on security situation, so as to overcome the problem in the prior art that heterogeneous data from multiple sources cannot be processed in a timely manner.

[0033] The embodiment of the present invention further provides a multi-source heterogeneous data processing device based on security situation, comprising:

[0034] A generation module, used to obtain security situation data of at least two security products, and generate an event stream of a real-time computing task corresponding to each security product based on the security situation data of the security product;

[0035] A creation module, configured to create a configuration flow based on at least two security product rules;

[0036] A connection module, used for connecting the configuration stream with the event stream of the real-time computing task corresponding to each of the security products after broadcasting the configuration stream;

[0037] A processing module is used to read the at least two security product rules in the configuration flow, and process the security situation data of the security products in each event flow connected to the configuration flow according to the at least two security product rules to obtain the processed security situation data of each security product.

[0038] In some embodiments, the creation module includes:

[0039] An acquisition unit, used to acquire a rule configuration template from a real-time analysis database;

[0040] A parsing unit, configured to parse the rule configuration template to obtain the at least two security product rules;

[0041] The creation unit is used to inherit the at least two security product rules through the abstract class in the distributed stream processing engine to create the configuration stream.

[0042] In some embodiments, the creation module further includes:

[0043] The refresh unit is used to dynamically refresh the rule configuration template in the real-time analysis database based on a refresh cycle.

[0044] In some embodiments, the security product rules include at least one of the following: field mapping rules, data merging rules, data filtering rules and sampling rules; wherein the field mapping rules include at least one of one-to-one mapping, combined dictionary mapping and combined configuration mapping.

[0045] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multi-source heterogeneous data processing method based on security situation when executing the computer program.

[0046] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multi-source heterogeneous data processing method based on security situation is implemented.

[0047] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the multi-source heterogeneous data processing method based on security situation is implemented.

[0048] The embodiments of the present invention provide a multi-source heterogeneous data processing method and device based on security situation. The present invention generates an event stream corresponding to the security product from the security situation data of each security product based on a real-time computing task, and creates a configuration stream for the security product rules based on the real-time computing task, and connects the configuration stream with the event stream by broadcasting the configuration stream, so that the security situation data of the security product in the event stream can be configured based on the security product rules in the configuration stream. The scheme of the present invention can meet the requirements of the rule logic of different security products by uniformly configuring at least two security product rules to form a configuration stream, making the deployment of the rule engine more convenient, without the need to input complex statement logic, and can be effectively configured based on the template. At the same time, the present invention configures unified rules to the security situation data of the security product in each event stream through the configuration stream, and can standardize the format of the security situation data of different security products, solving the problem that heterogeneous data from multiple sources cannot be processed in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0050] Figure 1 A flowchart of a multi-source heterogeneous data processing method based on security situation provided by the present invention;

[0051] Figure 2 A flowchart of another method for processing multi-source heterogeneous data based on security situation provided in an embodiment of the present invention;

[0052] Figure 3A flow chart of a method for implementing a sampling rule provided by an embodiment of the present invention;

[0053] Figure 4 A flowchart of a method for implementing a merging rule provided by an embodiment of the present invention;

[0054] Figure 5 A flowchart of another multi-source heterogeneous data processing method based on security situation provided by the present invention;

[0055] Figure 6 A structural diagram of a multi-source heterogeneous data processing device based on security situation provided by the present invention;

[0056] Figure 7 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0058] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with relevant regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0059] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with relevant regulations.

[0060] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0061] In recent years, based on the evaluation and use of commonly used multi-source heterogeneous data processing products, the following problems are commonly found in the use process:

[0062] The learning cost is high. Users need to understand the working principle and configuration method to better use this tool. The configuration file structure and some concepts may take some time to understand and master, and there is a certain learning curve.

[0063] Only offline batch data synchronization is supported, not real-time data synchronization. This means that if users need to synchronize data in real time, it may not meet their needs.

[0064] Compared to some open source projects, the community is relatively small, which may affect the efficiency of obtaining community support, especially in some unofficial usage scenarios.

[0065] This product is generally a stand-alone product and does not support distributed computing. When faced with high concurrency and large-scale data processing, the pressure on the stand-alone system may become very high, which may lead to problems such as performance degradation and prolonged response time.

[0066] Running full memory operations puts a lot of pressure on a single machine. Single machine environments usually have limited memory capacity, and processing large-scale data may be limited by memory. If the amount of data exceeds the memory capacity of a single machine, it may cause performance degradation or program crashes.

[0067] It is not possible to combine other data sources for rule processing. In some specific scenarios, additional work may be required to ensure the quality and accuracy of data transmission.

[0068] In order to solve the above technical problems, the present invention provides a multi-source heterogeneous data processing method based on security situation, and the execution subject of the method includes but is not limited to a computer, a processor, etc.

[0069] Figure 1 The flowchart of a multi-source heterogeneous data processing method based on security situation provided by the present invention. Figure 1 As shown, the method comprises the following steps:

[0070] S1. Obtain security situation data of at least two security products, and generate an event stream of a real-time computing task corresponding to each security product based on the security situation data of the security product.

[0071] Specifically, the security product can be various financial institutions, including but not limited to banks, insurance companies, securities companies, etc. Of course, the security product can also be other products, and the present disclosure does not limit this. The security situation data of the security product refers to the log data related to the security events of the security product. The real-time computing task refers to the computing task of processing and analyzing the data in the event stream when the event stream generates data in real time. Optionally, the real-time computing task of the present invention can be implemented by a distributed stream processing framework, and the distributed stream processing framework can be Apache Flink, and each type of security product has a separate Flink Jar task. Optionally, the present invention can obtain the data source structure of the security situation data of each security product according to the parameters, and obtain the event stream of the real-time computing task corresponding to the security product according to the data source structure of the security situation data of each security product. Among them, the parameter can be FlinkKafkaConsumer in Flink (a consumer connector provided by Apache Flink for reading data from Kafka topic) and the like.

[0072] S2. Create a configuration flow based on at least two security product rules.

[0073] Specifically, the security product rules refer to rules set to solve the heterogeneous format processing, complex multi-source rule processing and data suppression of the security situation data of each security product, and better support the format standardization of the security situation data. The at least two security product rules are unified configuration rules for all security products. The present invention can meet the requirements of the rule logic of different security products by uniformly configuring at least two security product rules to form a configuration flow, making the deployment of the rule engine more convenient, without the need to input complex statement logic, and can be effectively configured based on the template.

[0074] When processing data, the data needs to be configured according to the security product rules, so a configuration flow needs to be created.

[0075] S3. After broadcasting the configuration stream, connect the configuration stream with the event stream of the real-time computing task corresponding to each of the security products.

[0076] Specifically, the configuration stream is broadcast to each task manager in the distributed stream processing framework. Each task manager corresponds to an event stream and is responsible for executing tasks and processing data in the event stream. When the event stream and the configuration stream are connected, the event stream can read different security product rules from the configuration stream and configure the security situation data in the event stream.

[0077] S4. Read the at least two security product rules in the configuration flow, and process the security situation data of the security products in each event flow connected to the configuration flow according to the at least two security product rules to obtain the processed security situation data of each security product.

[0078] Specifically, when the event stream and the configuration stream are connected, the event stream can read each security product rule from the configuration stream, and configure the security situation data in the event stream based on the security product rule in the configuration stream. The present invention configures unified rules to the security situation data of security products in each event stream through the configuration stream, and can standardize the format of security situation data of different security products, solving the problem that the prior art cannot process heterogeneous data from multiple sources.

[0079] In some embodiments, the method not only includes the above steps S1 to S4, but also includes step S5, sending the processed security situation data of at least two security products to the security operation platform for sharing.

[0080] Specifically, the security situation data of the processed security products is data with a unified standardized structure, which is sent to the security operation platform for sharing. It can be used for common PBC situation reporting, which is more convenient and flexible. It can effectively generate reporting data and support effective analysis by operators, and can timely carry out financial risk prevention and emergency response, etc., thereby improving operation and disposal efficiency.

[0081] The present invention generates an event stream corresponding to the security product from the security situation data of each security product based on a real-time computing task, and creates a configuration stream for the security product rules based on the real-time computing task, and connects the configuration stream with the event stream by broadcasting the configuration stream, so that the security situation data of the security product in the event stream can be configured based on the security product rules in the configuration stream. The solution of the present invention can meet the requirements of the rule logic of different security products by uniformly configuring at least two security product rules to form a configuration stream, making the deployment of the rule engine more convenient, without the need to input complex statement logic, and can be effectively configured based on the template. At the same time, the present invention configures unified rules to the security situation data of the security product in each event stream through the configuration stream, and can standardize the format of the security situation data of different security products, solving the problem that the prior art cannot process heterogeneous data from multiple sources.

[0082] In some embodiments, step S2, creating a configuration flow according to at least two security product rules, specifically includes the following steps:

[0083] S21. Obtain a rule configuration template from a real-time analysis database.

[0084] S22: Parse the rule configuration template to obtain the at least two security product rules.

[0085] S23. Inheriting the at least two security product rules through the abstract class in the distributed stream processing engine, and creating the configuration stream.

[0086] Specifically, the real-time analysis database is used as a configuration data source and connected to the distributed stream processing engine (Flink). The real-time analysis database can include various rule templates. Users can customize association rules through unified rule templates. Association rules are divided into filter conditions, mapping logic, time windows, etc. Rule template parsing will extract different conditions and automatically determine which ones need to be executed by windowing and which ones need to be executed by filtering. For example, when the merge rule is executed by windowing, the corresponding window will be opened by default according to the merge time for processing when the task is started.

[0087] The real-time analysis database can be Druid. That is, the database connection pool is managed in conjunction with Druid, and the configuration stream is created by inheriting RichSourceFunction.

[0088] The present invention combines distributed stream processing and real-time analysis database to create a configuration stream and implement the configuration of the rule engine.

[0089] In some embodiments, step S2 includes not only the above steps S21 to S23, but also step S24: based on the refresh cycle, dynamically refresh the rule configuration template in the real-time analysis database. Correspondingly, the connection between the configuration stream and the event stream also needs to be reconnected based on the refresh cycle. This setting enables the created configuration stream to be refreshed in real time according to actual needs, so that the rule configuration information can be dynamically effective according to the changes in the rule configuration template, thereby making it more flexible when processing the security situation data of security products and meeting the needs of more rule logic.

[0090] In the present invention, real-time data processing mainly implements the configured rules and completes real-time data processing, relying on the Flink distributed computing engine to meet the requirements of high concurrency and large-scale data processing.

[0091] Figure 2 FIG. 1 is a flowchart of another method for processing multi-source heterogeneous data based on security situation provided in an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0092] S10, start the task, that is, start processing the multi-source heterogeneous data, which includes security situation data of at least two security products.

[0093] S20, judging whether to associate configuration. That is, whether it is necessary to configure the multi-source heterogeneous data according to the configuration information when processing the multi-source heterogeneous data. If it is not necessary to configure the multi-source heterogeneous data according to the configuration information when processing the multi-source heterogeneous data, the following step S80 is executed.

[0094] S80: Processing data, that is, processing the multi-source heterogeneous data.

[0095] In step S20, if it is necessary to configure the multi-source heterogeneous data according to the configuration information when processing the multi-source heterogeneous data, the following steps S30 to S80 are executed.

[0096] S30, add configuration data source, that is, inherit the RichSourceFunction class in the distributed stream processing engine and rewrite the run and open methods.

[0097] S40, connect to the configuration library. That is, connect to the real-time analysis database in the run method and close the connection after completion.

[0098] S50, start real-time refresh, that is, recreate the connection data real-time analysis library every period of time (for example, five minutes) and refresh the broadcast data.

[0099] S60, broadcast the configuration stream, that is, call the broadcast method to broadcast the configuration stream to each taskmanager.

[0100] S70, connect. That is, the event stream and the configuration stream call connect to connect, and then the event stream can obtain the configuration information in the configuration stream.

[0101] S80, process data. Since the connection is BroadcastConnectedStream, the subsequent map, filter and other methods need to inherit BroadcastProcessFunction for secondary implementation.

[0102] In some embodiments, the security product rules include at least one of the following: field mapping rules, data merging rules, data filtering rules and sampling rules; wherein the field mapping rules include at least one of one-to-one mapping, combined dictionary mapping and combined configuration mapping.

[0103] Specifically, the field mapping rule refers to personalized mapping according to the configuration in the configuration flow. The default is direct field mapping, and the mapping rule can also be specified. For example, A is mapped to B, A combined with B is mapped to C, A combined with third-party logic is mapped to B, etc., and the present invention does not limit this. The data filtering rule can be field filtering, which means filtering data according to the specified field. It can be direct filtering, regular filtering, or other filtering methods. The present invention does not limit this. The data merging rule refers to grouping the data according to the specified field, and then sending a message within a configurable time, and recording the total number of merged items under each group.

[0104] In some embodiments, the security product rule includes a sampling rule. Step S4, reading the at least two security product rules in the configuration stream, and processing the security situation data of the security products in each event stream connected to the configuration stream according to the at least two security product rules to obtain the processed security situation data of each security product, specifically includes the following steps:

[0105] S41. Read the sampling rules in the configuration flow.

[0106] S42: Determine, according to the sampling rule, a sampling ratio and a sampling time of the security situation data of the security product in each of the event streams connected to the configuration stream.

[0107] S43. According to the sampling time of the security situation data of each security product in the event stream, a sampling window is opened to cache the security situation data of the security product until the proportion of the cached security situation data reaches the sampling proportion of the security situation data of the security product, and the sampling window is closed to obtain the processed security situation data of each security product.

[0108] Specifically, by reading the sampling rules in the configuration flow, the key information in the sampling rules is extracted, and according to the key information, it can be determined whether it is necessary to sample and collect the security situation data of the security product in the event flow connected to the configuration flow and the sampling conditions. Among them, the sampling conditions include sampling time and sampling ratio.

[0109] The embodiment of the present invention can ensure that the security situation data of each security product after sampling has the same type of sampled data by sampling the security situation data of each security product according to a unified sampling rule.

[0110] Figure 3 A flow chart of a method for implementing a sampling rule provided by an embodiment of the present invention. Figure 3 As shown, the method specifically comprises the following steps:

[0111] S401, template parsing. That is, parsing association rules by reading configuration information in the configuration stream, and extracting key information such as sampling rules.

[0112] S402, determine whether sampling needs to be enabled. The core judgment condition is whether the sampling ratio in the sampling rule is less than 100%. If the sampling ratio in the sampling rule is less than 100%, and sampling needs to be enabled, execute the following steps S403 to S405; if the sampling ratio in the sampling rule is equal to 100%, it means that sampling does not need to be enabled, and directly execute the following step S406.

[0113] S403, opening a sampling window according to the sampling time. The core judgment condition for opening the sampling window is: according to the sampling time in the sampling rule. That is, the opening and closing of the sampling window is controlled by the sampling time.

[0114] S404, the sampled data is in the cache state. That is, according to the sampling ratio in the sampling rule, the number of sampled data is controlled, and a batch of sampled data is submitted for every 100 data (or a predetermined number of data, which can be flexibly set according to the demand).

[0115] S405, determine whether the sampling ratio is reached. Specifically, during the sampling process, the sampling status is recorded in real time, wherein the sampling status refers to the number of samplings at the current moment. Each time data sampling is completed, the current number of samplings is updated in real time, and whether the sampling ratio is met is determined based on the number of samplings at the current moment. If the sampling ratio has not been reached, return to step S404 to continue caching the sampled data until the sampling ratio is met, the sampling window is opened and the sampling window is closed, and step S406 is executed.

[0116] S406: Data output.

[0117] The embodiment of the present invention uses window execution according to the sampling rules. When the task is started, the corresponding sampling window will be opened by default according to the sampling time for processing, so that data processing is more efficient.

[0118] In some embodiments, the security product rule includes a merge rule. Step S4, reading the at least two security product rules in the configuration flow, and processing the security situation data of the security products in each event flow connected to the configuration flow according to the at least two security product rules, to obtain the processed security situation data of each security product, specifically comprising the following steps:

[0119] S51. Read the merging rules in the configuration flow.

[0120] S52: Determine, according to the merging rule, a merging field and a merging period for the security situation data of the security product in each of the event streams connected to the configuration stream.

[0121] S53: grouping the security situation data of the security products in each event stream according to the merging field of the security situation data of the security products in the event stream and according to the merging period of the security situation data to obtain a grouping result.

[0122] S54: Based on the grouping result of the security situation data of each security product in the event stream, obtain the processed security situation data of each security product.

[0123] Specifically, the merge field may be one or more. When there are multiple merge fields, the present invention can merge the security situation data of the security product in layers. The merging period refers to the time period corresponding to the security situation data of the security product, that is, the security situation data of which time period is selected for merging processing.

[0124] For example, if the merging period is within the past two months, and the merging fields are identity information and business type information, then the process of processing the security situation data according to the merging rules specifically includes: from the security situation data within the past two months, first group them according to the identity information to obtain the first-level grouping, and then, based on the first-level grouping, each first-level grouping is grouped according to the business type. Then, one or more data are selected from the first-level grouping and the second-level grouping for subsequent processing, for example, for submission to the security operation platform for data sharing and risk warning.

[0125] It should be noted that when there are multiple fields to be merged, the merging order of the multiple fields is not limited, and of course there can be multiple fields.

[0126] The present invention groups the security situation data of each security product according to a unified merging rule, and only one or more pieces of data can be selected from each group of data for subsequent processing, which can reduce the massive duplication of data and truly focus on valuable events.

[0127] In some embodiments, step S4 includes not only the above steps S51 to S54, but also the following steps:

[0128] S55. According to the merging field of the security situation data of the security product in each event stream, determine whether the current group in the security situation data of the security product in the event stream has been merged in the current merging cycle.

[0129] S56. If the security situation data of the security product in the event stream has been merged in the current merging cycle for the current group, an alarm corresponding to the current merging cycle is generated, the alarms corresponding to the current merging cycle are merged, and the merging counter of the current group is updated.

[0130] Specifically, the security situation data of the security product in the event stream can be used as the current group, and the group obtained after the security situation data of the security product in the event stream is merged according to the one or more merge fields can also be used as the current group. Wherein, when there are multiple merge fields, the data after the security situation data of the security product in the event stream is grouped according to at least part of the multiple merge fields can also be used as the current group. The current group has been merged in the current merging cycle, including the current group being merged or the merging is completed in the current merging cycle. Wherein, the alarm corresponding to the current merging cycle refers to the information that the current group has been merged in the current merging cycle. The purpose of the present invention to merge the alarms corresponding to the current merging cycle is to prevent repeated merging operations on the current group of the current merging cycle.

[0131] Figure 4 A flow chart of a method for implementing a merging rule provided by an embodiment of the present invention. Figure 4 As shown, the method specifically comprises the following steps:

[0132] S501, consuming upstream security product data. Specifically, consuming security product data in the upstream data source one by one.

[0133] S502: Determine whether merging is enabled. Specifically, determine whether the merging function is enabled according to the security product rules in the configuration flow. If merging is not enabled, execute the following step S505; if merging is enabled, execute the following step S503.

[0134] S503, determine whether it is in the merging cycle and whether the current group has been merged. Specifically, first determine whether the security situation data of the security product is data in the merging cycle, and then determine whether the data is being merged or whether the merging has been completed. If the security situation data of the security product is in the merging cycle and the current data has been merged, then execute step S504; if the security situation data of the security product is not in the merging cycle, or is in the merging cycle but has not been merged, then execute step S505.

[0135] S504: Update the merge counter. The merge counter is used to record the number of merged data. If the current group has been merged in the current merge period, this alarm needs to be merged, and the merge counter of the current group needs to be updated.

[0136] S505, write to downstream. If the current group has not been merged in the merging cycle, or the current group is not in the current merging cycle, this alarm does not need to be merged; or the upstream security product data is not merged, and does not need to be merged, then the security situation data is directly written to the downstream for processing by the downstream components.

[0137] In this invention, the main purpose of merging is to prevent the same event data from being too much, causing operators and downstream processing components to receive a large amount of information, increasing reporting and judgment costs. Its core process is to group data according to specified fields, then send only one piece of data information within a configurable merging time, and record the total number of merged items under each group.

[0138] Figure 5 The present invention provides a flowchart of another method for processing multi-source heterogeneous data based on security situation. Figure 5 As shown, the method specifically includes: first, based on Fink, each security product data generates an event stream corresponding to each security product data, and then based on Fink, each security product rule of the rule template in the real-time analysis database generates a unified configuration stream. Among them, the rule template includes but is not limited to sampling rules, field mapping rules, merging rules and filtering rules. After broadcasting the configuration stream in Fink, each event stream is connected to the configuration stream. For each event stream after connecting the configuration stream, the security product data in the event stream can be configured by reading the sampling rules, field mapping rules, field filtering rules and / or data merging rules in the configuration stream, thereby completing the processing of multi-source heterogeneous data based on the security situation, and finally outputting the processed data.

[0139] Based on the security operation platform, the present invention integrates the Flink distributed real-time computing engine, MySQL storage, and ElasticSearch search engine technology to realize the processing, filtering, and cleaning of security product logs (security situation data). The implementation method mainly includes two parts: rule engine configuration and real-time data processing.

[0140] The core process of rule engine configuration is to configure different rule templates according to different rule security products, including field mapping rules, field-level data merging rules, and field-level data filtering rules. Field mapping rules include one-to-one mapping, dictionary mapping, configuration mapping and other mapping relationships. Field-level merging mainly controls the merging window by configuring fields and merging time; field-level data filtering mainly configures filtering rules by configuring filtering logical relationships and filtering matching conditions.

[0141] Real-time data processing mainly relies on the characteristics of the Flink program. Each type of security product has a separate Flink Jar task. The program is connected to the existing data source channel in real time, and the Flink task is configured to start. Different security product rules are read in the task according to the configuration information. Since the security product rules are in MySQL, the Flink task opens the MySQL configuration stream and event stream by default. The configuration stream is refreshed every 5 minutes, so the rule configuration information and filter conditions can be dynamically effective according to the changes in the configuration. The window is opened according to the merge time of the merge rule, and the state is stored in the window according to the merge field. It is automatically submitted at the time to complete the merge action.

[0142] The data processing method provided by the present invention can be used for common pedestrian situation reporting, which is more convenient and flexible, can effectively generate reporting data and support effective analysis by operators, and improve operational handling efficiency. The overall advantages are summarized as follows:

[0143] Through the unified rule configuration interface, the deployment of the rule engine is more convenient. There is no need to enter complex statement logic, and effective configuration can be performed based on templates.

[0144] The streaming rules can be configured uniformly to form rules to meet the needs of different rule logics.

[0145] Automatic data merging after configuration can reduce the massive duplication of data and truly focus on valuable events.

[0146] Optimize rule processing performance and make full use of the advantages and strengths of the streaming engine.

[0147] The embodiment of the present invention also provides a multi-source heterogeneous data processing device based on security situation, as described in the following embodiment. Since the principle of solving the problem by the device is similar to the multi-source heterogeneous data processing method based on security situation, the specific implementation details and technical effects of the device can be referred to the implementation of the multi-source heterogeneous data processing method based on security situation, and the repeated parts will not be repeated.

[0148] Figure 6 The structure diagram of a multi-source heterogeneous data processing device based on security situation provided by the present invention. Figure 6 As shown, the processing device 600 includes a generating module 601 , a creating module 602 , a connecting module 603 and a processing module 604 .

[0149] The generation module 601 is used to obtain security situation data of at least two security products, and generate an event stream of a real-time computing task corresponding to each security product based on the security situation data of the security product.

[0150] The creation module 602 is used to create a configuration flow according to at least two security product rules.

[0151] The connection module 603 is used to connect the configuration stream with the event stream of the real-time computing task corresponding to each of the security products after broadcasting the configuration stream.

[0152] The processing module 604 is used to read the at least two security product rules in the configuration flow, and process the security situation data of the security products in each event flow connected to the configuration flow according to the at least two security product rules to obtain the processed security situation data of each security product.

[0153] In some embodiments, the creation module 602 includes an acquisition unit 612 , a parsing unit 622 , and a creation unit 632 .

[0154] The acquisition unit 612 is used to acquire the rule configuration template from the real-time analysis database.

[0155] The parsing unit 622 is used to parse the rule configuration template to obtain the at least two security product rules.

[0156] The creation unit 632 is used to inherit the at least two security product rules through the abstract class in the distributed stream processing engine to create the configuration flow.

[0157] In some embodiments, the creation module 602 includes not only an acquisition unit 612, a parsing unit 622 and a creation unit 632, but also a refresh unit 642, wherein the refresh unit 642 is used to dynamically refresh the rule configuration template in the real-time analysis database based on a refresh cycle.

[0158] In some embodiments, the security product rules include at least one of the following: field mapping rules, data merging rules, data filtering rules and sampling rules; wherein the field mapping rules include at least one of one-to-one mapping, combined dictionary mapping and combined configuration mapping.

[0159] In some embodiments, the processing module 604 includes a reading unit 614 , a determining unit 624 , and a processing unit 634 .

[0160] The reading unit 614 is used to read the sampling rules in the configuration flow.

[0161] The determination unit 624 is used to determine the sampling ratio and sampling time of the security situation data of the security product in each of the event streams connected to the configuration stream according to the sampling rule.

[0162] The processing unit 634 is used to open a sampling window to cache the security situation data of the security product according to the sampling time of the security situation data of each security product in the event stream, until the proportion of the cached security situation data reaches the sampling proportion of the security situation data of the security product, and close the sampling window to obtain the processed security situation data of each security product.

[0163] In some embodiments, the processing module 604 includes not only a reading unit 614 , a determining unit 624 , and a processing unit 634 , but also a grouping unit 644 .

[0164] The reading unit 614 is further configured to read the merging rules in the configuration flow.

[0165] The determination unit 624 is further configured to determine, according to the merging rule, a merging field and a merging period for the security situation data of the security product in each of the event streams connected to the configuration stream.

[0166] The grouping unit 644 is used to group the security situation data of the security products in each event stream according to the merging field of the security situation data of the security products in the event stream and according to the merging period of the security situation data to obtain a grouping result.

[0167] The processing unit 634 is further configured to obtain processed security situation data of each security product based on the grouping result of the security situation data of the security product in each event stream.

[0168] In some embodiments, the processing module 604 includes not only a reading unit 614, a determination unit 624, a processing unit 634 and a grouping unit 644, but also a judgment unit 654 and an updating unit 664. The judgment unit 654 is used to judge whether the current group in the security situation data of the security product in the event stream has been merged in the current merging cycle according to the merging field of the security situation data of the security product in each of the event streams. The updating unit 664 is used to generate an alarm corresponding to the current merging cycle if the current group in the security situation data of the security product in the event stream has been merged in the current merging cycle, merge the alarms corresponding to the current merging cycle, and update the merging counter of the current group.

[0169] In some embodiments, the processing device 600 not only includes a generation module 601, a creation module 602, a connection module 603 and a processing module 604, but also includes a sending module 605, which is used to send the processed security situation data of at least two security products to the security operation platform for sharing.

[0170] It should be noted that the multi-source heterogeneous data processing method and device based on security situation provided in the embodiments of the present invention can be used in the financial field, and can also be used in any technical field other than the financial field. The embodiments of the present invention do not limit the application field of the multi-source heterogeneous data processing method and device based on security situation.

[0171] Figure 7 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device includes: a processor (processor) 501, a memory (memory) 502 and a bus 503.

[0172] The processor 501 and the memory 502 communicate with each other via a bus 503 .

[0173] The processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above method embodiments.

[0174] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multi-source heterogeneous data processing method of the above security situation is implemented.

[0175] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the multi-source heterogeneous data processing method of the above security situation is implemented.

[0176] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0180] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-source heterogeneous data processing method based on security situation, characterized in that: The method comprises: Acquire security situation data of at least two security products, and generate an event stream of a real-time computing task corresponding to each security product based on the security situation data of the security product; Create a configuration flow based on at least two security product rules; After broadcasting the configuration stream, connecting the configuration stream with the event stream of the real-time computing task corresponding to each of the security products; The at least two security product rules in the configuration flow are read, and according to the at least two security product rules, the security situation data of the security products in each event flow connected to the configuration flow are processed to obtain the processed security situation data of each security product.

2. The method according to claim 1, characterized in that The step of creating a configuration flow according to at least two security product rules includes: Obtain the rule configuration template from the real-time analysis database; Parsing the rule configuration template to obtain the at least two security product rules; The configuration flow is created by inheriting the at least two security product rules through an abstract class in a distributed stream processing engine.

3. The method according to claim 2, characterized in that The step of creating a configuration flow according to at least two security product rules further includes: Based on a refresh cycle, the rule configuration template in the real-time analysis database is dynamically refreshed.

4. The method according to claim 1, characterized in that: The security product rules include at least one of the following: field mapping rules, data merging rules, data filtering rules and sampling rules; wherein the field mapping rules include at least one of one-to-one mapping, dictionary-combined mapping and configuration-combined mapping.

5. The method according to claim 1, characterized in that The security product rule includes a sampling rule, and the at least two security product rules in the configuration flow are read, and the security situation data of the security products in each event flow connected to the configuration flow are processed according to the at least two security product rules to obtain the processed security situation data of each security product, including: Reading sampling rules in the configuration stream; Determine, according to the sampling rule, a sampling ratio and a sampling time of the security situation data of the security product in each of the event streams connected to the configuration stream; According to the sampling time of the security situation data of each security product in the event stream, a sampling window is opened to cache the security situation data of the security product until the proportion of the cached security situation data reaches the sampling proportion of the security situation data of the security product, and the sampling window is closed to obtain the processed security situation data of each security product.

6. The method according to claim 1, characterized in that The security product rule includes a merging rule, and the step of reading the at least two security product rules in the configuration flow and processing the security situation data of the security products in each event flow connected to the configuration flow according to the at least two security product rules to obtain the processed security situation data of each security product includes: Reading the merging rules in the configuration stream; Determine, according to the merging rule, the merging fields and merging cycles of the security situation data of the security product in each of the event streams connected to the configuration stream; According to the merging fields of the security situation data of the security products in each of the event streams, and according to the merging period of the security situation data, the security situation data of the security products in the event stream are grouped to obtain a grouping result; Based on the grouping result of the security situation data of each security product in the event stream, the processed security situation data of each security product is obtained.

7. The method according to claim 6, characterized in that The step of reading the at least two security product rules in the configuration flow, and processing the security situation data of the security products in each event flow connected to the configuration flow according to the at least two security product rules to obtain the processed security situation data of each security product, further includes: According to the merging field of the security situation data of the security product in each of the event streams, determining whether the current group in the security situation data of the security product in the event stream has been merged in the current merging cycle; If the security situation data of the security product in the event stream has been merged in the current merging period for the current group, an alarm corresponding to the current merging period is generated, the alarms corresponding to the current merging period are merged, and the merging counter of the current group is updated.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The processed security situation data of at least two security products are sent to the security operation platform for sharing.

9. A multi-source heterogeneous data processing device based on security situation, characterized in that: The device comprises: A generation module, used to obtain security situation data of at least two security products, and generate an event stream of a real-time computing task corresponding to each security product based on the security situation data of the security product; A creation module, configured to create a configuration flow based on at least two security product rules; A connection module, used for connecting the configuration stream with the event stream of the real-time computing task corresponding to each of the security products after broadcasting the configuration stream; A processing module is used to read the at least two security product rules in the configuration flow, and process the security situation data of the security products in each event flow connected to the configuration flow according to the at least two security product rules to obtain the processed security situation data of each security product.

10. The device according to claim 9, characterized in that The creation module includes: An acquisition unit, used to acquire a rule configuration template from a real-time analysis database; A parsing unit, configured to parse the rule configuration template to obtain the at least two security product rules; The creation unit is used to inherit the at least two security product rules through the abstract class in the distributed stream processing engine to create the configuration stream.

11. The device according to claim 10, characterized in that The creation module also includes: The refresh unit is used to dynamically refresh the rule configuration template in the real-time analysis database based on a refresh cycle.

12. The device according to claim 10, characterized in that The security product rules include at least one of the following: field mapping rules, data merging rules, data filtering rules and sampling rules; wherein the field mapping rules include at least one of one-to-one mapping, dictionary-combined mapping and configuration-combined mapping.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images