Alarm data merging processing method and device, electronic equipment and storage medium

Abstract

The application provides an alarm data merging processing method and device, electronic equipment and a storage medium, wherein the alarm data merging processing method comprises the following steps: determining a merging rule of to-be-processed alarm data, wherein the merging rule is configured based on a business requirement and can be adjusted in real time; and performing merging processing on a plurality of to-be-processed alarm data based on the merging rule to obtain a merging result. Through the alarm data merging processing method provided by the application, the merging processing operation can be adaptively adjusted according to the business requirement, and the merging processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an alarm data merging and processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] According to relevant technologies, a large amount of alarm data can be generated in red team vs. blue team scenarios. Such alarm data usually has no practical meaning or analytical value. Therefore, it is necessary to merge and process such alarm data.

[0003] However, in the process of merging alarm data, the merging operation needs to be tied to business requirements. When business requirements change, the merging operation cannot be adaptively adjusted, and it must be implemented by modifying the code logic, which will bring a significant workload to users and affect the efficiency of the merging process. Therefore, finding an alarm data merging method that can adaptively adjust the merging operation according to business requirements has become a research hotspot. Summary of the Invention

[0004] This invention provides an alarm data merging and processing method, apparatus, electronic device, and storage medium, which can adaptively adjust the merging and processing operation according to business needs, thereby improving the merging and processing efficiency.

[0005] This invention provides a method for merging alarm data, the method comprising: determining merging rules for alarm data to be processed, wherein the merging rules are configured based on business requirements and can be adjusted in real time; and merging multiple alarm data to be processed based on the merging rules to obtain a merging result.

[0006] According to an alarm data merging processing method provided by the present invention, before determining the merging rules for the alarm data to be processed, the method further includes: obtaining business requirements; configuring a set of merging rules for merging alarm data based on the business requirements; the determination of the merging rules for the alarm data to be processed specifically includes: determining the merging rules in the merging rule set that match the alarm data to be processed as the merging rules for the alarm data to be processed; or, selecting the merging rule corresponding to the alarm data to be processed from the merging rule set; adjusting the merging rules based on specific business requirements, and determining the adjusted merging rules as the merging rules for the alarm data to be processed.

[0007] According to a method for merging alarm data provided by the present invention, the merging rule includes a first merging rule, wherein the first merging rule is used to instruct the alarm data to be merged according to a merging field in the alarm data to be processed, and the merging field is a field in the alarm data to be processed that represents alarm information; when the merging rule is the first merging rule, the step of merging multiple alarm data to be processed based on the merging rule specifically includes: determining the merging field corresponding to the first merging rule in each of the multiple alarm data to be processed; and merging the multiple alarm data to be processed based on the merging field.

[0008] According to an alarm data merging processing method provided by the present invention, the step of merging multiple alarm data to be processed based on the merging field specifically includes: determining the timestamp corresponding to each alarm data to be processed; determining the merging identifier of each alarm data to be processed based on the timestamp and the merging field; and merging multiple alarm data to be processed based on the merging identifier.

[0009] According to an alarm data merging and processing method provided by the present invention, determining the timestamp corresponding to each alarm data to be processed includes: determining the alarm information generation time corresponding to each alarm data to be processed; and determining the timestamp corresponding to each alarm data to be processed based on the alarm information generation time, wherein the timestamp corresponding to the alarm data to be processed whose alarm information generation time is within a preset time range is the first timestamp, and the timestamp corresponding to the alarm data to be processed whose alarm information generation time is outside the preset time range is the second timestamp.

[0010] According to an alarm data merging processing method provided by the present invention, determining the merging identifier of each of the alarm data to be processed based on the timestamp and the merging field includes: obtaining the merging identifier of each of the alarm data to be processed through encrypted calculation based on the timestamp and the merging field, wherein when the timestamp and the merging field are the same, the merging identifier obtained by encrypted calculation is the same; and performing merging processing on multiple alarm data to be processed based on the merging identifier includes: merging the alarm data to be processed with the same merging identifier among the multiple alarm data to be processed.

[0011] According to an alarm data merging and processing method provided by the present invention, the first merging rule further includes a merging grouping field, which is used to group the alarm data to be processed. The method further includes: determining the merging grouping field based on the first merging rule; grouping multiple alarm data to be processed based on the merging grouping field; and merging the multiple alarm data to be processed based on the merging field specifically includes: merging multiple alarm data to be processed belonging to the same group based on the merging field.

[0012] According to a method for merging alarm data provided by the present invention, the merging rule includes a second merging rule, wherein the second merging rule is used to instruct the merging of the alarm data to be processed that includes a preset field; when the merging rule is the second merging rule, the step of merging multiple alarm data to be processed based on the merging rule specifically includes: determining whether each alarm data to be processed includes the preset field based on the second merging rule; and merging the alarm data to be processed that includes the preset field.

[0013] According to an alarm data merging processing method provided by the present invention, the merging rule further includes a merging result bypass output rule. After obtaining the merging result, the method further includes: controlling the merging result to bypass output based on the merging result bypass output rule.

[0014] According to an alarm data merging processing method provided by the present invention, after obtaining the merging result, the method further includes: registering a timer, wherein the timer is used to determine the update time of the merging processing; and, if the update time of the timer is determined to be reached, performing update merging processing on multiple newly acquired alarm data to be processed and the merging result based on the merging rules.

[0015] According to an alarm data merging processing method provided by the present invention, the step of updating and merging multiple newly acquired alarm data to be processed and the merging result specifically includes: updating and merging multiple newly acquired alarm data to be processed and the merging result based on an asynchronous update mechanism.

[0016] According to an alarm data merging processing method provided by the present invention, the step of updating and merging multiple newly acquired alarm data to be processed and the merging result based on the merging rule specifically includes: if the update and merging processing of multiple newly acquired alarm data to be processed and the merging result based on the merging rule fails, the newly acquired alarm data to be processed that failed to update is saved to a preset queue; according to a preset time interval, the newly acquired alarm data to be processed that failed to update in the preset queue is consumed again, and the newly acquired alarm data to be processed that failed to update and the merging result that were successfully consumed are updated and merged again.

[0017] According to an alarm data merging and processing method provided by the present invention, after consuming the newly acquired alarm data that failed to update in the preset queue again according to a preset time interval, the method further includes: if the newly acquired alarm data that failed to update for a preset number of consecutive consumption failures fails to be consumed, storing the newly acquired alarm data that failed to update for a preset number of consecutive consumption failures into a preset list.

[0018] The present invention also provides an alarm data merging and processing apparatus, the apparatus comprising: a first module for determining merging rules for alarm data to be processed, wherein the merging rules are configured based on business requirements and can be adjusted in real time; and a second module for merging multiple alarm data to be processed based on the merging rules to obtain a merging result.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the alarm data merging processing method as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the alarm data merging processing method as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the alarm data merging processing method as described above.

[0022] The alarm data merging and processing method, apparatus, electronic device, and storage medium provided by this invention determine the merging rules for the alarm data to be processed. The merging rules are configured based on business requirements and can be adjusted in real time to ensure that the program and business can be decoupled during program operation. This allows the merging rules to be configured without restarting the program. Based on the merging rules, multiple alarm data to be processed are merged to obtain the merging result. This enables the merging and processing operation to be adaptively adjusted according to business requirements, thereby improving the merging and processing efficiency. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is one of the flowcharts illustrating the alarm data merging and processing method provided by the present invention;

[0025] Figure 2 This is the second flowchart of the alarm data merging and processing method provided by the present invention;

[0026] Figure 3 This is the third flowchart of the alarm data merging and processing method provided by the present invention;

[0027] Figure 4 This is the fourth flowchart of the alarm data merging and processing method provided by the present invention;

[0028] Figure 5 This is a schematic diagram illustrating an application scenario of the alarm data merging and processing method provided by the present invention;

[0029] Figure 6 This is a schematic diagram of the alarm data merging and processing device provided by the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] According to relevant technologies, red team / blue team exercises can generate a large amount of duplicate and invalid alarm data. During application, the filtering logic for alarm data through merging often changes. This filtering operation is a streaming computation, where streaming merging logic is written based on business requirements. Therefore, when business needs change, the code logic needs to be modified, which increases the merging workload.

[0033] The alarm data merging and processing method provided by this invention can decouple the program from the business, and enable the configuration of merging rules without restarting the program. Based on the configured merging rules, multiple alarm data to be processed are merged to obtain the merging result, thereby enabling the merging and processing operation to be adaptively adjusted according to business needs.

[0034] Figure 1 This is one of the flowcharts illustrating the alarm data merging and processing method provided by the present invention.

[0035] In an exemplary embodiment of the present invention, combined with Figure 1 As can be seen, the alarm data merging and processing method may include steps 110 and 120, which will be described in detail below.

[0036] In step 110, the merging rules for the alarm data to be processed are determined. The merging rules are configured based on business requirements and can be adjusted in real time.

[0037] In step 120, based on the merging rules, multiple alarm data to be processed are merged to obtain the merged result.

[0038] In one embodiment, merging rules can be set in a merging rule table. These rules can be configured according to business needs and adjusted in real time based on changes in those needs. In this embodiment, configuring merging rules based on business needs ensures decoupling between the program and business logic during program execution, allowing merging rules to be configured without restarting the program.

[0039] Furthermore, multiple pending alarm data can be merged based on the configured merging rules to obtain the merged result. This embodiment allows for adaptive adjustment of the merging rules according to business needs, and then merging of the pending alarm data based on these rules. This enables adaptive adjustment of the merging process without modifying the program code logic, thus improving merging efficiency.

[0040] The alarm data merging and processing method provided by this invention determines the merging rules for the alarm data to be processed. The merging rules are configured based on business requirements and can be adjusted in real time to ensure that the program and business can be decoupled during program operation. This allows the merging rules to be configured without restarting the program. Based on the merging rules, multiple alarm data to be processed are merged to obtain the merging result. This enables the merging and processing operation to be adaptively adjusted according to business requirements, thereby improving the merging and processing efficiency.

[0041] In yet another exemplary embodiment of the present invention, continuing with Figure 1 The above embodiment is used as an example for illustration. Before determining the merging rules for the alarm data to be processed (corresponding to step 110), the alarm data merging processing method may also include the following steps:

[0042] Obtain business requirements;

[0043] Configure a set of merging rules for merging alarm data based on business requirements;

[0044] The merging rules for the alarm data to be processed can be determined in the following ways:

[0045] Merge rules that match the alarm data to be processed in the merge rule set are determined as the merge rules for the alarm data to be processed; or,

[0046] Select the merge rule corresponding to the alarm data to be processed from the merge rule set;

[0047] The merging rules are adjusted based on specific business needs, and the adjusted merging rules are determined as the merging rules for alarm data to be processed.

[0048] In one embodiment, the merge rule set may include multiple merge rules configured according to business needs. During application, the merge rule in the merge rule set that matches the alarm data to be processed can be used as the merge rule for the alarm data to be processed.

[0049] In one example, a user can issue a processing command for pending alarm data. This command includes a specific business requirement. During application, based on this business requirement, a merging rule matching the requirement can be selected from the merging rule set and used as the merging rule for the pending alarm data. It should be noted that the matching degree between the merging rule and the pending alarm data must meet a matching degree threshold.

[0050] In another embodiment, a merging rule corresponding to the alarm data to be processed can be selected from the merging rule set. The merging rule corresponding to the alarm data to be processed can be the merging rule with the highest matching degree to the alarm data to be processed. Here, "highest matching degree" can be understood as the merging rule with the highest matching degree in the merging rule set, even if it does not reach the set matching degree threshold.

[0051] Furthermore, the aforementioned merging rules can be adjusted based on specific business needs, and the adjusted merging rules can be used as the merging rules for the alarm data to be processed. It is understood that the specific merging rules are not configured based on routine business needs or historical business needs, but rather on the personalized needs of a specific customer or newly generated business needs. Therefore, there is no merging rule in the merging rule set corresponding to a specific business need. In application, the aforementioned merging rules can be adjusted based on specific business needs to make the adjusted merging rules correspond to the specific business needs, and the adjusted merging rules can be used as the merging rules for the alarm data to be processed. Through this embodiment, merging rules that meet any business needs can be configured, thereby laying the foundation for the merging processing of alarm data to be processed.

[0052] It should be noted that the merge rule configuration mainly involves defining the merge decision rules, which can include Identity merge rules (corresponding to the first merge rule) and rule merge rules (corresponding to the second merge rule). The following will combine... Figure 2 The first merge rule is explained.

[0053] Figure 2 This is the second flowchart of the alarm data merging and processing method provided by the present invention.

[0054] In an exemplary embodiment of the present invention, the merging rule may include a first merging rule, wherein the first merging rule is used to instruct the alarm data to be processed to be merged according to the merging field in the alarm data to be processed, and the merging field is the field in the alarm data to be processed that represents alarm information.

[0055] Combination Figure 2 As can be seen, when the merging rule is the first merging rule, the alarm data merging processing method may include steps 210 to 230. Step 210 is the same as or similar to step 110. For its specific implementation and beneficial effects, please refer to the previous description. It will not be repeated in this embodiment. Steps 220 and 230 will be introduced below.

[0056] In step 220, the merge field corresponding to the first merge rule is determined from the multiple alarm data to be processed.

[0057] In step 230, multiple alarm data to be processed are merged based on the merge field.

[0058] In one example, the alarm data to be processed can be streaming data from streaming computation.

[0059] In one embodiment, merging based on the merging field corresponding to the first merging rule can more accurately merge the alarm data to be processed. In one example, the merging field may include fields representing attackers, fields representing victims, fields representing attack types, etc.

[0060] In one embodiment, alarm data to be processed can be merged based on multiple merging fields corresponding to the first merging rule. In this embodiment, the first merging rule is determined according to business requirements, and consequently, the merging fields are also determined based on business requirements, thereby achieving adaptive adjustment of the merging processing operation according to business needs and improving merging processing efficiency.

[0061] In another embodiment, a merge item configuration can also be set. This configuration primarily defines how different merge fields are processed when stream data is merged. The merge processing methods can include overlay merging, append merging, and cumulative merging. Overlay merging can be applied to fields representing the time of the last attack; append merging can be applied to fields representing victims in a red-blue team merging scenario; and cumulative merging can be applied to fields representing the total number of attacks.

[0062] In yet another exemplary embodiment of the present invention, continuing with Figure 2 The above embodiment is used as an example to illustrate that, based on the merging field, merging multiple pending alarm data can be achieved in the following way:

[0063] Determine the timestamp corresponding to each pending alarm data;

[0064] Based on the timestamp and the merge field, determine the merge identifier for each pending alarm data;

[0065] Multiple pending alarm data are merged based on the merge identifier.

[0066] In yet another exemplary embodiment of the present invention, the determination of the timestamp corresponding to each alarm data to be processed, as described above, can be achieved in the following manner:

[0067] Determine the alarm information generation time corresponding to each pending alarm data;

[0068] Based on the alarm information generation time, the timestamp corresponding to each alarm data to be processed is determined. The timestamp corresponding to the alarm data to be processed whose alarm information generation time is within a preset time range is the first timestamp, and the timestamp corresponding to the alarm data to be processed whose alarm information generation time is outside the preset time range is the second timestamp.

[0069] In one embodiment, the alarm information can be information that characterizes alarm data, such as information characterizing the attacker, the attack type, or the attack method. During application, the alarm information generation time corresponding to each piece of alarm data to be processed can be determined. Further, based on the alarm information generation time, a timestamp corresponding to each piece of alarm data to be processed is determined. The timestamp corresponding to the alarm data whose alarm information generation time falls within a preset time range is the first timestamp, and the timestamp corresponding to the alarm data whose alarm information generation time falls outside the preset time range is the second timestamp. In one example, the preset time range can be adjusted according to actual conditions and is not specifically limited in this embodiment; for example, it can be daily, hourly, or a specific time period.

[0070] In another embodiment, a merge identifier (also known as an identity) for each pending alarm data can be determined based on a timestamp and a merge field, and multiple pending alarm data can be merged based on the merge identifier. During application, pending alarm data with the same merge identifier can be merged.

[0071] In yet another exemplary embodiment of the present invention, the merging identifier of each alarm data to be processed is determined based on the timestamp and the merging field, which can be achieved in the following way:

[0072] Based on the timestamp and the merge field, the merge identifier of each alarm data to be processed is obtained through encrypted calculation. When the timestamp and the merge field are the same, the merge identifier obtained by encrypted calculation is the same.

[0073] Furthermore, merging multiple pending alarm data based on the merging identifier can be achieved in the following way:

[0074] Multiple pending alarm data with the same merging identifier are merged.

[0075] In another embodiment, the merge identifier can also be calculated using a configured encryption method by combining the timestamp and the merge field value. In this embodiment, the merge identifier is obtained by encrypting the merge field and the timestamp, thereby ensuring the privacy of the merge field in the alarm data to be processed.

[0076] Figure 3This is the third flowchart of the alarm data merging and processing method provided by the present invention.

[0077] To further introduce the alarm data merging and processing method provided by this invention, the following will be combined with... Figure 3 Please provide an explanation.

[0078] In an exemplary embodiment of the present invention, the first merging rule may further include a merging grouping field, which is used to group the alarm data to be processed. Figure 3 As can be seen, the alarm data merging and processing method may include steps 310 to 330, and each step will be described below.

[0079] In step 310, the merge grouping field is determined based on the first merge rule.

[0080] In one embodiment, a merge keyby (also known as grouping) configuration can also be performed based on a first merge rule. The merge grouping configuration is primarily used to define which fields (corresponding to merge grouping fields) are used to group the streaming data (corresponding to the alarm data to be processed). The merge grouping field can be included in the merge field; otherwise, a logical error will occur. In one example, for a red-blue team scenario, the grouping field is a field used to characterize the alarm type.

[0081] In step 320, multiple alarm data to be processed are grouped based on the merge grouping field.

[0082] In step 330, multiple pending alarm data belonging to the same group are merged based on the merging field.

[0083] In one embodiment, multiple pending alarm data can be grouped based on a merge grouping field. It is understood that pending alarm data belonging to the same group may not necessarily be grouped together, and pending alarm data not belonging to the same group definitely cannot be grouped together. Therefore, in this embodiment, multiple pending alarm data are grouped based on a merge grouping field, and further, multiple pending alarm data belonging to the same group are merged based on multiple merge fields. In this embodiment, pending alarm data not belonging to the same group are no longer merged, thereby speeding up the merging process.

[0084] The following will combine Figure 4 The merge rule is explained as the second merge rule.

[0085] Figure 4 This is the fourth flowchart of the alarm data merging and processing method provided by the present invention.

[0086] In an exemplary embodiment of the present invention, the merging rule may include a second merging rule, wherein the second merging rule is used to instruct the merging of alarm data to be processed, including preset fields.

[0087] Combination Figure 4 As can be seen, when the merging rule is the second merging rule, the alarm data merging processing method can include steps 410 and 420, which will be described in detail below.

[0088] In step 410, based on the second merging rule, it is determined whether each alarm data to be processed includes a preset field.

[0089] In step 420, the alarm data to be processed, including preset fields, are merged.

[0090] The preset fields are determined based on user requirements. In one example, a preset field could be used to indicate that the alarm type is a weak password, or it could be used to indicate that the HTTP response contains a specified keyword.

[0091] In one embodiment, a second merging rule can be used to determine whether each piece of alarm data to be processed includes a preset field. When it is determined that the alarm data to be processed includes the preset field, the alarm data to be processed that includes the preset field can be merged. In this embodiment, the second merging rule is determined according to business requirements. By using the second merging rule to determine whether each piece of alarm data to be processed includes the preset field, and then merging the alarm data to be processed that includes the preset field, the merging operation can be adaptively adjusted according to business requirements, thereby improving the efficiency of the merging operation.

[0092] In yet another exemplary embodiment of the present invention, the merging rule may further include a merging result bypass output rule. Continuing with... Figure 1 The above embodiment is used as an example for illustration. After obtaining the merging result (corresponding to step 120), the alarm data merging processing method may further include the following steps:

[0093] Based on the bypass output rules of the merge results, the merge results are controlled to be output in a bypass manner.

[0094] In one embodiment, the merge result can be controlled to be output in a bypass output manner according to the bypass output rule of the merge result.

[0095] In another embodiment, the merge result can be the data result obtained from the initial calculation. During application, a bypass output can be used to persist the merge result and performance to disk. This embodiment avoids affecting the streaming data context of Flink operators, thus preventing data accumulation due to the merging characteristics and ensuring smooth downstream output.

[0096] In yet another exemplary embodiment of the present invention, continuing with Figure 1 The above embodiment is used as an example for illustration. After obtaining the merging result (corresponding to step 120), the alarm data merging processing method may further include the following steps:

[0097] Register a timer, where the timer is used to determine the update time for the merge process;

[0098] Once the update timer is determined, the newly acquired pending alarm data and the merge results are updated and merged based on the merging rules.

[0099] In one embodiment, after the merge result is successfully output via bypass, a Flink Timer is registered. This timer determines the update time for the merge process. During application, the timer starts counting down after instance initialization. At the preset update time, an update operation is triggered. When the update begins, the data to be updated (corresponding to newly acquired alarm data) is grouped. In one example, every 1000 data items are grouped for updating. After the update is triggered, the cache corresponding to this group is cleared. If there are fewer than 1000 data items to be updated, another update is triggered to update the scattered data. When there are zero data items to be updated, a long-interval trigger is registered again to prepare for the update. This prevents a small amount of data from still flowing into the cache after the update is completed without triggering an update. Once no more data flows in, there will be no new triggers, and this data will remain in memory and cannot be updated. After the cache expires, this data will be permanently lost. This embodiment enables batch updates of data in the cache, ensuring that all data in the cache is updated without omissions.

[0100] In another exemplary embodiment of the present invention, updating and merging multiple newly acquired alarm data to be processed and merging results can be achieved in the following manner:

[0101] Based on the asynchronous update mechanism, multiple newly acquired alarm data to be processed and the merging results are updated and merged.

[0102] In one embodiment, when an update is triggered, an asynchronous updater is invoked to persist the data to disk. The asynchronous updater persists the updated data to the data source passed in when the update was triggered. The asynchronous update can be implemented using Java's Executor thread pool with a feature thread pattern. The thread pool is a fixed-size blocking thread pool, and the rejection policy is CallerRunPolicy.

[0103] In this invention, the fault tolerance and program flexibility of merging or other big data update processing operations are improved by using configurable output sources (corresponding to merge results).

[0104] In another exemplary embodiment of the present invention, updating and merging multiple newly acquired alarm data to be processed and the merging results based on merging rules can be achieved in the following manner:

[0105] If the merging process fails to update multiple newly acquired alarm data to be processed and the merging result based on the merging rules, the newly acquired alarm data to be processed that failed to be updated will be saved to a preset queue.

[0106] According to the preset time interval, newly acquired alarm data that failed to update in the preset queue is consumed again, and the newly acquired alarm data that failed to update and the merged result are updated and merged.

[0107] In one embodiment, the preset queue can be a Kafka queue, also known as an error queue. During application, if the update is successful, a result is returned; if the update fails, the failed data, along with update-related parameters, data source information, etc. (corresponding to the number of newly acquired pending alarms), can be stored in the Kafka queue (hereinafter referred to as the error queue). Additionally, a patrol thread can be configured to cyclically execute at preset time intervals, consuming the contents of the error queue. For example, each loop could have a 1-second interval to prevent excessively rapid retries. Consuming the contents of the error queue can be understood as retrieving newly acquired pending alarms from the error queue that have failed updates and merging these alarms. If consumption fails, the current execution ends; if consumption is successful, the data is updated again.

[0108] In another example, synchronous updates can be invoked to reduce the pressure on asynchronous update threads, and execution ends upon successful update. Here, "consumption failure" refers to the failure to merge and process newly acquired pending alarms that failed to update; "consumption success" refers to the successful merging and processing of newly acquired pending alarms that failed to update.

[0109] In yet another exemplary embodiment of the present invention, continuing with the aforementioned embodiments as an example, after consuming newly acquired alarm data that failed to update in the preset queue again according to a preset time interval, the alarm data merging and processing method may further include:

[0110] If the newly acquired pending alarm data fails to be processed for a preset number of consecutive times due to failed updates, the newly acquired pending alarm data that failed to be processed for a preset number of consecutive times will be stored in a preset list.

[0111] In one embodiment, after an update fails, the system can check whether the number of errors in acquiring newly acquired pending alarm data (corresponding to a preset number of consecutive failed consumptions) has reached a threshold. If the threshold is reached, the newly acquired pending alarm data that failed to be consumed for the preset number of consecutive times is stored in a preset list (also known as a failure table). If the threshold is not reached, it is put back into the error queue. In this embodiment, storing the newly acquired pending alarm data that failed to be consumed for the preset number of consecutive times in the failure table indicates that this data requires manual intervention and correction before it can be successfully persisted to disk, thus avoiding the waste of system resources through endless retries.

[0112] It should be noted that the present invention ensures the eventual consistency of system data through an error compensation mechanism.

[0113] Figure 5 This is a schematic diagram illustrating an application scenario of the alarm data merging and processing method provided by the present invention.

[0114] To further introduce the alarm data merging and processing method provided by this invention, the following will be combined with... Figure 5 Please provide an explanation.

[0115] In an exemplary embodiment of the present invention, combined with Figure 5 It can be seen that, Figure 5 The diagram illustrates the collaboration process of the merging process. The alarm data merging processing method can be implemented through a merging program, which may include the following components: a merging rule configuration table, a pre-processor, a merging cache management system, a merging main program, a merging identity policy group, a merging merge item processor, a bypass update, and a compensation mechanism.

[0116] The merge rule table may include the following parts:

[0117] 1) Merge keyby (grouping) configuration: This mainly defines which input fields are used to group streaming data. The keyby field must be included in the merge rule fields; otherwise, a logical error will occur. In one example, in a red-blue team adversarial merge scenario, the keyby field can be the alarm type.

[0118] 2) Merge Identity Generation Configuration: This mainly defines which fields are used to generate the merge identity and what encryption method is employed. The merge identity generation configuration must include a `keyby` field. The merge identity is the basis for determining whether data needs to be merged; if two data entries have the same merge identity, it proves that the data needs to be merged. In one example, in a red-blue team reconnaissance merge requirement, the fields in the merge identity generation configuration could be attacker, attack type, attack method, etc.

[0119] 3) Merge cache configuration, which mainly defines which Flink backend will support the merge cache, such as heap memory, RocksDB, etc., as well as the cache's TTL lifecycle, including cache eviction policy, expiration time, etc.

[0120] 4) Merge Item Configuration: This mainly defines which fields should be merged and how when streaming data is merged. The available merging methods are: Overwrite, for example, to save the last attack time; Append, to save all victims in red team vs. blue team merging scenarios; and Accumulate, for example, to save the total number of attacks.

[0121] 5) Merge rule configuration, which mainly defines the merge decision rules. In one example, the merge rules can be of the following two types: Identity merge (corresponding to the first merge rule) and rule merge (corresponding to the second merge rule).

[0122] 6) Output configuration, which mainly defines the bypass output data source, error queue related configuration, compensation mechanism retry count, etc.

[0123] In another example, the front-end processor is primarily responsible for two tasks: first, reading the merge keyby configuration to distribute data, where data not in the same stream will never be merged; second, reading the merge cache configuration to initialize the cache. This can be used for alarm types in red-blue team / blue team and weak password services.

[0124] In another example, merge cache management primarily involves configuring merge rules to determine the Flink backend type for the merge cache and initializing the TTL strategy. The main merge program is mainly responsible for loading the merge rule table, scheduling the Identity generation strategy, and determining whether data needs to be merged. Once the data to be merged is identified, the merge item management function is invoked to generate the merged item cache. The merge rules involved in the main program can be dynamically changed at runtime.

[0125] In another example, the Identity Merge strategy group is primarily responsible for generating merged identities based on the configured strategy. The execution flow is as follows: it accepts the fields required for identity generation, calculates the merge period timestamp, selects the encryption method for identity generation, and calls the strategy to generate the identity. The merge rules involved in the identity generation strategy can be dynamically changed at runtime.

[0126] In another example, the merge item processor is mainly responsible for generating merge items according to the configuration. It can customize the merge fields and the field merging method, and supports three methods by default: append, overwrite, and cumulative merging.

[0127] In another embodiment, input data source A-input data source C can correspond to the previously mentioned alarm data to be processed. During application, the merging rules for the alarm data to be processed can be configured in real time based on user instructions. In one example, merging rules can be set through a configured merging rule table. Furthermore, multiple alarm data to be processed can be merged based on the configured merging rules to obtain the merged result (corresponding to the output data source). Through this embodiment, the merging rules can be adaptively adjusted according to business needs, and then the alarm data to be processed can be merged based on the merging rules. This allows for adaptive adjustment of the merging operation without modifying the program code logic, improving the efficiency of the merging process.

[0128] It should be noted that, based on the configured merging rules, the merging of multiple pending alarm data can also include the embodiments described above regarding merging multiple pending alarm data based on the first merging rule and merging multiple pending alarm data based on the second merging rule.

[0129] In another embodiment, the merge result can be controlled to be output in a bypass output manner based on a bypass output and asynchronous update system. In yet another embodiment, the merge result can be the data result obtained from the initial calculation. During application, bypass output can be used to persist the merge result and performance to disk. Through this embodiment, the streaming data context of Flink operators can be maintained, thus preventing data accumulation due to the characteristics of merging and avoiding impact on downstream output.

[0130] In another embodiment, after the merge result is successfully output via bypass, a FlinkTimer is registered. This timer can be used to determine the update time for the merge process. During application, the timer starts counting down after instance initialization. After the preset update time, an update operation is triggered. When the update begins, the data to be updated (corresponding to newly acquired alarm data to be processed) is grouped.

[0131] In another embodiment, when an update is triggered, an asynchronous updater is invoked to persist the data to disk. The asynchronous updater then persists the updated data to the data source passed in when the update was triggered.

[0132] As described above, the alarm data merging and processing method provided by this invention decouples the program from the business through rule configuration and broadcast stream operator updates; and enables configuration modifications to take effect immediately without restarting the program through hot configuration updates. This allows for adaptive adjustment of the merging and processing operation according to business needs, improving merging and processing efficiency.

[0133] Based on the same concept, the present invention also provides an alarm data merging and processing device.

[0134] The alarm data merging and processing apparatus provided by the present invention will be described below. The alarm data merging and processing apparatus described below and the alarm data merging and processing method described above can be referred to in correspondence.

[0135] Figure 6 This is a schematic diagram of the alarm data merging and processing device provided by the present invention.

[0136] In an exemplary embodiment of the present invention, combined with Figure 6 As can be seen, the alarm data merging and processing device may include a first module 610 and a second module 620, and each module will be described below.

[0137] The first module 610 can be configured to determine the merging rules for the alarm data to be processed. The merging rules are configured based on business requirements and can be adjusted in real time. The second module 620 can be configured to merge multiple alarm data to be processed based on the merging rules to obtain the merged result.

[0138] In an exemplary embodiment of the present invention, the first module 610 may further be configured to:

[0139] Obtain business requirements;

[0140] Configure a set of merging rules for merging alarm data based on business requirements;

[0141] The first module 610 can determine the merging rules for the alarm data to be processed in the following way:

[0142] Merge rules that match the alarm data to be processed in the merge rule set are determined as the merge rules for the alarm data to be processed; or,

[0143] Select the merge rule corresponding to the alarm data to be processed from the merge rule set;

[0144] The merging rules are adjusted based on specific business needs, and the adjusted merging rules are determined as the merging rules for alarm data to be processed.

[0145] In an exemplary embodiment of the present invention, the merging rule may include a first merging rule, wherein the first merging rule is used to instruct the alarm data to be processed to be merged according to the merging field in the alarm data to be processed, and the merging field is the field in the alarm data to be processed that represents alarm information.

[0146] When the merging rule is the first merging rule, the second module 620 can perform merging processing on multiple pending alarm data based on the merging rule in the following way:

[0147] In multiple pending alarm data, determine the merging field corresponding to the first merging rule;

[0148] Based on the merge field, multiple pending alarm data are merged.

[0149] In an exemplary embodiment of the present invention, the second module 620 can perform merging processing on multiple pending alarm data based on the merging field in the following manner:

[0150] Determine the timestamp corresponding to each pending alarm data;

[0151] Based on the timestamp and the merge field, determine the merge identifier for each pending alarm data;

[0152] Multiple pending alarm data are merged based on the merge identifier.

[0153] In an exemplary embodiment of the present invention, the second module 620 may determine the timestamp corresponding to each alarm data to be processed in the following manner:

[0154] Determine the alarm information generation time corresponding to each pending alarm data;

[0155] Based on the alarm information generation time, the timestamp corresponding to each alarm data to be processed is determined. The timestamp corresponding to the alarm data to be processed whose alarm information generation time is within a preset time range is the first timestamp, and the timestamp corresponding to the alarm data to be processed whose alarm information generation time is outside the preset time range is the second timestamp.

[0156] In an exemplary embodiment of the present invention, the second module 620 can determine the merge identifier of each alarm data to be processed based on the timestamp and the merge field in the following manner:

[0157] Based on the timestamp and the merge field, the merge identifier of each alarm data to be processed is obtained through encrypted calculation. When the timestamp and the merge field are the same, the merge identifier obtained by encrypted calculation is the same.

[0158] The second module 620 can perform merging processing on multiple pending alarm data based on the merging identifier in the following way:

[0159] Multiple pending alarm data with the same merging identifier are merged.

[0160] In an exemplary embodiment of the present invention, the first merging rule further includes a merging grouping field, which is used to group the alarm data to be processed. The second module 620 can also be configured to:

[0161] Based on the first merge rule, determine the merge grouping field;

[0162] Based on the merge grouping field, multiple pending alarm data are grouped.

[0163] The second module 620 can perform merging processing on multiple pending alarm data based on the merging field in the following way:

[0164] Based on the merge field, multiple pending alarm data belonging to the same group are merged.

[0165] In an exemplary embodiment of the present invention, the merging rule may include a second merging rule, wherein the second merging rule is used to instruct the merging of pending alarm data including preset fields; when the merging rule is the second merging rule, the second module 620 may perform merging processing on multiple pending alarm data based on the merging rule in the following manner:

[0166] Based on the second merging rule, determine whether each alarm data to be processed includes a preset field;

[0167] The pending alarm data, including preset fields, will be merged.

[0168] In an exemplary embodiment of the present invention, the second module 620 may further be configured to:

[0169] Based on the bypass output rules of the merge results, the merge results are controlled to be output in a bypass manner.

[0170] In an exemplary embodiment of the present invention, the second module 620 may further be configured to:

[0171] Register a timer, where the timer is used to determine the update time for the merge process;

[0172] Once the update timer is determined, the newly acquired pending alarm data and the merge results are updated and merged based on the merging rules.

[0173] In an exemplary embodiment of the present invention, the second module 620 may perform update and merge processing on multiple newly acquired alarm data to be processed and the merge result in the following manner:

[0174] Based on the asynchronous update mechanism, multiple newly acquired alarm data to be processed and the merging results are updated and merged.

[0175] In an exemplary embodiment of the present invention, the second module 620 may perform update and merge processing on multiple newly acquired alarm data to be processed and merge results based on merge rules in the following manner:

[0176] If the merging process fails to update multiple newly acquired alarm data to be processed and the merging result based on the merging rules, the newly acquired alarm data to be processed that failed to be updated will be saved to a preset queue.

[0177] According to the preset time interval, newly acquired alarm data that failed to update in the preset queue is consumed again, and the newly acquired alarm data that failed to update and the merged result are updated and merged.

[0178] In an exemplary embodiment of the present invention, the second module 620 may further be configured to:

[0179] If the newly acquired pending alarm data fails to be processed for a preset number of consecutive times due to failed updates, the newly acquired pending alarm data that failed to be processed for a preset number of consecutive times will be stored in a preset list.

[0180] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can invoke logical instructions in the memory 730 to execute an alarm data merging processing method. This method includes: determining merging rules for the alarm data to be processed, wherein the merging rules are configured based on business requirements and can be adjusted in real time; and merging multiple alarm data to be processed based on the merging rules to obtain a merging result.

[0181] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the alarm data merging processing method provided by the above methods. The method includes: determining merging rules for alarm data to be processed, wherein the merging rules are configured based on business requirements and can be adjusted in real time; and merging multiple alarm data to be processed based on the merging rules to obtain a merging result.

[0183] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the alarm data merging processing method provided by the above methods. The method includes: determining merging rules for alarm data to be processed, wherein the merging rules are configured based on business requirements and can be adjusted in real time; and merging multiple alarm data to be processed based on the merging rules to obtain a merging result.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0186] It is further understood that although the operations are described in a specific order in the accompanying drawings in the embodiments of the present invention, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for merging and processing alarm data, characterized in that, The method includes: Determine the merging rules for the alarm data to be processed, wherein the merging rules are configured based on business requirements and can be adjusted in real time; Based on the merging rule, multiple alarm data to be processed are merged to obtain a merging result. The method further includes, before determining the merging rule for the alarm data to be processed: Obtain business requirements; Configure a set of merging rules for merging alarm data based on the aforementioned business requirements; The rules for determining the merging of alarm data to be processed specifically include: The merging rules that match the alarm data to be processed in the merging rule set are determined as the merging rules for the alarm data to be processed; or, Select the merge rule corresponding to the alarm data to be processed from the set of merge rules; The merging rules are adjusted based on specific business needs, and the adjusted merging rules are determined as the merging rules for the alarm data to be processed.

2. The alarm data merging and processing method according to claim 1, characterized in that, The merging rule includes a first merging rule, wherein the first merging rule is used to instruct the alarm data to be merged according to the merging field in the alarm data to be processed, and the merging field is the field in the alarm data to be processed that represents alarm information. When the merging rule is the first merging rule, the step of merging multiple alarm data to be processed based on the merging rule specifically includes: In the multiple alarm data to be processed, the merging field corresponding to the first merging rule is determined respectively; Based on the merging field, multiple alarm data to be processed are merged.

3. The alarm data merging and processing method according to claim 2, characterized in that, The process of merging multiple alarm data to be processed based on the merging field specifically includes: Determine the timestamp corresponding to each pending alarm data; Based on the timestamp and the merging field, determine the merging identifier for each of the alarm data to be processed; The multiple alarm data to be processed are merged based on the merging identifier.

4. The alarm data merging and processing method according to claim 3, characterized in that, Determine the timestamp corresponding to each pending alarm data, including: Determine the alarm information generation time corresponding to each pending alarm data; Based on the alarm information generation time, a timestamp corresponding to each alarm data to be processed is determined. The timestamp corresponding to the alarm data to be processed whose alarm information generation time is within a preset time range is the first timestamp, and the timestamp corresponding to the alarm data to be processed whose alarm information generation time is outside the preset time range is the second timestamp.

5. The alarm data merging and processing method according to claim 3, characterized in that, The step of determining the merging identifier for each of the pending alarm data based on the timestamp and the merging field includes: Based on the timestamp and the merging field, the merging identifier of each of the alarm data to be processed is obtained through encrypted calculation. When the timestamp and the merging field are the same, the merging identifier obtained by encrypted calculation is the same. The process of merging multiple pending alarm data based on the merging identifier includes: The alarm data with the same merging identifier among the multiple alarm data to be processed are merged.

6. The alarm data merging and processing method according to claim 2, characterized in that, The first merging rule also includes a merging grouping field, which is used to group the alarm data to be processed. The method further includes: Based on the first merging rule, determine the merging grouping field; Based on the merge grouping field, the multiple alarm data to be processed are grouped. The process of merging multiple alarm data to be processed based on the merging field specifically includes: Based on the merging field, multiple alarm data to be processed belonging to the same group are merged.

7. The alarm data merging and processing method according to claim 1, characterized in that, The merging rule includes a second merging rule, wherein the second merging rule is used to instruct the merging of the alarm data to be processed, which includes a preset field; When the merging rule is the second merging rule, the step of merging multiple pending alarm data based on the merging rule specifically includes: Based on the second merging rule, determine whether each piece of alarm data to be processed includes the preset field; The alarm data to be processed, including the preset fields, will be merged.

8. The alarm data merging and processing method according to claim 1, characterized in that, The merging rule also includes a merging result bypass output rule. After obtaining the merging result, the method further includes: Based on the bypass output rule of the merge result, the merge result is controlled to be output in a bypass manner.

9. The alarm data merging and processing method according to claim 1 or 8, characterized in that, After obtaining the merge result, the method further includes: Register a timer, wherein the timer is used to determine the update time for the merge process; If the update time of the timer is determined to be reached, the multiple newly acquired alarm data to be processed and the merging result are updated and merged based on the merging rules.

10. The alarm data merging and processing method according to claim 9, characterized in that, The process of updating and merging multiple newly acquired alarm data to be processed and the merging result specifically includes: Based on the asynchronous update mechanism, multiple newly acquired alarm data to be processed and the merging result are updated and merged.

11. The alarm data merging and processing method according to claim 9, characterized in that, The process of updating and merging multiple newly acquired alarm data to be processed and the merging result based on the merging rules specifically includes: If the merging process fails to update multiple newly acquired alarm data to be processed and the merging result based on the merging rule, the newly acquired alarm data to be processed that failed to be updated will be saved to a preset queue. According to a preset time interval, the newly acquired alarm data that failed to update in the preset queue is consumed again, and the newly acquired alarm data that failed to update and the merge result are updated and merged.

12. The alarm data merging and processing method according to claim 11, characterized in that, After consuming the newly acquired pending alarm data that failed to update in the preset queue again at preset time intervals, the method further includes: If the newly acquired pending alarm data fails to be processed for a preset number of consecutive times due to update failure, the newly acquired pending alarm data that failed to be processed for the preset number of consecutive times will be stored in a preset list.

13. An alarm data merging and processing device, characterized in that, The apparatus is used to implement the alarm data merging and processing method according to any one of claims 1 to 12, the apparatus comprising: The first module is used to determine the merging rules for alarm data to be processed. The merging rules are configured based on business requirements and can be adjusted in real time. The second module is used to merge multiple alarm data to be processed based on the merging rules to obtain the merging result.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the alarm data merging processing method as described in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the alarm data merging processing method as described in any one of claims 1 to 12.

Images