Scanning log data processing method and device, electronic equipment and storage medium

Abstract

This application discloses a method, apparatus, electronic device, and computer storage medium for processing scan log data. The method includes: acquiring a scan log data stream; performing streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field; and storing any scan log record data in a first message queue after obtaining any scan log record data; and streaming consumption of each scan log record data in the first message queue according to the purpose identifier field of each scan log record data. This method can effectively reduce development tasks and achieve efficient scan log data processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for processing scan log data. Background Technology

[0002] When processing scanned samples such as executable files or datasets, the resulting scan log data contains a wealth of useful information. Analysts typically extract information from this scan log data that meets specific criteria based on user needs. For example, when a user needs to obtain scan log information for executable files uploaded within a certain time period, analysts can write corresponding filtering scripts based on user-provided filtering conditions (such as sample type, sample upload time, etc.), and then use these scripts to perform offline processing tasks on the scan log data, thereby filtering out the scan log information that meets the user's requirements.

[0003] However, current methods for processing scan log data rely on writing and executing filtering scripts separately for each new user requirement. As user requirements increase, this leads to the need to develop many offline processing tasks with the same workflow, impacting the processing speed of scan log data. Therefore, a more efficient method for processing scan log data is needed. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and computer storage medium for processing scan log data, which can achieve efficient scan log data processing. The above technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a method for processing scan log data, including:

[0006] Obtain the scan log data stream;

[0007] Perform streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, store the scan log record data into the first message queue;

[0008] Based on the purpose identifier field of each scan log record, stream the scan log record data in the first message queue.

[0009] In one possible implementation, the scan log data stream is stream-washed to obtain at least one scan log record containing a purpose identifier field, including:

[0010] Based on the hash value field corresponding to each of the multiple scan log data in the scan log data stream, perform streaming deduplication on the scan log data stream;

[0011] Based on the current rule file, perform streaming rule matching on the deduplicated scan log data stream to obtain at least one scan log record that includes a purpose identifier field.

[0012] In one possible implementation, the current rule file includes at least one preset rule instance; each preset rule instance includes a preset filtering rule, a preset extraction rule, and a preset identification rule.

[0013] Based on the current rule file, perform streaming rule matching on the deduplicated scan log data stream to obtain at least one scan log record that includes a purpose identifier field, including:

[0014] Based on the preset filtering rules in each preset rule instance, the scan log data stream after streaming deduplication is stream-filtered to obtain the stream-filtered scan log data stream.

[0015] Based on the preset extraction rules in each preset rule instance, stream field extraction is performed on the scan log data stream after stream filtering to obtain at least one scan log extraction information. After obtaining any scan log extraction information, the additional identifier field corresponding to any scan log extraction information is determined based on the preset rule instance that the scan log extraction information matches. The additional identifier field includes the purpose identifier field.

[0016] Based on the information extracted from each scan log and the corresponding additional identifier field, at least one scan log record data including a purpose identifier field is obtained.

[0017] In one possible implementation, based on the purpose identifier field of each scan log record, the scan log record data in the first message queue is streamed and consumed, including:

[0018] If any scan log record data has a purpose identifier field that is a scan purpose identifier field, then any scan log record data will be streamed to the distributed file system;

[0019] If any scan log record data has an extraction purpose identifier field, then any scan log record data is streamed to the second message queue.

[0020] In one possible implementation, after streaming any scan log record data to the distributed file system, if the purpose identifier field of any scan log record data is a scan purpose identifier field, the method further includes:

[0021] Based on the behavior of the sample backscan platform in reading data from each scan log record in the distributed file system, corresponding scan behavior records are generated.

[0022] In one possible implementation, after streaming any scan log record data to the second message queue if the purpose identifier field of any scan log record data is an extraction purpose identifier field, the method further includes:

[0023] Based on the consumer application's behavior of consuming data from each scan log record in the second message queue, corresponding consumption behavior records are generated.

[0024] In one possible implementation, before performing streaming rule matching on the deduplicated scan log data stream according to the current rule file to obtain at least one scan log record including a purpose identifier field, the method further includes:

[0025] Check whether the rule file has been updated at preset time intervals;

[0026] If a rule file update is detected, the most recently updated rule file will be used as the current rule file.

[0027] Secondly, embodiments of this application provide a scan log data processing apparatus, comprising:

[0028] The acquisition module is used to acquire scan log data streams;

[0029] The cleaning module is used to perform streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, store the scan log record data into the first message queue.

[0030] The consumption module is used to stream and consume the scan log data in the first message queue based on the purpose identifier field of each scan log record.

[0031] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory; wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps provided in the first aspect of embodiments of this application.

[0032] Fourthly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading and executing the method steps provided in the first aspect of embodiments of this application by a processor.

[0033] The aforementioned scan log data processing method, apparatus, electronic device, and computer storage medium acquire scan log data streams, perform streaming cleansing on the scan log data streams to obtain at least one scan log record data including a purpose identifier field, and store any scan log record data in a first message queue after obtaining any scan log record data. Based on the purpose identifier field of each scan log record data, the system streams and consumes each scan log record data in the first message queue. This not only allows for the merging of multiple scan log data of the same type onto a single streaming task line for streaming cleansing, responding promptly to new data input, but also facilitates subsequent streaming consumption of each scan log record data in the first streaming consumption queue by adding a purpose identifier field to the original scan log data during the streaming cleansing process. The entire process reduces development tasks and achieves efficient scan log data processing. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A schematic diagram of the architecture of a scan log data processing system provided for an exemplary embodiment of this application;

[0036] Figure 2 A flowchart illustrating a scan log data processing method provided for an exemplary embodiment of this application;

[0037] Figure 3 A flowchart illustrating another scan log data processing method provided as an exemplary embodiment of this application;

[0038] Figure 4 A data flow diagram of a scan log data processing method provided in an exemplary embodiment of this application;

[0039] Figure 5 A flowchart illustrating another scan log data processing method provided as an exemplary embodiment of this application;

[0040] Figure 6 A schematic diagram of the structure of a scan log data processing apparatus provided for an exemplary embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] In the description of this application, it should be understood that terms such as "scan log" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0044] Please see Figure 1 This is a schematic diagram of the architecture of a scan log data processing system provided as an exemplary embodiment of this application. The streaming server 10 communicates with the message queue server 20 and the distributed file system server 30 via a network. The data storage system 40 can store the rule files that the streaming server 10 needs to process. The data storage system 40 can be integrated onto the streaming server 10, or it can be located in the cloud or on other network servers.

[0045] In some possible embodiments, the streaming server 10 acquires a scan log data stream, performs streaming cleansing on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, stores the scan log record data in a first message queue configured on the message queue server 20, and then streams and consumes each scan log record data in the first message queue according to the purpose identifier field of each scan log record data. Specifically, the streaming server 10 streams and consumes each scan log record data in the first message queue according to the purpose identifier field of each scan log record data, including: if the purpose identifier field of any scan log record data is a scan purpose identifier field, streaming the scan log record data to a distributed file system deployed on the distributed file system server 30; if the purpose identifier field of any scan log record data is an extraction purpose identifier field, streaming the scan log record data to a second message queue configured on the message queue server 20.

[0046] It is worth noting that the aforementioned stream processing server 10 can be, but is not limited to, a Flink server, the aforementioned message queue server 20 can be, but is not limited to, a Kafka server, and the aforementioned distributed file system server 30 can be, but is not limited to, an HDFS server. The stream processing server 10, message queue server 20, and distributed file system server 30 can all be implemented using independent servers or server clusters composed of multiple servers.

[0047] In one embodiment, such as Figure 2 As shown, a method for processing scan log data is provided. Taking the application of this method to a streaming processing server as an example, the method includes the following steps:

[0048] S201: Obtain scan log data stream.

[0049] The scan log data stream consists of multiple scan log data entries, each of which may be, but is not limited to, XML format data, and each scan log data entry contains a hash value field for identification and tracking.

[0050] Optionally, after completing the scanning of the scan samples, the log scanning platform writes the generated scan log data to a distributed message queue (such as a Kafka message queue). A streaming server (such as a Flink server) reads incremental scan log data streams from the distributed message queue to process each XML-formatted scan log data entry in real time. Each XML-formatted scan log data entry contains a hash value field (such as an MD5 field) used to uniquely identify the scan log data and to track associated scan samples.

[0051] In this embodiment, the streaming server reads incremental scan log data streams from a distributed message queue, enabling it to merge multiple types of scan log data into a single streaming task line for streaming cleaning. This allows for timely responses to new data inputs, helps reduce development tasks, and achieves efficient scan log data processing.

[0052] S202: Perform streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, store the scan log record data into the first message queue.

[0053] Streaming cleaning includes streaming deduplication, streaming rule matching, and streaming formatting. The purpose identifier field distinguishes the specific purpose of each scan log record. For example, the purpose identifier field can be used to distinguish whether each scan log record is used to backscan related scan samples or to extract specific scan log information (i.e., scan log record data) corresponding to related scan samples. The first message queue can be, but is not limited to, a message queue pre-configured on a message queue server.

[0054] Optionally, the streaming processing server performs streaming deduplication on the scanning log data stream based on the hash value fields corresponding to each of the multiple scanning log data in the scanning log data stream. It then performs streaming rule matching on the streaming deduplicated scanning log data stream according to the current rule file and converts the data obtained from the streaming rule matching from XML format to JSON format, thereby obtaining at least one scanning log record data including a usage identifier field (usetype field). After obtaining any scanning log record data, the server stores the data in the first pre-configured message queue (such as Kafka topic1) on the message queue server.

[0055] In this embodiment, the streaming processing server performs streaming cleansing on the scan log data stream to obtain at least one scan log record data including a purpose identifier field. After obtaining any scan log record data, the server immediately stores the scan log record data into the first message queue. This not only reduces a large number of development tasks, but also ensures that each scan log record data generated based on the original scan log data includes a purpose identifier field through streaming cleansing. It also ensures that these record data are not duplicated or omitted and are in a standard format, which facilitates subsequent process processing and improves the efficiency of scan log data processing.

[0056] S203: Based on the purpose identifier field of each scan log record data, stream consume each scan log record data in the first message queue.

[0057] Optionally, the streaming server distributes each scan log record in the first message queue to a distributed file system (such as HDFS) or a second message queue (such as Kafka topic2) based on the purpose identifier field of each scan log record data.

[0058] Specifically, the streaming server extracts any scan log record from the first message queue and identifies the purpose identifier field corresponding to that scan log record. If the purpose identifier field of any scan log record is identified as a scan purpose identifier field, the streaming server distributes the scan log record to a distributed file system, which then stores the scan log record data in the corresponding directory based on the hash value field and additional identifier field contained in the scan log record data. Subsequently, the sample backscanning platform extracts the scan log record data from the corresponding directory according to a preset sample backscanning strategy and performs a backscan of the associated scan samples corresponding to that scan log record based on the hash value field contained in the scan log record data. If the purpose identifier field of any scan log record is identified as an extraction purpose identifier field, the streaming server distributes the scan log record data to a second message queue. The consumer application then extracts the scan log record data from the second message queue, generates a notification email corresponding to that scan log record data, and sends the notification email to the user terminal.

[0059] In this embodiment, the streaming server consumes each scan log record data from the first message queue according to the purpose identifier field of each scan log record data. While ensuring the completion of scan log data cleaning, it improves the efficiency of data distribution, thereby reducing development tasks and realizing efficient scan log data processing.

[0060] In the above-described scan log data processing method, the streaming server acquires the scan log data stream, performs streaming cleansing on the stream to obtain at least one scan log record containing a purpose identifier field. After obtaining any scan log record, it stores the data in a first message queue. Based on the purpose identifier field of each scan log record, the server streams and consumes all scan log records in the first message queue. This not only allows for the merging of multiple scan log data of the same type onto a single streaming task line for streaming cleansing, responding promptly to new data input, but also facilitates subsequent streaming consumption of scan log records in the first message queue by adding a purpose identifier field to the original scan log data during the streaming cleansing process. The entire process reduces development tasks and achieves efficient scan log data processing.

[0061] In one embodiment, such as Figure 3 As shown, another method for processing scan log data is provided. Taking the application of this method to a streaming processing server as an example, the method includes the following steps:

[0062] S301: Obtain scan log data stream.

[0063] Specifically, S301 is the same as S201, and will not be repeated here.

[0064] S302: Check if the rule file has been updated at preset time intervals. If yes, proceed to S303; otherwise, proceed to S304.

[0065] The rule file is a pre-configured rule configuration file used to define filtering rules, extraction rules, and additional identifier rules for the scan log data stream.

[0066] Optionally, the streaming server queries the data storage system at preset time intervals to check if the rule files stored therein have been updated. If the rule files have been updated, the latest rule files are loaded, meaning the most recently updated rule files are used as the current rule files. If the rule files have not been updated, the current rule files are used, and streaming rule matching is performed on the deduplicated scan log data stream based on the current rule files.

[0067] Specifically, the streaming server can record the timestamp or version number of the rule file each time it reads the rule file, and compare the timestamp or version number of the detected rule file with the most recently recorded timestamp or version number during subsequent detection. If a change in the timestamp or version number is found, the rule file is considered to have been updated.

[0068] S303: Use the most recently updated rule file as the current rule file.

[0069] Optionally, when the streaming server detects an update to the rule file, it uses the most recently updated rule file as the current rule file to perform streaming rule matching on the deduplicated scan log data stream based on the current rule file.

[0070] In this embodiment, the streaming server queries the rule file in the data storage system for updates at preset time intervals. If there are updates, the new rule file is loaded and used to perform streaming rule matching on the deduplicated scan log data stream. This not only reduces a lot of development tasks but also facilitates the management of pre-set rule configurations, achieving efficient scan log data processing.

[0071] S304: Perform streaming deduplication on the scan log data stream based on the hash value field corresponding to each of the multiple scan log data in the scan log data stream.

[0072] Optionally, the streaming server extracts the hash value field (such as the MD5 field) corresponding to each of the multiple scan log data in the scan log data stream, and performs streaming deduplication on the scan log data stream based on the hash value field corresponding to each of the multiple scan log data.

[0073] Specifically, for any scan log data in the scan log data stream, the streaming server first queries the hash value field status of that scan log data during processing. If the hash value field status of the scan log data is empty, the processing of that scan log data continues, and the hash value field status is changed to exist; if the hash value field status of the scan log data is existent, it is considered that the scan log data has occurred repeatedly, and the scan log data is discarded.

[0074] S305: Perform streaming rule matching on the deduplicated scan log data stream according to the current rule file to obtain at least one scan log record data, and after obtaining any scan log record data, store the scan log record data in the first message queue.

[0075] The current rules file includes at least one preset rule instance, and each preset rule instance includes preset filtering rules, preset extraction rules, and preset identification rules. Each scan log record includes a purpose identifier field.

[0076] Optionally, the streaming processing server performs streaming filtering on the deduplicated scan log data stream according to the preset filtering rules in each preset rule instance to obtain the streaming filtered scan log data stream; then, according to the preset extraction rules in each preset rule instance, it performs streaming field extraction on the streaming filtered scan log data stream to obtain at least one scan log extraction information; after obtaining any scan log extraction information, it determines the additional identifier field corresponding to any scan log extraction information according to the preset rule instance that the scan log extraction information matches; finally, based on each scan log extraction information and the additional identifier field corresponding to each scan log extraction information, it obtains at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, it stores the scan log record data in the first message queue (such as Kafka topic1).

[0077] Furthermore, the additional identifier fields include a usetype field and a sign field. Specifically, the usetype field distinguishes the specific purpose of each scan log record. For example, it can differentiate whether a scan log record is used to backscan related scan samples or to extract specific scan log information (i.e., scan log record data) corresponding to those samples. The sign field distinguishes the rule identifiers matched by each scan log record, facilitating downstream classification of the scan log records according to rules and writing them to different data files. For instance, a subsequent second message queue (such as Kafka topic2) can classify the scan log records in the first message queue according to rules based on the sign field of each scan log record, writing each category of data to different files, thus enabling more effective management and analysis of different types of scan log records.

[0078] It is worth noting that this embodiment only describes a preferred implementation of this application. After the streaming processing server performs streaming rule matching on the streaming deduplication scan log data stream according to the current rule file, it can also convert the data obtained by streaming rule matching from XML format to JSON format, which is more compatible and simpler, to form at least one scan log record data. After obtaining any scan log record data, the scan log record data is stored in the first message queue pre-configured on the message queue server to ensure that the format of each scan log record data is more lightweight and easier to process. This application embodiment does not limit whether format conversion is performed during the streaming cleaning process, or what kind of format conversion is performed.

[0079] In this embodiment, the streaming processing server performs streaming deduplication on the scan log data stream, matches the deduplicated scan log data stream with streaming rules according to the current rule file, obtains at least one scan log record, and stores any scan log record data in the first message queue after obtaining any scan log record data. This not only ensures the efficient completion of the scan log data stream cleaning task, but also helps to distribute the cleaned scan log record data to different downstream processes according to the purpose identifier field, thereby reducing development tasks and achieving efficient scan log data processing.

[0080] S306: Stream extracts data from each scan log record in the first message queue, and identifies the purpose identifier field of any scan log record after extracting it.

[0081] Optionally, the streaming server extracts each scan log record data from the first message queue in a streaming manner, and immediately identifies the purpose identifier field (usetype field) of any scan log record data after extraction, so as to distribute each scan log record data to the corresponding downstream process according to the purpose identifier field.

[0082] S307: If any scan log record data has a purpose identifier field that is a scan purpose identifier field, then any scan log record data is streamed to the distributed file system.

[0083] Optionally, if the purpose identifier field of a scan log record is a scan purpose identifier field, the streaming server distributes the scan log record data to a distributed file system (such as HDFS), so that the distributed file system stores the scan log record data in the corresponding directory based on the hash value field and additional identifier field contained in the scan log record data. Subsequently, the sample backscanning platform will extract the scan log record data from the corresponding directory according to the preset sample backscanning strategy, and perform backscanning on the associated scan samples corresponding to the scan log record data based on the hash value field contained in the scan log record data.

[0084] S308: If the purpose identifier field of any scan log record data is the extraction purpose identifier field, the scan log record data is streamed to the second message queue.

[0085] Optionally, if the purpose identifier field of a scan log record is the extraction purpose identifier field, the streaming server distributes the scan log record data to a second message queue (such as Kafka topic2). Subsequently, the consumer application extracts the scan log record data from the second message queue, generates a corresponding notification email, and sends the notification email to the user terminal.

[0086] In this embodiment, the streaming server distributes the scan log data in the first message queue to the distributed file system or the second message queue according to the purpose identifier field of each scan log record data. This reduces development tasks while achieving efficient scan log data processing.

[0087] S309: Generate corresponding scan behavior records based on the behavior of the sample backscan platform in reading the scan log data in the distributed file system.

[0088] Understandably, after the streaming server distributes the scan log data to the distributed file system, the sample backscanning platform can extract the scan log data from the directory of the distributed file system according to a preset sample backscanning strategy. Based on the hash value field contained in each scan log data, it can backscan the associated scan samples corresponding to each scan log record. Therefore, the streaming server can determine the access history of the sample backscanning platform by recording the timestamps and specific entries of each scan log data read from the distributed file system, thereby generating corresponding scan behavior records.

[0089] In this embodiment, the streaming server generates corresponding scanning behavior records by reading the timestamps and specific entries of each scan log record data from the distributed file system through the sample backscan platform. This facilitates efficient management and auditing of the generated scan log record data, improving the reliability of scan log data processing and the user experience.

[0090] S310: Generate corresponding consumption behavior records based on the behavior of the consumer application in consuming data from each scan log record in the second message queue.

[0091] Understandably, after the streaming server distributes the scan log data to the second message queue, the consumer application can extract the scan log data from the second message queue. Based on the rule-identifying fields of each scan log data entry, the application categorizes the data according to these rules, writes it to disk into different data files, and finally writes it to the file system accessible to users. Therefore, the streaming server can determine the access history of each consumer application by recording the timestamps and specific entries of each scan log data entry read from the distributed file system, thereby generating corresponding consumption behavior records.

[0092] In this embodiment, the streaming server generates corresponding consumption behavior records by recording the timestamps and specific entries of each scan log record data read by the consumer application from the distributed file system. This facilitates efficient management and auditing of the generated scan log record data, improving the reliability of scan log data processing and the user experience.

[0093] For example, please see Figure 4This is a data flow diagram illustrating a scan log data processing method provided in an exemplary embodiment of this application. After obtaining a scan log data stream from an upstream distributed message queue, the streaming server performs streaming cleansing on the scan log data stream to obtain at least one scan log record data including a purpose identifier field. After obtaining any scan log record data, the server stores the scan log record data in a first message queue. Then, based on the purpose identifier field of each scan log record, the server streams and distributes the scan log record data in the first message queue to a distributed file system or a second message queue.

[0094] After the streaming server distributes the scan log data to the distributed file system, the sample backscan platform extracts the scan log data from the directory of the distributed file system according to a preset sample backscan strategy. Based on the hash value field contained in each scan log record, it performs a backscan on the associated scan samples corresponding to each scan log record. At this time, the streaming server determines the access history of the sample backscan platform by recording the timestamps and specific entries of each scan log record read from the distributed file system, thereby generating corresponding scan behavior records.

[0095] After the streaming server distributes the scan log data to the second message queue, the consumer application extracts the scan log data from the second message queue. Based on the rule-identifying fields of each scan log data entry, the application categorizes the data according to these rules, writes it to different data files, and finally writes it to the file system accessible to users. At this point, the streaming server records the timestamps and specific entries of each scan log data entry read from the distributed file system by the consumer application to determine the consumer application's access history, thereby generating corresponding consumption behavior records.

[0096] In the aforementioned scan log data processing method, the streaming processing server queries the rule file in the data storage system at preset time intervals to check for updates. If updates are found, the new rule file is loaded and used to perform streaming rule matching on the deduplicated scan log data stream. This not only reduces development tasks but also facilitates the management of pre-configured rule settings. By performing streaming deduplication on the scan log data stream and matching it with streaming rules based on the current rule file, at least one scan log record is obtained. After obtaining any scan log record, it is stored in the first message queue. This ensures efficient completion of the scan log data stream cleaning task and facilitates the subsequent distribution of the cleaned scan log record data to different downstream processes based on the usage identifier field. By recording the historical behavior of the sample backscan platform and each consumer application in reading each scan log record data from the distributed file system, scan behavior records and consumption behavior records are generated, facilitating efficient management and auditing of the generated scan log record data. The entire process reduces development tasks while improving the efficiency and reliability of scan log data processing.

[0097] In one embodiment, such as Figure 5 As shown, another method for processing scan log data is provided. Taking the application of this method to a streaming processing server as an example, the method includes the following steps:

[0098] S501: Obtain scan log data stream.

[0099] Specifically, S501 is the same as S201, and will not be repeated here.

[0100] S502: Check if the rule file has been updated at preset time intervals. If yes, proceed to S503; otherwise, proceed to S504.

[0101] Specifically, S502 is the same as S302, and will not be repeated here.

[0102] S503: Use the most recently updated rule file as the current rule file.

[0103] Specifically, S503 is the same as S303, and will not be repeated here.

[0104] S504: Perform streaming deduplication on the scan log data stream based on the hash value field corresponding to each of the multiple scan log data in the scan log data stream.

[0105] Specifically, S504 is the same as S304, and will not be repeated here.

[0106] S505: Based on the preset filtering rules in each preset rule instance, perform streaming filtering on the scan log data stream after streaming deduplication to obtain the stream-filtered scan log data stream.

[0107] The preset filtering rules include preset filtering conditions, which are used to determine the scan log data in the scan log data stream that meets the filtering requirements.

[0108] Optionally, each scan log data entry may include a scan sample type field and a scan sample upload time field. If the preset filtering rule in a preset rule instance is set in advance to: the scan sample type is a specified executable file and the scan sample upload time belongs to a specified time period, then during the streaming filtering process, the streaming engine will stream-extract the information under the scan sample type field and the scan sample upload time field from each scan log data entry. Based on the information under the above two fields, the stream-filtered scan log data will be retained, satisfying both the scan sample type being a specified executable file and the scan sample upload time belonging to the specified time period, thus obtaining the stream-filtered scan log data stream.

[0109] S506: Based on the preset extraction rules in each preset rule instance, perform streaming field extraction on the scan log data stream after streaming filtering to obtain at least one scan log extraction information. After obtaining any scan log extraction information, determine the additional identifier field corresponding to any scan log extraction information based on the preset rule instance that the scan log extraction information matches.

[0110] Each preset rule instance includes a preset rule identifier and a preset purpose identifier. The preset extraction rule includes preset extraction fields, which guide the streaming engine to automatically extract pre-defined fields from each scan log data entry. Additional identifier fields include a rule identifier field and a purpose identifier field.

[0111] For example, assuming a preset extraction rule is set to extract the hash value field and the scan sample type field from the scan log data, the streaming processing engine will perform streaming extraction on the information under the hash value field and the information under the scan sample type field in each scan log data obtained after streaming filtering, to obtain at least one scan log extraction information. After obtaining any scan log extraction information, the engine will determine the rule identifier field and the purpose identifier field corresponding to any scan log extraction information based on the preset rule instance that the scan log extraction information matches.

[0112] Understandably, since the preset extraction rules in each preset rule instance may be set differently, a certain scan log data obtained after streaming filtering may hit multiple preset rule instances at the same time. In this case, the streaming engine will generate multiple scan log extraction information according to the multiple preset rule instances hit by the scan log data.

[0113] S507: Based on the information extracted from each scan log and the additional identifier field corresponding to each scan log extraction information, obtain at least one scan log record data.

[0114] Each scan log record includes a hash value field, a purpose identifier field, and a rule identifier field.

[0115] Optionally, the streaming engine extracts information from each scan log and its corresponding rule identifier field and purpose identifier field to obtain at least one scan log record data. In subsequent processing, the purpose identifier field in the additional identifier field can facilitate streaming distribution by the streaming engine, and the hash value field and rule identifier field in the additional identifier field can facilitate the distributed file system and the second message queue to classify and store each scan log record data according to rules.

[0116] In this embodiment, the streaming processing server performs streaming filtering on each scan log data in the scan log data stream according to the preset filtering rules in each preset rule instance; it performs streaming field extraction on each scan log data in the scan log data stream according to the preset extraction rules in each preset rule instance; and it determines the rule identifier field and purpose identifier field corresponding to each scan log extracted information according to the preset identifier rules in each preset rule instance. This not only ensures the efficient completion of the scan log data stream cleaning task, but also helps to distribute the cleaned scan log record data to different downstream processes according to the purpose identifier field, and facilitates the downstream processes to classify and store each scan log record data according to the rules based on the rule identifier field. This reduces development tasks while improving the efficiency and reliability of scan log data processing.

[0117] S508: Stream extracts data from each scan log record in the first message queue, and identifies the purpose identifier field of any scan log record after extracting it.

[0118] Specifically, S508 is the same as S306, and will not be repeated here.

[0119] S509: If any scan log record data has a purpose identifier field that is a scan purpose identifier field, then any scan log record data is streamed to the distributed file system.

[0120] Specifically, S509 is the same as S307, and will not be repeated here.

[0121] S510: If the purpose identifier field of any scan log record data is the extraction purpose identifier field, the scan log record data is streamed to the second message queue.

[0122] Specifically, S510 is identical to S308, and will not be repeated here.

[0123] S511: Generate corresponding scan behavior records based on the behavior of the sample backscan platform in reading the scan log data in the distributed file system.

[0124] Specifically, S511 is the same as S309, and will not be repeated here.

[0125] S512: Generate corresponding consumption behavior records based on the behavior of the consumer application in consuming data from each scan log record in the second message queue.

[0126] Specifically, S512 is the same as S310, and will not be repeated here.

[0127] In the aforementioned scan log data processing method, the streaming server performs streaming filtering, streaming field extraction, and determines the rule identifier field and purpose identifier field corresponding to each scan log data entry based on the preset rule instances in the current rule file. This ensures efficient completion of the scan log data stream cleaning task. Secondly, it facilitates the subsequent distribution of the cleaned scan log records to different downstream processes based on the purpose identifier field. Thirdly, it allows downstream processes to classify and store each scan log record according to rules based on the hash value field and the rule identifier field. The entire process reduces development tasks while improving the efficiency and reliability of scan log data processing.

[0128] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0129] Based on the inventive concept of the above-mentioned scan log data processing method, such as Figure 6As shown, this application embodiment also provides a scan log data processing apparatus 600 for implementing the scan log data processing method described above. The scan log data processing apparatus 600 includes:

[0130] Module 601 is used to acquire scan log data streams;

[0131] The cleaning module 602 is used to perform streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, store the scan log record data into the first message queue.

[0132] The consumption module 603 is used to stream the scan log data in the first message queue according to the purpose identifier field of each scan log record.

[0133] In one possible implementation, the cleaning module 602 is specifically used to perform streaming deduplication on the scanning log data stream based on the hash value fields corresponding to each of the multiple scanning log data in the scanning log data stream; and to perform streaming rule matching on the streaming deduplicated scanning log data stream according to the current rule file to obtain at least one scanning log record data including a purpose identifier field.

[0134] In one possible implementation, the current rule file includes at least one preset rule instance; each preset rule instance includes a preset filtering rule, a preset extraction rule, and a preset identification rule; the cleaning module 602 is specifically used to perform streaming filtering on the scan log data stream after streaming deduplication according to the preset filtering rule in each preset rule instance, to obtain a streaming filtered scan log data stream; to perform streaming field extraction on the streaming filtered scan log data stream according to the preset extraction rule in each preset rule instance, to obtain at least one scan log extraction information, and after obtaining any scan log extraction information, to determine the additional identification field corresponding to any scan log extraction information according to the preset rule instance hit by any scan log extraction information; the additional identification field includes a purpose identification field; based on each scan log extraction information and the additional identification field corresponding to each scan log extraction information, at least one scan log record data including a purpose identification field is obtained.

[0135] In one possible implementation, the consumer module 603 is specifically used to stream any scan log record data to the distributed file system when the purpose identifier field of any scan log record data is a scan purpose identifier field; and to stream any scan log record data to the second message queue when the purpose identifier field of any scan log record data is an extraction purpose identifier field.

[0136] In one possible implementation, the scan log data processing device 600 further includes a recording module, which generates corresponding scan behavior records based on the behavior of the sample backscan platform reading scan log record data in the distributed file system.

[0137] In one possible implementation, the aforementioned recording module is also used to generate corresponding consumption behavior records based on the behavior of the consumer application consuming data from each scan log record in the second message queue.

[0138] In one possible implementation, the scan log data processing device 600 further includes a detection module for detecting whether the rule file has been updated at a preset time interval; if an update to the rule file is detected, the most recently updated rule file is used as the current rule file.

[0139] Each module in the aforementioned scan log data processing device 600 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0140] This application also provides an electronic device, which can be a server, and its internal structure diagram can be as follows: Figure 7 As shown, this electronic device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores pre-configured rule files, etc. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. The processor executes computer programs to implement a scan log data processing method.

[0141] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one possible implementation, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0143] Obtain the scan log data stream;

[0144] Perform streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, store the scan log record data into the first message queue;

[0145] Based on the purpose identifier field of each scan log record, stream the scan log record data in the first message queue.

[0146] In one possible implementation, the processor, when executing the computer program, also performs the following steps: performing streaming deduplication on the scanning log data stream based on the hash value fields corresponding to each of the multiple scanning log data in the scanning log data stream; and performing streaming rule matching on the streaming deduplicated scanning log data stream according to the current rule file to obtain at least one scanning log record data including a purpose identifier field.

[0147] In one possible implementation, the current rule file includes at least one preset rule instance; each preset rule instance includes a preset filtering rule, a preset extraction rule, and a preset identification rule; when the processor executes the computer program, it further implements the following steps: according to the preset filtering rule in each preset rule instance, performing streaming filtering on the stream of scan log data after streaming deduplication to obtain a stream of stream-filtered scan log data; according to the preset extraction rule in each preset rule instance, performing streaming field extraction on the stream of stream-filtered scan log data to obtain at least one scan log extraction information; and after obtaining any scan log extraction information, determining the additional identification field corresponding to any scan log extraction information according to the preset rule instance matched by any scan log extraction information; the additional identification field includes a purpose identifier; based on each scan log extraction information and the additional identification field corresponding to each scan log extraction information, obtaining at least one scan log record data including a purpose identifier field.

[0148] In one possible implementation, the processor, when executing the computer program, further performs the following steps: if the purpose identifier field of any scan log record data is a scan purpose identifier field, stream the scan log record data to a distributed file system; if the purpose identifier field of any scan log record data is an extraction purpose identifier field, stream the scan log record data to a second message queue.

[0149] In one possible implementation, the processor, when executing the computer program, also performs the following steps: generating corresponding scan behavior records based on the behavior of the sample backscan platform reading data from each scan log record in the distributed file system.

[0150] In one possible implementation, the processor, when executing the computer program, also performs the following steps: generating corresponding consumption behavior records based on the behavior of the consumer application in consuming data from each scan log record in the second message queue.

[0151] In one possible implementation, the processor, when executing the computer program, also performs the following steps: detecting whether the rule file has been updated at preset time intervals; if an update is detected, the most recently updated rule file is used as the current rule file.

[0152] This application also provides a computer storage medium storing instructions that, when run on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the above-described computer-readable storage medium.

[0153] In one possible implementation, a computer storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, it performs the following steps:

[0154] Obtain the scan log data stream;

[0155] Perform streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, store the scan log record data into the first message queue;

[0156] Based on the purpose identifier field of each scan log record, stream the scan log record data in the first message queue.

[0157] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: performing streaming deduplication on the scanning log data stream based on the hash value fields corresponding to each of the multiple scanning log data in the scanning log data stream; and performing streaming rule matching on the streaming deduplicated scanning log data stream according to the current rule file to obtain at least one scanning log record data including a purpose identifier field.

[0158] In one possible implementation, the current rule file includes at least one preset rule instance; each preset rule instance includes a preset filtering rule, a preset extraction rule, and a preset identification rule; when the computer program is executed by the processor, it further implements the following steps: performing streaming filtering on the stream of scan log data after streaming deduplication according to the preset filtering rules in each preset rule instance to obtain a stream of stream-filtered scan log data; performing streaming field extraction on the stream of stream-filtered scan log data according to the preset extraction rules in each preset rule instance to obtain at least one scan log extraction information; and after obtaining any scan log extraction information, determining the additional identification field corresponding to any scan log extraction information according to the preset rule instance matched by any scan log extraction information; the additional identification field includes a purpose identification field; and obtaining at least one scan log record data including a purpose identification field based on each scan log extraction information and the additional identification field corresponding to each scan log extraction information.

[0159] In one possible implementation, when the computer program is executed by the processor, it further performs the following steps: if the purpose identifier field of any scan log record data is a scan purpose identifier field, stream the scan log record data to the distributed file system; if the purpose identifier field of any scan log record data is an extraction purpose identifier field, stream the scan log record data to the second message queue.

[0160] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: generating corresponding scan behavior records based on the behavior of the sample backscan platform reading data from each scan log record in the distributed file system.

[0161] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: generating corresponding consumption behavior records based on the behavior of the consumer application in consuming data from each scan log record in the second message queue.

[0162] In one possible implementation, when the computer program is executed by the processor, it also performs the following steps: detecting whether the rule file has been updated at preset time intervals; if an update is detected, the most recently updated rule file is used as the current rule file.

[0163] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer storage medium or transmitted through the computer storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0165] The embodiments described above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Any modifications and improvements made by those skilled in the art to the technical solutions of this application without departing from the spirit of this application should fall within the protection scope defined by the claims.

[0166] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for processing scan log data, characterized in that, The method includes: Obtain the scan log data stream; The scan log data stream is stream-cleaned to obtain at least one scan log record data including a purpose identifier field, and after obtaining any of the scan log record data, the scan log record data is stored in the first message queue; Based on the purpose identifier field of each scan log record data, the scan log record data in the first message queue is consumed in a streaming manner.

2. The method as described in claim 1, characterized in that, The streaming cleansing of the scan log data stream yields at least one scan log record data including a purpose identifier field, including: Based on the hash value field corresponding to each of the multiple scan log data in the scan log data stream, stream-based deduplication is performed on the scan log data stream. Based on the current rule file, perform streaming rule matching on the deduplicated scan log data stream to obtain at least one scan log record that includes a purpose identifier field.

3. The method as described in claim 2, characterized in that, The current rule file includes at least one preset rule instance; each preset rule instance includes a preset filtering rule, a preset extraction rule, and a preset identification rule. The step of performing streaming rule matching on the deduplicated scan log data stream according to the current rule file to obtain at least one scan log record data including a purpose identifier field includes: According to the preset filtering rules in each of the preset rule instances, the scan log data stream after streaming deduplication is stream-filtered to obtain the stream-filtered scan log data stream. Based on the preset extraction rules in each of the preset rule instances, streaming field extraction is performed on the scan log data stream after streaming filtering to obtain at least one scan log extraction information. After obtaining any scan log extraction information, an additional identifier field corresponding to any scan log extraction information is determined based on the preset rule instance that any scan log extraction information matches. The additional identifier field includes the purpose identifier field. Based on the extracted scan log information and the corresponding additional identifier field, at least one scan log record data including a purpose identifier field is obtained.

4. The method as described in claim 1, characterized in that, The step of streaming the scan log data in the first message queue according to the purpose identifier field of each scan log record includes: If any of the scan log record data has a purpose identifier field that is a scan purpose identifier field, then any of the scan log record data will be streamed to the distributed file system; If any of the scan log record data has a purpose identifier field that is an extraction purpose identifier field, then any of the scan log record data is streamed to the second message queue.

5. The method as described in claim 4, characterized in that, In the case that any of the scan log record data uses a scan purpose identifier field, after streaming the scan log record data to the distributed file system, the method further includes: Based on the behavior of the sample backscan platform in reading the scan log records in the distributed file system, corresponding scan behavior records are generated.

6. The method as described in claim 4, characterized in that, In the case that the purpose identifier field of any of the scan log record data is an extraction purpose identifier field, after streaming the scan log record data to the second message queue, the method further includes: Based on the consumer application's behavior of consuming the scan log data recorded in the second message queue, corresponding consumption behavior records are generated.

7. The method as described in claim 2, characterized in that, Before performing streaming rule matching on the deduplicated scan log data stream according to the current rule file to obtain at least one scan log record including a purpose identifier field, the method further includes: Check whether the rule file has been updated at preset time intervals; If an update to the rule file is detected, the most recently updated rule file will be used as the current rule file.

8. A scanning log data processing device, characterized in that, The device method includes: The acquisition module is used to acquire scan log data streams; The cleaning module is used to perform streaming cleaning on the scan log data stream to obtain at least one scan log record data including a purpose identifier field, and after obtaining any scan log record data, store any scan log record data into the first message queue; The consumption module is used to stream the scan log records in the first message queue according to the purpose identifier field of each scan log record.

9. An electronic device, characterized in that, include: A processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-7.

Images