Data processing method, device, computer equipment and storage medium

Abstract

The present application relates to a data processing method, wherein the data processing method comprises: obtaining multiple groups of original data from multiple data sources; unifying the format of each group of the original data to obtain multiple groups of data with unified formats; inputting the multiple groups of data with unified formats into a big data model to obtain threat intelligence in corresponding data formats; sending the threat intelligence to a data platform for processing corresponding data formats, thereby solving the problem that the logstash open source data collection engine adopts a ruby-based configuration syntax, which makes it difficult to troubleshoot threat situations, and realizing the rapid collection and processing of data and troubleshooting of threat situations through a big data model.

Description

Technical Field

[0001] The present application relates to the field of data processing engines, and in particular to data processing methods, devices, computer equipment and storage media. Background Art

[0002] Since there are many types of systems involved, and each system has its own development tools and programming languages, when data is transmitted between multiple systems, it is inevitable that the development and operation costs will increase, and it is also difficult to ensure data consistency. In most companies, more than half of the development time is spent on data transmission between systems. Therefore, commonly used data processing engines process data in multiple formats.

[0003] Due to the huge number of EDR logs, conventional data processing engines cannot quickly collect and process them. The common logstash open source data collection engine can collect data from different data sources and output the data to multiple output sources after processing. However, the logstash open source data collection engine uses a configuration syntax based on ruby ​​(an object-oriented programming scripting language), which makes it difficult to detect threats. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, computer device and storage medium to at least solve the problem in the related art that the logstash open source data collection engine adopts a ruby-based configuration syntax, which makes it difficult to troubleshoot problems.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Acquire multiple groups of raw data from multiple data sources; unify the format of each group of raw data to obtain multiple groups of data in unified formats; input the multiple groups of data in unified formats into a big data model to obtain threat intelligence in corresponding data formats; and send the threat intelligence to a data platform that processes the corresponding data formats.

[0007] In one of the embodiments, the obtaining of multiple groups of raw data from multiple data sources includes: obtaining the raw data from multiple data sources through multiple distributively deployable data input streamers; the raw data includes one or more of active defense format logs, collector logs, EDR format logs, and user-defined format logs.

[0008] In one embodiment, the formatting of each group of the original data is unified to obtain multiple groups of data with unified format, including: storing each group of the original data in different queues; preprocessing the original data in each queue to obtain preprocessed data; the preprocessing includes at least one of deletion, padding and integration; formatting the preprocessed data in each queue to obtain multiple groups of data with unified format.

[0009] In one of the embodiments, the formatting of each group of the original data is unified respectively to obtain multiple groups of data with unified format, which includes: sending the multiple groups of data with unified format to a storage unit; if a backup completion signal is received from the storage unit, the backup is completed; the backup completion signal is a signal fed back by the storage unit after receiving the complete multiple groups of data with unified format; if the backup completion signal is not received from the storage unit, the multiple groups of data with unified format are sent to the storage unit again until the backup is completed.

[0010] In one of the embodiments, the step of inputting the multiple groups of data in the unified format into a big data model to obtain threat intelligence in the corresponding data format includes: obtaining false positive data and field missing data in the multiple groups of data in the unified format; deleting the false positive data and field missing data in the multiple groups of data in the unified format to obtain secondary processed data.

[0011] In one of the embodiments, inputting multiple groups of data in the unified format into a big data model to obtain threat intelligence in the corresponding data format includes: inputting multiple groups of data in the unified format and / or secondary processed data into the big data model; the big data model obtains threat intelligence according to a preset algorithm; the preset algorithm includes at least one of a classification algorithm, an association algorithm, and an aggregation algorithm.

[0012] In one of the embodiments, the step of sending the threat intelligence to a data platform for processing a corresponding data format includes: deleting backup data in a storage unit.

[0013] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0014] Acquisition module: used to obtain multiple sets of raw data from multiple data sources;

[0015] Preprocessing module: used for unifying the format of each group of raw data to obtain multiple groups of data with unified format;

[0016] Parsing module: used to input multiple groups of data in the unified format into the big data model to obtain threat intelligence in the corresponding data format;

[0017] Output module: used to send the threat intelligence to a data platform for processing in a corresponding data format.

[0018] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data processing method as described in the first aspect above is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in the first aspect above.

[0020] Compared with the related art, the data processing method provided in the embodiment of the present application obtains multiple groups of original data from multiple data sources; unifies the format of each group of the original data to obtain multiple groups of data with unified formats; inputs the multiple groups of data with unified formats into a big data model to obtain threat intelligence in corresponding data formats; and sends the threat intelligence to a data platform for processing corresponding data formats. This solves the problem that the logstash open source data collection engine uses a ruby-based configuration syntax, which makes it difficult to troubleshoot threat situations, and realizes the rapid collection and processing of data and the troubleshooting of threat situations through a big data model.

[0021] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flow chart of a data processing method according to an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of a data processing method according to a preferred embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a data processing device according to an embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0028] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0029] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0030] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0031] The data processing method of this application is mainly used to process EDR logs. EDR means endpoint detection and response. Endpoint detection and response is a new research hotspot in the field of endpoint security in the world. It is completely different from the previous passive protection ideas of endpoints. Instead, it actively discovers security threats from external or internal sources through cloud threat intelligence, machine learning, abnormal behavior analysis, attack indicators, etc., and performs automated blocking, evidence collection, remediation and tracing, thereby effectively protecting endpoints.

[0032] When the EDR client finds a threat, it will generate an EDR log, which includes illegal external connection logs, login protection logs, active defense logs, virus detection logs, process protection logs, file access control logs, micro-isolation logs, anti-port scanning logs, traffic profiling logs, website vulnerability protection logs, CC attack protection logs, website access control logs, website backdoor detection logs, ransomware protection logs, file push logs, mining protection logs, intelligence cloud brain logs, and customer-defined threats. If you use the rest interface directly and then send it directly to elasticsearch, the data from various sources and in different formats will increase the development and operation costs, and it will also be difficult to maintain data consistency. Rest (Representational State Transfer, referred to as rest, representational state transfer) is a set of architectural constraint principles. Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine. It can easily enable large amounts of data to have the ability to search, analyze, and explore. If you try to use the traditional logstash data processing engine to input EDR logs into elasticsearch, you can collect data from different data sources and output the data to multiple output sources after processing, but it takes up a lot of system resources, and because the logstash data processing engine is based on the ruby ​​configuration syntax, the syntax is relatively complex and it is difficult to troubleshoot problems. The data collection and processing method of this application can quickly collect, accurately collect, and accurately clean huge EDR logs, achieving high throughput, high concurrency, and high efficiency, and a data processing engine that takes up less system resources. This allows EDR to respond quickly to the terminal and keep threats out.

[0033] This embodiment also provides a data processing method. Figure 1 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0034] Step S101, obtaining multiple groups of original data from multiple data sources;

[0035] Specifically, the data source is preferably an EDR (Endpoint Detection and Response) client, including: a PC device and a Linux server. The log data from the data source is received through a data receiver.

[0036] In one of the embodiments, raw data from multiple data sources are obtained through multiple data input streamers that can be deployed in a distributable manner; the raw data includes one or more of active defense format logs, collector logs, EDR format logs, and user-defined format logs. Specifically, the data receiver can be composed of multiple data input streamers. The data input streamer can receive logs in different formats, and these data input streamers adopt a distributed manner that can be deployed in a distributed manner to achieve the purpose of receiving logs in multiple formats. The data receiver is mainly used to collect EDR logs, but it can also collect active defense format logs, logs collected by other collectors, and logs in user-specified formats. The use of multiple data input streamers that can be deployed in a distributable manner can increase the rate of receiving log data, reduce the performance consumption of the system, and receive huge amounts of data in a short time.

[0037] Step S102, unifying the format of each group of the original data to obtain multiple groups of data with unified formats;

[0038] Specifically, the log data collected by the data receiver is stored in different message queues. According to the preset data structure style rules, the log data stored in the message queue can be retained and selected, and the fields can be supplemented and integrated. After processing the log data, the message queue using the same data structure style rule can obtain data in the same format. Therefore, multiple groups of message queues and data structure style rules are set to obtain multiple groups of data with unified formats. The structure style rules can use at least one of feature matching algorithms and fuzzy matching algorithms. The message queue is a container for storing messages during message transmission. The message queue manager acts as an intermediary when relaying messages from its source to its target. The main purpose of the queue is to provide routing and ensure the delivery of messages; if the recipient is unavailable when sending a message, the message queue will retain the message until it can be successfully delivered. The data structure style rule is an encoding algorithm, including forced conversion to UTF-8 (8-bit, Universal Character Set / Unicode Transformation Format) and JSON format. UTF-8 is a variable-length character encoding for Unicode. It can be used to represent any character in the Unicode standard, and the first byte in its encoding is still compatible with ASCII, so that the original software that processes ASCII characters can continue to be used without or with only a few modifications. Therefore, it has gradually become the preferred encoding for email, web pages and other applications that store or transmit text. JSON (JavaScript Object Notation, JS Object Notation) is a lightweight data exchange format. It is based on a subset of ECMAScript (js specification developed by the European Computer Association) and uses a text format that is completely independent of the programming language to store and represent data. The concise and clear hierarchical structure makes JSON an ideal data exchange language. It is easy for people to read and write, and it is also easy for machines to parse and generate, and effectively improves network transmission efficiency. Due to the existence of a variety of data structure style rules, when the log enters the message queue, according to the data structure style rules, the retention and rejection of this data, as well as the supplementation and integration of field data can be completed in sub-second time, greatly improving the collection and processing efficiency of log data.

[0039] In one embodiment, each group of the raw data is stored in a different queue; the raw data in each queue is preprocessed to obtain preprocessed data; the preprocessing includes at least one of deletion, filling and integration; the preprocessed data in each queue is formatted in a unified manner to obtain multiple groups of data with unified formats. Specifically, a field refers to a part of the content of the log. It is determined whether the JSON format log has basic fields missing. If the basic field loss is too serious to be completed, the data belonging to the field is deleted. If the field can be completed, the field is supplemented and further integrated. The purpose of the integration is to make the log more complete.

[0040] In one of the embodiments, multiple groups of data in the unified format are sent to a storage unit; if a backup completion signal is received from the storage unit, the backup is completed; the backup completion signal is a signal fed back after the storage unit receives the complete multiple groups of data in the unified format; if the backup completion signal is not received from the storage unit, the multiple groups of data in the unified format are sent to the storage unit again until the backup is completed. Specifically, this embodiment is a compensation mechanism for sending preprocessed data. The preprocessed data is sent from the message queue to the core data parser. After the core data parser receives the complete preprocessed data, it saves the preprocessed data to the hard disk and sends a feedback signal to the message queue, and the sending of the preprocessed data is completed. If the feedback signal is not received, the preprocessed data is sent again according to the preset time until the preprocessed data is backed up. This compensation mechanism ensures that data loss will not occur during the sending and saving of data.

[0041] Step S103, inputting multiple groups of data in the unified format into a big data model to obtain threat intelligence in a corresponding data format;

[0042] Specifically, multiple sets of processed data in a unified format are input into the big data model to generate threat intelligence in the corresponding format. Threat intelligence is a vulnerability marker used to identify and detect threats, which is obtained through big data analysis logs.

[0043] In one of the embodiments, inputting multiple groups of data in the unified format into the big data model, and obtaining threat intelligence in the corresponding data format includes: obtaining false alarm data and field loss data in the multiple groups of data in the unified format; deleting the false alarm data and field loss data in the multiple groups of data in the unified format to obtain secondary processed data. Specifically, before the data is input into the big data model, the pre-processed data can be further processed by setting an interceptor. The interceptor mainly completes the parsing of request parameters, assigning page form parameters to corresponding attributes in the value stack, performing function verification, program exception debugging, etc. The interceptor can delete the data with serious field loss that is not deleted in the pre-processing data stage, and can also delete the false alarm data, so as to achieve the purpose of secondary clarification and filtering of the data, obtain secondary processed data, and make the log input into the big data model more complete and perfect.

[0044] In one of the embodiments, multiple groups of data in the unified format are input into a big data model, and obtaining threat intelligence in the corresponding data format includes: inputting multiple groups of data in the unified format and / or secondary processed data into the big data model; the big data model obtains threat intelligence according to a preset algorithm; the preset algorithm includes at least one of a classification algorithm, an association algorithm, and an aggregation algorithm. Specifically, the preferred big data model should include a classification algorithm, an association algorithm, and an aggregation algorithm. The processed data is input into the big data model for calculation. The classification algorithm can classify data in the same format, and the association algorithm can associate related data applied to the same problem. Finally, the data related to the same problem or threat is aggregated through the aggregation algorithm to generate threat intelligence.

[0045] Step S104: Send the threat intelligence to a data platform that processes the corresponding data format.

[0046] Specifically, the threat intelligence generated by the core data parser is sent to the data platform that processes the corresponding data format via the distributed deployment output module. The distributed deployment architecture can simultaneously send multiple threat intelligence to multiple corresponding data platforms to improve the sending efficiency.

[0047] In one embodiment, after sending the threat intelligence to the data platform for processing the corresponding data format, the process includes: deleting the backup data in the storage unit. Specifically, after the threat intelligence is sent to the corresponding data platform through the output module, the output module will feed back the output completion signal to the core data parser. After receiving the output completion signal, the core data parser will delete the backup data of the data corresponding to the signal from the hard disk, saving storage space for the hard disk.

[0048] Through the above steps, this application uses a distributed data receiver and output module to achieve the purpose of high throughput data. The big data model is used to process the relevant data, which greatly improves the data processing speed. Since the message exists in the queue and the big data calculation model used is not based on the ruby ​​configuration syntax, the difficulty of troubleshooting the problem is reduced. In addition, the system resources occupied by the data processing engine composed of the data processing method are also greatly reduced, and the data can be collected quickly and accurately, and the huge EDR logs can be cleaned accurately. This allows EDR to respond quickly to the terminal and generate threat intelligence.

[0049] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0050] Figure 2 Schematic diagram of a data processing method according to a preferred embodiment of the present application. Figure 2 As shown, a large number of logs in four formats, including active defense format logs, other collector logs, EDR format logs, and user-defined format logs, are collected by the distributed collection module. The distributed collection module ensures that multiple data input streams can collect data at the same time. In the collection module, the data collected by the data input stream is stored in a queue. According to the data structure style rules, the data with missing fields in the data stored in the queue is deleted, retained, supplemented, and integrated. The data processed by the same collection module is in the same format; multiple collection modules process data through multiple data structure style rules to obtain multiple groups of data in different formats, ensuring the orderliness of the data. The multiple groups of data in different formats obtained after the data processing are input into the core data parser. A notification unit can also be set in the collection module, which is used to ensure the communication between the collection module and the core data processing module. If the core data parser receives the data, it is stored in the hard disk backup and a feedback signal is sent to the collection module. If the collection module does not receive the feedback signal, the data is cached and sent again after a preset time period until the backup is completed. The interceptor in the core data parser can clean and filter the data again, and the secondary processed data is input into the big data model for classification, association and integration to generate threat intelligence which is sent to the corresponding data platform by the output module, such as one or more of the Elasticsearch, graphite, customer big data module and EDR big data module in the figure.

[0051] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] This embodiment also provides a data processing device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module", "unit", "subunit" and the like can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0053] Figure 3 is a structural block diagram of a data processing device according to an embodiment of the present application, such as Figure 3 As shown, the device comprises:

[0054] Acquisition module 10: used to acquire multiple sets of raw data from multiple data sources;

[0055] Preprocessing module 20: used for unifying the format of each group of original data to obtain multiple groups of data with unified format;

[0056] Parsing module 30: used for inputting multiple groups of data in the unified format into the big data model to obtain threat intelligence in the corresponding data format;

[0057] Output module 40: used for sending the threat intelligence to a data platform for processing in a corresponding data format.

[0058] The collection module 10 is also used to obtain original data from multiple data sources through multiple data input streams that can be deployed in a distributed manner; the original data includes one or more of active defense format logs, collector logs, EDR format logs, and user-defined format logs.

[0059] The preprocessing module 20 is also used to store each group of the original data into different queues; preprocess the original data in each queue to obtain preprocessed data; the preprocessing includes at least one of deletion, filling and integration; the preprocessed data in each queue is formatted in a unified manner to obtain multiple groups of data with unified formats.

[0060] The preprocessing module 20 is also used to send multiple groups of data in the unified format to the storage unit; if a backup completion signal is received from the storage unit, the backup is completed; the backup completion signal is a signal fed back after the storage unit receives the complete multiple groups of data in the unified format; if the backup completion signal is not received from the storage unit, the multiple groups of data in the unified format are sent to the storage unit again until the backup is completed.

[0061] The parsing module 30 is further used to obtain false positive data and field missing data in the plurality of sets of data in the unified format; delete the false positive data and field missing data in the plurality of sets of data in the unified format to obtain secondary processed data.

[0062] The parsing module 30 is also used to input multiple groups of unified format data and / or secondary processed data into the big data model; the big data model obtains threat intelligence according to a preset algorithm; the preset algorithm includes at least one of a classification algorithm, an association algorithm and an aggregation algorithm.

[0063] The output module 40 is also used to delete the backup data in the storage unit.

[0064] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0065] In addition, combined Figure 1 The data method described in the embodiment of the present application can be implemented by a computer device. Figure 4 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0066] The computer device may include a processor 41 and a memory 42 storing computer program instructions.

[0067] Specifically, the processor 41 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0068] Among them, the memory 42 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 42 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 42 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 42 may be inside or outside the data processing device. In a specific embodiment, the memory 42 is a non-volatile memory. In a specific embodiment, the memory 42 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0069] The memory 42 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 41 .

[0070] The processor 41 implements any one of the data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 42 .

[0071] In some embodiments, the computer device may further include a communication interface 43 and a bus 4. Figure 4 As shown, the processor 41, the memory 42, and the communication interface 43 are connected via a bus 4 and communicate with each other.

[0072] The communication interface 43 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 43 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0073] The bus 4 includes hardware, software or both, and couples the components of the computer device to each other. The bus 4 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 4 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 4 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0074] The computer device can execute the data processing method in the embodiment of the present application based on the acquired computer program instructions, thereby realizing the combination Figure 1 Describe the data processing methods.

[0075] In addition, in combination with the data processing method in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data processing methods in the above embodiments is implemented.

[0076] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: include: Obtain multiple sets of raw data from multiple data sources; Including, obtaining raw data from multiple data sources through multiple data input streams that can be deployed in a distributed manner; The raw data includes one or more of active defense format logs, collector logs, EDR format logs, and user-defined format logs; Unifying the format of each group of the original data to obtain multiple groups of data with unified formats; Inputting multiple groups of data in the unified format into a big data model to obtain threat intelligence in a corresponding data format; Sending the threat intelligence to a data platform that processes corresponding data formats; The threat intelligence obtained in the corresponding data format includes: The big data model obtains threat intelligence according to a preset algorithm; the preset algorithm includes a classification algorithm, an association algorithm and an aggregation algorithm; the classification algorithm is used to classify data in the same format, the association algorithm is used to associate related data applied to the same problem, and the aggregation algorithm is used to aggregate data related to the same problem or threat to generate threat intelligence.

2. The data processing method according to claim 1, characterized in that: The format of each group of the original data is unified to obtain multiple groups of data with unified formats, including: Storing each group of the original data into different queues respectively; Preprocessing the original data in each queue to obtain preprocessed data; the preprocessing includes at least one of deletion, filling and integration; The pre-processed data in each queue are formatted in a unified manner to obtain multiple groups of data in a unified format.

3. The data processing method according to claim 2, characterized in that: The step of unifying the format of each group of the original data to obtain multiple groups of data with unified formats comprises: Sending multiple groups of data in the uniform format to a storage unit; If a backup completion signal from the storage unit is received, the backup is completed; the backup completion signal is a signal fed back by the storage unit after receiving the complete multiple sets of the unified format data; If the backup completion signal from the storage unit is not received, multiple groups of data in the uniform format are sent to the storage unit again until the backup is completed.

4. The data processing method according to claim 1, characterized in that: The step of inputting the plurality of sets of data in the unified format into the big data model to obtain threat intelligence in the corresponding data format includes: Acquire false positive data and field missing data from a plurality of sets of data in the unified format; False positive data and field missing data in the plurality of groups of data in the unified format are deleted to obtain secondary processed data.

5. The data processing method according to claim 4, characterized in that: The step of inputting the plurality of sets of data in the unified format into the big data model comprises: Input a plurality of sets of the format-unified data and / or secondary-processed data into the big data model.

6. The data processing method according to claim 5, characterized in that: The step of sending the threat intelligence to a data platform for processing in a corresponding data format includes: Delete the backup data in the storage unit.

7. A data processing device, characterized in that: include: Acquisition module: used to obtain multiple sets of raw data from multiple data sources; Including, obtaining raw data from multiple data sources through multiple data input streamers that can be deployed in a distributed manner; the raw data includes one or more of active defense format logs, collector logs, EDR format logs, and user-defined format logs; Preprocessing module: used for unifying the format of each group of raw data to obtain multiple groups of data with unified format; Parsing module: used to input multiple groups of data in the unified format into the big data model to obtain threat intelligence in the corresponding data format; Output module: used to send the threat intelligence to a data platform for processing corresponding data formats; The parsing module is also used for the big data model to obtain threat intelligence according to a preset algorithm; the preset algorithm includes a classification algorithm, an association algorithm and an aggregation algorithm; the classification algorithm is used to classify data in the same format, the association algorithm is used to associate related data applied to the same problem, and the aggregation algorithm is used to aggregate data related to the same problem or threat to generate threat intelligence.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images