Data aggregation method and device

Abstract

The present disclosure proposes a data aggregation method and device, which relates to the field of data processing. The present disclosure decouples and separates various tasks of data to be processed and processes them in parallel, and based on the offset management mechanism of various tasks, persists the executed common parts of various tasks, thereby completing data aggregation efficiently and quickly.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data aggregation method and device. Background Art

[0002] With the development of information technology, the network security situation is becoming increasingly severe. In order to effectively perceive the security situation and prevent security attacks as early as possible, it is necessary to effectively aggregate malicious IPs, malicious domain names, malicious files and their related intelligence data, establish a threat intelligence library, and provide security warnings for corporate assets.

[0003] In addition to collecting the company's own intelligence data, the threat intelligence database also needs to collect public intelligence sources on the Internet through web crawlers. As the amount of intelligence data grows, the original intelligence aggregation mechanism requires more and more computing resources and the response time is getting longer and longer. Therefore, there is an urgent need for a solution that can complete intelligence aggregation efficiently and quickly. Summary of the invention

[0004] The present invention decouples and separates various tasks of the data to be processed and processes them in parallel, and based on the offset management mechanism of various tasks, persists the executed common parts of various tasks, thereby completing data aggregation efficiently and quickly. In addition, processing resources are flexibly allocated during task execution, effectively improving resource utilization and reducing data processing time.

[0005] Some embodiments of the present disclosure provide a data aggregation method, including:

[0006] Get data to be processed in batches;

[0007] Performing various task processing of different task types on the data in parallel, wherein an offset is set for each task type to record the execution progress of the task of the corresponding task type;

[0008] Determine the intersection of the executed parts of the various tasks based on the respective offsets;

[0009] The data in the intersection is processed persistently.

[0010] In some embodiments, determining the intersection of executed portions of various tasks according to the offsets includes:

[0011] Get the complement of the executed part of the persistence task;

[0012] Get the first intersection of the executed parts of various tasks except the persistence task;

[0013] An intersection of the complement set and the first intersection set is determined as an intersection of executed parts of the various tasks.

[0014] In some embodiments, the performing various task processing of different task types on the data in parallel includes:

[0015] Slice each task of the data to obtain multiple subtasks of the corresponding task;

[0016] Determine the priority of each subtask;

[0017] According to the priority of each subtask, priority is given to scheduling processing resources for subtasks with high priority;

[0018] The scheduled multiple processing resources are used to process the corresponding subtasks in parallel.

[0019] In some embodiments, slicing each task of the data to obtain a plurality of subtasks of the corresponding task includes:

[0020] According to ∑ i t i With t i The ratio information of determines the length of the subtask slice of task i of a certain task type, where t i Indicates the time required to execute task i for a preset amount of data;

[0021] According to the length of the subtask slice of task i of each task type, task i is sliced ​​to obtain multiple subtasks of task i.

[0022] In some embodiments, determining the length of the subtask slice of task i of a certain task type includes:

[0023]

[0024] Among them, len i Represents the length of the subtask slice of task i, and l represents a configurable constant.

[0025] In some embodiments, determining the priority of each subtask includes:

[0026] Determine the priority of each subtask of the corresponding task based on the total number and execution offset of tasks of each task type, the processing resources expected to be required by the subtasks of the task, and the available processing resources of the current container.

[0027] Among them, the priority of a subtask of a task of a certain task type is positively correlated with the ratio information of the total number of tasks and the execution offset of the task, and is positively correlated with the ratio information of the available processing resources of the current container and the processing resources expected to be required by the subtask of the task.

[0028] In some embodiments, determining the priority of each subtask includes:

[0029]

[0030] Among them, C represents the priority of a subtask of a task of a certain task type, m represents the total number of tasks, n represents the execution offset of the task, r represents the available processing resources of the current container, and w represents the processing resources expected to be required by the subtasks of the task.

[0031] In some embodiments, the method further includes: if the priority of the first subtask in the subtask queue formed according to the priority is less than 0, blocking the subtask queue and waiting for the release of processing resources.

[0032] In some embodiments, various tasks of different task types include normalization tasks, information completion tasks, association tasks, and persistence tasks.

[0033] Some embodiments of the present disclosure provide a data aggregation device, including:

[0034] Memory,

[0035] as well as

[0036] A processor is coupled to the memory, and the processor executes the data aggregation method of any one of the embodiments based on instructions stored in the memory.

[0037] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of the data aggregation method of any embodiment when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0039] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0040] Figure 1 The present invention is a flowchart of a data aggregation method according to some embodiments of the present invention.

[0041] Figure 2 The present invention is a flowchart of an intelligence data aggregation method according to some embodiments of the present invention.

[0042] Figure 3 A schematic diagram of the implementation process of task decoupling and parallel scheduling is shown.

[0043] Figure 4A schematic diagram of the implementation process of flexibly allocating processing resources during task execution is shown.

[0044] Figure 5 It is a schematic diagram of the structure of a data aggregation device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

[0046] Figure 1 The present invention is a flowchart of a data aggregation method according to some embodiments of the present invention.

[0047] like Figure 1 As shown, the method of this embodiment includes: steps 11-14.

[0048] In step 11, data to be processed is acquired in batches.

[0049] The amount of data to be processed in batches is usually large, usually including multiple data items.

[0050] In step 12, various task processing of different task types are respectively performed on the data in parallel, wherein an offset is set for each task type to record the execution progress of the task of the corresponding task type.

[0051] Various tasks of different task types include, for example, normalization tasks, information completion tasks, association tasks, and persistence tasks, but are not limited to the examples given. In the examples given, a data can only execute the persistence task after the normalization task, information completion task, and association task are completed.

[0052] In some embodiments, performing various task processing of different task types on the data in parallel includes steps 121-124:

[0053] In step 121, each task of the data is sliced ​​to obtain a plurality of subtasks of the corresponding task.

[0054] A slicing method comprises: according to ∑ i t i With t i The ratio information, for example, using the formula Determine the length of the subtask slice of task i of a certain task type, where t i Indicates the time required to execute task i with a preset amount of data, len i represents the length of the subtask slice of task i, l represents a configurable constant, and the size of the slice can be adjusted; according to the length of the subtask slice of task i of each task type, task i is sliced ​​to obtain multiple subtasks of task i.

[0055] For example, the time taken to execute the normalization task, information supplementation task, information association task, and persistence task for 10,000 pieces of data is 18s, 24s, 26s, and 32s respectively. The constant parameter l is defined as 8000, then the length of the normalization subtask slice n = (18+24+26+32) / 18*8000, and the result of rounding n down is 44445.

[0056] This slicing method fully considers the differences in execution time of different types of tasks. Tasks with longer execution time have smaller slice lengths and more corresponding slices, which provides more opportunities to allocate processing resources and helps improve overall resource utilization.

[0057] In step 122, the priority of each subtask is determined.

[0058] In some embodiments, determining the priority of each subtask includes: determining the priority of each subtask of the corresponding task based on the total number of tasks of each task type and the execution offset and the processing resources expected to be required for the subtasks of the task and the available processing resources of the current container, wherein the priority of a subtask of a task of a certain task type is positively correlated with the ratio information of the total number of tasks and the execution offset of the task, and is positively correlated with the ratio information of the available processing resources of the current container and the processing resources expected to be required for the subtasks of the task.

[0059] For example, the following formula is used to determine the priority of each subtask.

[0060]

[0061] Among them, C represents the priority of a subtask of a task of a certain task type, m represents the total number of tasks, n represents the execution offset of the task, r represents the available processing resources of the current container, and w represents the processing resources expected to be required by the subtasks of the task.

[0062] For example, if the execution offset of the normalization task is 5000, the total number of normalization tasks is 10000, the normalization subtask is estimated to occupy 100M of memory, and the current container (such as a computer) has 1000M of remaining memory, then the priority of the normalization subtask is C = (10000 / 5000)*log(1000 / 100) = 2.

[0063] This priority determination method ensures that tasks with lagging execution progress will be given priority in allocating processing resources, making the execution progress of various types of tasks roughly the same, thereby avoiding the huge execution progress gap between different types of tasks affecting the final persistence processing, thereby effectively improving resource utilization and reducing data processing time.

[0064] In addition, if the priority of the first subtask in the subtask queue formed according to the priority is less than 0, it means If it is less than 1, the current container has no suitable subtasks. In this case, the subtask queue can be blocked and processing resources can be released.

[0065] In step 123, according to the priorities of the subtasks, processing resources are preferentially scheduled for the subtasks with higher priorities.

[0066] The processing resources here may be, for example, processing resources of a container, and the container may be, for example, a computer.

[0067] In step 124, the scheduled multiple processing resources are used to process the corresponding subtasks in parallel.

[0068] In step 13, the intersection of the executed parts of the various tasks is determined according to the respective offsets.

[0069] In some embodiments, determining the intersection of the executed parts of various tasks according to the offsets includes: obtaining the complement of the executed parts of the persistent task; obtaining the first intersection of the executed parts of various tasks other than the persistent task; and determining the intersection of the complement and the first intersection as the intersection of the executed parts of the various tasks. The formula is:

[0070] X=A∩B∩C∩CD

[0071] Among them, A represents the executed part of the data normalization task, B represents the executed part of the information completion task, C represents the executed part of the information association task, D represents the executed part of the persistence task, CD represents the complement of the executed part of the persistence task, and X represents the intersection of the executed parts of the various tasks determined.

[0072] For example, assuming there are 10,000 data items, A = [Normalization, 0, 4000], indicating that 0-4000 data items out of the 10,000 data items have been normalized, B = [Information Completion, 0, 3000], indicating that 0-3000 data items out of the 10,000 data items have been completed The task of completing the information has been executed, C = [Association, 0, 5000], indicating that 0-5000 data items out of the 10,000 data items have been associated with the task, and D = [Persistence, 0, 2000]), indicating that 0-2000 data items out of the 10,000 data items have been persistently stored. The intersection X is [2000, 3000], that is, 2000-3000 data items can be persisted in the subsequent steps.

[0073] In step 14, the data in the intersection is persisted, that is, the storage state of the data in the intersection is changed from a transient state to a persistent state, that is, the data in the intersection is saved from the memory to the database.

[0074] In the above embodiment, various tasks of the data to be processed are decoupled and separated and processed in parallel, and based on the offset management mechanism of various tasks, the executed common parts of various tasks are persistently processed, so as to efficiently and quickly complete data aggregation. In addition, processing resources are flexibly allocated during task execution. For example, tasks with lagging execution progress will be given priority in allocating processing resources, so that the execution progress of various types of tasks is roughly the same, so as to avoid the huge execution progress gap between different types of tasks affecting the final persistence processing, thereby effectively improving resource utilization and reducing data processing time.

[0075] The following uses intelligence data as an example to illustrate the above data aggregation method.

[0076] like Figure 2 The intelligence data aggregation method shown is as follows: first, the intelligence data of each intelligence source is crawled in batches, the intelligence data is numbered sequentially, and the intelligence data is cached in the computer memory; then, the normalization task, information completion task, intelligence association task, and data persistence task of the intelligence data are sliced ​​according to the set rules (the slicing method is referred to above), and after the slicing is completed, the scheduling priority of each subtask is sorted according to the evaluation algorithm, and the resource scheduler allocates computing resources to the subtasks in the scheduling queue in sequence according to the resource occupancy of the system, and executes each sliced ​​subtask in parallel. After the slicing subtask is completed, the progress offset of the corresponding task will be updated; finally, based on the offset of each task, the executed part of each task is intersectionally aggregated to obtain the data segment stored in the library, and it is persisted in the distributed database.

[0077] In the above intelligence data aggregation process, Figure 3 The implementation process of task decoupling and parallel scheduling is shown:

[0078] Step 31: Batch pull intelligence data. Use crawler technology to batch crawl intelligence data from various intelligence sources, number the intelligence data in sequence, and cache the intelligence data in computer memory.

[0079] Step 32: Process each task in parallel based on its offset identifier. Perform data normalization (such as deduplication of information), intelligence information completion (such as geographic location information completion), intelligence association, persistence, and other tasks in parallel, and use each task's own offset to record the execution progress of each task.

[0080] Step 33: The intersections of the tasks are aggregated to obtain the data segment X stored in the library.

[0081] X=A∩B∩C∩CD, A represents the executed part in the data normalization task, B represents the executed part in the information completion task, C represents the executed part in the information association task, D represents the executed part in the persistence task, and CD represents the complement of the executed part in the persistence task.

[0082] Step 34: Data persistence: The data in the data segment X in the above step is sequentially persisted in the distributed database.

[0083] The above-mentioned embodiment decouples each link of intelligence aggregation based on the offset, realizes the parallel execution of tasks such as data normalization, information completion, associated intelligence, persistence, etc., thereby completing data aggregation efficiently and quickly, with good scalability and good system fault tolerance.

[0084] In the above intelligence data aggregation process, Figure 4 The following figure shows the implementation process of elastically allocating processing resources during task execution:

[0085] Step 41: Slicing of intelligence aggregation tasks. First, the tasks of intelligence data normalization, intelligence information completion, intelligence association, and persistence are sliced ​​to obtain multiple subtasks.

[0086] For example, using the formula Determine the length of the subtask slice of task i, where t i represents the time required to execute task i for 10,000 pieces of intelligence data, i represents all tasks such as intelligence data normalization, intelligence information completion, intelligence association, and persistence, len i It represents the length of the subtask slice of task i, and l represents a configurable constant that can adjust the size of the slice.

[0087] Step 42: Evaluate intelligence aggregation subtask priorities.

[0088] For example, the following formula is used to determine the priority of each subtask.

[0089]

[0090] Among them, C represents the priority of a subtask of a task of a certain task type, m represents the total number of tasks, n represents the execution offset of the task, r represents the available processing resources of the current container, and w represents the processing resources expected to be required by the subtasks of the task.

[0091] Step 43: Scheduling of intelligence aggregation subtasks. Sort the subtasks by their calculated priorities and put them into the resource scheduling queue (43.1). If the priority of the first element in the queue is less than 0, the queue is blocked to wait for resource release, and then step 42 (43.2) is re-executed.

[0092] Step 44: Sub-tasks of intelligence aggregation are issued. Specifically, 44.1 the sub-tasks in the queue are sequentially submitted to the task scheduler to apply for processing resources, 44.2 the task scheduler allocates the processing resource address to the scheduling queue, 44.3 the scheduling queue issues the sub-tasks, and 44.4 the task scheduler starts the corresponding sub-tasks.

[0093] The above embodiment realizes dynamic and flexible allocation of resources. First, the task is divided into slices, then the subtasks are arranged by priority sorting, and finally the task scheduler is used to flexibly allocate resources to the subtasks, thereby realizing efficient execution of business and effective utilization of resources in the process of intelligence data aggregation.

[0094] Figure 5 It is a schematic diagram of the structure of a data aggregation device according to some embodiments of the present disclosure.

[0095] like Figure 5 As shown, the data aggregation device 50 of this embodiment includes: a memory 51, and a processor 52 coupled to the memory, and the processor executes the data aggregation method of any embodiment based on the instructions stored in the memory.

[0096] The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0097] The present disclosure also proposes a non-transitory computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the data aggregation method of any embodiment are implemented.

[0098] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A data aggregation method, characterized in that: include: Get data to be processed in batches; Performing various task processing of different task types on the data in parallel, wherein an offset is set for each task type to record the execution progress of the task of the corresponding task type; Determine the intersection of the executed parts in various tasks according to the offsets, including: obtaining the complement of the executed parts in the persistence task; Get the first intersection of the executed parts of various tasks except the persistence task; Determine the intersection of the complement and the first intersection as the intersection of the executed parts of the various tasks; The data in the intersection is processed persistently.

2. The method according to claim 1, characterized in that The performing various task processing of different task types on the data in parallel includes: Slice each task of the data to obtain multiple subtasks of the corresponding task; Determine the priority of each subtask; According to the priority of each subtask, priority is given to scheduling processing resources for subtasks with high priority; The scheduled multiple processing resources are used to process the corresponding subtasks in parallel.

3. The method according to claim 2, characterized in that The slicing of each task of the data to obtain a plurality of subtasks of the corresponding task includes: according to and The proportion information determines the tasks of a certain task type The length of the subtask slice, where Indicates a preset number of data execution tasks the time required; By task type The length of the subtask slice for the task Slice to get the task Multiple subtasks.

4. The method according to claim 3, characterized in that: The task of determining a certain task type The length of the subtask slices includes: in, Indicates the task The length of the subtask slice, Represents a configurable constant.

5. The method according to claim 2, characterized in that: Determining the priority of each subtask includes: Determine the priority of each subtask of the corresponding task based on the total number and execution offset of tasks of each task type, the processing resources expected to be required by the subtasks of the task, and the available processing resources of the current container. Among them, the priority of a subtask of a task of a certain task type is positively correlated with the ratio information of the total number of tasks and the execution offset of the task, and is positively correlated with the ratio information of the available processing resources of the current container and the processing resources expected to be required by the subtask of the task.

6. The method according to claim 5, characterized in that Determining the priority of each subtask includes: Among them, C represents the priority of a subtask of a task of a certain task type, m represents the total number of tasks, n represents the execution offset of the task, r represents the available processing resources of the current container, and w represents the processing resources expected to be required by the subtasks of the task.

7. The method according to claim 6, characterized in that Also includes: If the priority of the first subtask in the subtask queue formed according to the priority is less than 0, the subtask queue is blocked and waits for the processing resources to be released.

8. The method according to any one of claims 1 to 7, characterized in that: Various tasks of different task types include normalization tasks, information completion tasks, association tasks, and persistence tasks.

9. A data aggregation device, characterized in that: include: Memory, as well as A processor coupled to the memory, wherein the processor executes the data aggregation method according to any one of claims 1 to 8 based on instructions stored in the memory.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data aggregation method according to any one of claims 1 to 8.

Images