A vulnerability collection system based on big data

Abstract

The application discloses a kind of based on big data's vulnerability collection system, comprising: data set capture module, the data set of all data in a collection cycle is obtained and generated;Data preprocessing module, the characteristic word information of the metadata is collected;Initial grouping construction module, the metadata in data set is sorted according to the number of characteristic word information, selects the several metadata of the front of sorting and generates the initial grouping corresponding thereto respectively, and the characteristic word information of the metadata in the initial grouping is extracted as the label of initial grouping;Data classification module, call the metadata outside initial metadata in data set, the metadata is divided into different initial grouping;Data screening module, the metadata in the same processing group is parsed, and sequentially compared with initial metadata.In a large amount of data, the integration of phase and system data is completed, so as to solve the influence of conventional data collection vulnerability on computer computing power.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and more specifically to a vulnerability acquisition system based on big data. Background Technology

[0002] Big data, also known as massive data, refers to information whose scale is so large that it cannot be captured, managed, processed, and organized into a form that helps businesses make more proactive decisions within a reasonable timeframe using mainstream software tools. It is characterized by four main features: massive data volume, rapid data flow, diverse data types, and low value density.

[0003] Therefore, for a massive and complex amount of data, it inevitably contains a large amount of similar, identical, or even incomplete data, or the content of a certain data may be covered by other data. In a sense, this data can be called useless data, but during the big data collection process, there is no way to identify this data. As a result, useless data is also entered into the database and stored. The existence of this data collection vulnerability will consume a considerable amount of server computing power, and will also require users to repeatedly look up the same or similar data when they want to access it, affecting users' normal data access. Summary of the Invention

[0004] The purpose of this invention is to provide a vulnerability collection system based on big data, which solves the following technical problems:

[0005] Conventional big data collection methods often fail to identify identical or highly similar data, thus consuming additional server computing power.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A vulnerability acquisition system based on big data includes:

[0008] The dataset capture module acquires all data within a collection period and generates a dataset, wherein each element within the dataset is defined as metadata.

[0009] The data preprocessing module collects the feature term information of the metadata, wherein the feature term information is the keywords that appear more than a set threshold in the metadata;

[0010] The initial grouping module sorts the metadata in the dataset according to the number of feature terms, selects the top-ranked metadata to generate initial groups corresponding to them, and extracts the feature terms of the metadata in the initial groups as the labels of the initial groups. The metadata is defined as initial metadata.

[0011] The data classification module retrieves metadata from the dataset beyond the initial metadata and divides the metadata into different initial groups according to the similarity between the metadata feature terms and the tags of the initial groups, thus generating processing groups.

[0012] The data filtering module parses the metadata in the same processing group and compares it with the initial metadata in turn. If the similarity between the two exceeds the set maximum threshold, only the information in the metadata that is different from the initial metadata is retained and merged into the initial metadata to complete the update of the initial metadata; if the similarity between the two does not exceed the set maximum threshold, the metadata is retained.

[0013] As a further aspect of the present invention: in the initial grouping construction module, the number of initial groups generated is determined by the dataset, and the specific steps are as follows:

[0014] Set the maximum capacity L for a single initial packet and the standard data capacity M;

[0015] Get the total data size N of the dataset;

[0016] The number of standard data points S contained in the dataset is calculated by S = N / M;

[0017] The initial number of groups K is calculated using K = L / S.

[0018] As a further aspect of the present invention: during the process of generating the initial group and selecting the top K metadata, if the similarity of the feature term information of any two metadata exceeds a set threshold, the metadata with the lower ranking is determined to be invalid metadata; then new metadata is selected to replace the invalid metadata, and the reselected metadata is the metadata with the highest ranking other than the already selected metadata.

[0019] As a further aspect of the present invention, the specific method for generating the processing group is as follows:

[0020] Retrieve the feature term information of the aforementioned metadata;

[0021] The feature terms are compared sequentially with the labels of different initial groups;

[0022] The initial group corresponding to the tag with the highest similarity to the aforementioned feature term is selected as the metadata processing group.

[0023] As a further aspect of the present invention: during the data filtering process of the processing group, after each comparison process between the initial metadata and the metadata is completed, the initial metadata is updated, and then compared with the next metadata after the update.

[0024] As a further aspect of the present invention: during the data filtering process of the processing group, if the similarity between the initial metadata and the metadata does not exceed the threshold, then the initial metadata remains unchanged during this update process.

[0025] As a further aspect of the present invention: during the data filtering process of the processing group, if the similarity between the initial metadata and the metadata is lower than the set minimum threshold, the metadata is determined to be metadata to be determined, the metadata to be determined is deleted from the processing group, and the initial metadata of other processing groups is compared in turn, and the metadata to be determined is included in the processing group containing the initial metadata with the highest similarity.

[0026] As a further aspect of the present invention: when none of the initial metadata has a similarity to the metadata to be determined exceeding the set minimum threshold, the metadata to be determined is generated into a separate processing group.

[0027] The beneficial effects of this invention are as follows: In this invention, the collection period is a preset time period, which can be 1 hour or 1 day. The specific choice should be determined according to the server's computing power and the data processing requirements, and can be freely set by the user. A longer time period results in a larger dataset and more data to be processed at once. Conversely, if the data demand is urgent, the period should be appropriately shortened. The data in the dataset is crawled from the network by the corresponding collection module, thus presenting a chaotic and disordered state. Therefore, this invention first extracts high-frequency terms from different metadata. These high-frequency terms, i.e., feature term information, can reflect the content of the metadata to some extent. Furthermore, in the subsequent sorting process, the number of feature term information is used as the sorting criterion, taking into account the number of high-frequency terms. The more data there is, the richer and more comprehensive the metadata becomes. Therefore, using this initial metadata as the basis for grouping allows for the inclusion of other related metadata within a single group. This invention uses feature classification to systematically divide large amounts of data into several processing groups. These processing groups can be further subdivided into smaller groups, following similar steps. After dividing the unordered data stream into ordered processing groups, the next step involves deleting metadata completely included in the initial metadata within the same processing group, and merging metadata with similarity exceeding a set maximum threshold into the initial metadata, which is deemed unworthy of separate storage. This process integrates the data from the initial metadata with the system data, thus mitigating the impact of conventional data acquisition vulnerabilities on computing power. Attached Figure Description

[0028] The invention will now be further described with reference to the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of the structure of a vulnerability collection system based on big data according to the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please see Figure 1 As shown, this invention is a vulnerability acquisition system based on big data, comprising:

[0032] The dataset capture module acquires all data within a collection period and generates a dataset, wherein each element within the dataset is defined as metadata.

[0033] The data preprocessing module collects the feature term information of the metadata, wherein the feature term information is the keywords that appear more than a set threshold in the metadata;

[0034] The initial grouping module sorts the metadata in the dataset according to the number of feature terms, selects the top-ranked metadata to generate initial groups corresponding to them, and extracts the feature terms of the metadata in the initial groups as the labels of the initial groups. The metadata is defined as initial metadata.

[0035] The data classification module retrieves metadata from the dataset beyond the initial metadata and divides the metadata into different initial groups according to the similarity between the metadata feature terms and the tags of the initial groups, thus generating processing groups.

[0036] The data filtering module parses the metadata in the same processing group and compares it with the initial metadata in turn. If the similarity between the two exceeds the set maximum threshold, only the information in the metadata that is different from the initial metadata is retained and merged into the initial metadata to complete the update of the initial metadata; if the similarity between the two does not exceed the set maximum threshold, the metadata is retained.

[0037] In this invention, the collection period is a preset time period, which can be 1 hour or 1 day. The specific choice should be determined according to the server's computing power and the data processing requirements. It can be freely set by the user because the longer the time period, the larger the dataset and the more data needs to be processed at once. Similarly, if the data demand is urgent, the period should be appropriately shortened. The data in the dataset is crawled from the network by the corresponding collection module, so the data is in a chaotic and disordered state. Therefore, in this invention, the first step is to extract high-frequency terms from different metadata. These high-frequency terms, i.e., feature term information, can reflect the content of the metadata to some extent. In the subsequent sorting process, the number of feature term information is also taken into consideration. The more high-frequency terms there are, the better the metadata content. The more data there is, the richer and more comprehensive it becomes. Therefore, using this initial metadata as the basis for grouping allows for the inclusion of as much related metadata as possible within a single group. Thus, this invention uses feature classification to systematically divide large amounts of data into several processing groups. These processing groups can be further subdivided into smaller groups, following similar steps. After dividing the unordered data stream into ordered processing groups, the next step involves deleting metadata completely included in the initial metadata within the same processing group, and merging metadata with similarity exceeding a set maximum threshold into the initial metadata, which is deemed not valuable for separate storage. This process integrates similar and system data from a large dataset, thereby mitigating the impact of conventional data collection vulnerabilities on computing power.

[0038] In the initial grouping module, the number of initial groups generated is determined by the dataset. The specific steps are as follows:

[0039] Set the maximum capacity L for a single initial packet and the standard data capacity M;

[0040] Get the total data size N of the dataset;

[0041] The number of standard data points S contained in the dataset is calculated by S = N / M;

[0042] The initial number of groups K is calculated using K = L / S.

[0043] During the process of generating the initial group and selecting the top K metadata, if the similarity of the feature terms of any two metadata exceeds a set threshold, the metadata with the lower ranking is determined to be invalid metadata. Then, new metadata is selected to replace the invalid metadata, and the newly selected metadata is the metadata with the highest ranking other than the already selected metadata.

[0044] The specific method for generating the processing group is as follows:

[0045] Retrieve the feature term information of the aforementioned metadata;

[0046] The feature terms are compared sequentially with the labels of different initial groups;

[0047] The initial group corresponding to the tag with the highest similarity to the aforementioned feature term is selected as the metadata processing group.

[0048] During the data filtering process of the processing group, after each comparison of the initial metadata with the new metadata is completed, the initial metadata will be updated and then compared with the next set of metadata.

[0049] During the data filtering process of the processing group, if the similarity between the initial metadata and the metadata does not exceed the threshold, the initial metadata will remain unchanged during this update process.

[0050] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A big data based vulnerability harvesting system, characterized in that, include: The dataset capture module acquires all data within a collection period and generates a dataset, wherein each element within the dataset is defined as metadata. The data preprocessing module collects the feature term information of the metadata, wherein the feature term information is the keywords that appear more than a set threshold in the metadata; The initial grouping module sorts the metadata in the dataset according to the number of feature terms, selects the top-ranked metadata to generate initial groups corresponding to them, and extracts the feature terms of the metadata in the initial groups as the labels of the initial groups. The metadata is defined as initial metadata. The data classification module retrieves metadata from the dataset beyond the initial metadata and divides the metadata into different initial groups according to the similarity between the metadata feature terms and the tags of the initial groups, thus generating processing groups. The data filtering module parses the metadata in the same processing group and compares it with the initial metadata in turn. If the similarity between the two exceeds the set maximum threshold, only the information in the metadata that is different from the initial metadata is retained and merged into the initial metadata to complete the update of the initial metadata. If the similarity between the two does not exceed the set maximum threshold, then the metadata is retained; During the data filtering process of the processing group, if the similarity between the initial metadata and the metadata is lower than the set minimum threshold, the metadata is determined to be metadata to be determined, the metadata to be determined is deleted from the processing group, and the initial metadata of other processing groups are compared in turn, and the metadata to be determined is included in the processing group containing the initial metadata with the highest similarity. If none of the initial metadata has a similarity to the metadata to be determined that exceeds the set minimum threshold, then the metadata to be determined is generated into a separate processing group.

2. The big data based vulnerability collection system according to claim 1, wherein, In the initial grouping module, the number of initial groups generated is determined by the dataset. The specific steps are as follows: Set the maximum capacity L for a single initial packet and the standard data capacity M; Get the total data size N of the dataset; The number of standard data points S contained in the dataset is calculated using S=N / M; The initial number of groups K is calculated using K=L / S.

3. The big data based vulnerability collection system of claim 2, wherein, During the process of generating the initial group and selecting the top K metadata, if the similarity of the feature terms of any two metadata exceeds a set threshold, the metadata with the lower ranking is determined to be invalid metadata. Then, new metadata is selected to replace the invalid metadata, and the newly selected metadata is the metadata with the highest ranking other than the already selected metadata.

4. The vulnerability acquisition system based on big data according to claim 1, characterized in that, The specific method for generating the processing group is as follows: Retrieve the feature term information of the aforementioned metadata; The feature terms are compared sequentially with the labels of different initial groups; The initial group corresponding to the tag with the highest similarity to the aforementioned feature term is selected as the metadata processing group.

5. A vulnerability acquisition system based on big data according to claim 1, characterized in that, During the data filtering process of the processing group, after each comparison of the initial metadata with the new metadata is completed, the initial metadata will be updated and then compared with the next set of metadata.

6. A vulnerability acquisition system based on big data according to claim 5, characterized in that, During the data filtering process of the processing group, if the similarity between the initial metadata and the metadata does not exceed the threshold, the initial metadata will remain unchanged during this update process.

Images