A data collection method and system for multiple data sources

Through the data acquisition method and system for multi-data sources, the IP scheduling module and template multiplexing technology, combined with distributed technology, the problems of IP failure and high development costs in multi-data source scenarios are solved, and efficient data acquisition and Massive data processing.

CN117633329BActive Publication Date: 2024-11-26THE QUARTERMASTER RES INST OF THE GENERAL LOGISTICS DEPT OF THE CPLA
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410110066.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-26
Publication Date
2024-11-26
Estimated Expiration
2044-01-26

AI Technical Summary

Technical Problem

The existing technology has failed to effectively solve the IP failure problem in multi-data source scenarios, resulting in low collection efficiency and high development costs, making it difficult to cope with the challenge of massive data crawling.

Method used

Using data acquisition methods and systems for multiple data sources, IP scheduling module is based on priority sorting IP queues and template multiplexing, combined with distributed technology, multi-threaded concurrent crawling is carried out to ensure IP utilization and acquisition efficiency.

Benefits of technology

It improves the utilization rate of IP, reduces development costs, significantly improves data acquisition efficiency, can effectively process massive data, and concurrently crawl component source code, binary and related data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117633329B_ABST
    Figure CN117633329B_ABST
Patent Text Reader

Abstract

The present invention proposes a data collection method and system for multiple data sources, belonging to the field of data processing technology. The method crawls data from multiple data sources based on a data collection system, and the data collection system includes a task module, a data collection module, an IP scheduling module, a monitoring module, a data cleaning module and a data persistence module. The task module generates data collection tasks and maintains a task queue based on the data from the multiple data sources, and simultaneously obtains the data type of the data from the multiple data sources, and configures a data collection template based on the data type; the data collection module obtains the data collection task from the task queue, selects the configured data collection template, uses the IP address from the IP scheduling module, and crawls valid data from the data from the multiple data sources through a crawler.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data collection method and system for multiple data sources. Background Art

[0002] Multi-data source information collection technology has very important significance and value in the modern information society, which is mainly reflected in the following aspects: Improve data quality and accuracy: By obtaining information from multiple data sources at the same time, the situation of insufficient or inaccurate data caused by the limitation of a single data source can be reduced. Therefore, the use of multi-data source information collection technology can greatly improve data quality and accuracy. Expand data coverage: Different data sources often contain different data types and contents. Multi-data source information collection technology can integrate these data sources, thereby expanding the coverage of data. This is very beneficial for the comprehensiveness and in-depth analysis of information. Improve business response speed: Multi-data source information collection technology can provide more timely, comprehensive and accurate information, enabling enterprises or organizations to make decisions and take actions faster. Promote the intelligent development of enterprises: On the basis of multi-data source information collection technology, data mining, analysis and other technologies can be further applied to improve the efficiency and accuracy of data processing and analysis, and provide support for the intelligent development of enterprises. The crawling of massive multi-data sources has always been a key issue of concern in the industry. When collecting data, the existing technology does not consider the scenario of multi-data sources. Due to the existence of the anti-crawl mechanism, some IPs will become invalid; if an IP is invalidated, it still has the same probability of being selected next time as an IP that has not expired. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a data collection solution for multiple data sources.

[0004] The first aspect of the present invention proposes a data collection method for multiple data sources. The method crawls data from multiple data sources based on a data collection system, and the data collection system includes a task module, a data collection module, an IP scheduling module, a monitoring module, a data cleaning module, and a data persistence module; the method specifically includes:

[0005] Step S1, the task module generates a data collection task according to the data from the multiple data sources and maintains a task queue, and at the same time obtains the data type of the data from the multiple data sources, and configures a data collection template based on the data type;

[0006] Step S2, the data collection module obtains the data collection task from the task queue, selects the configured data collection template, uses the IP address from the IP scheduling module, and crawls valid data from the data from multiple data sources through a crawler;

[0007] The IP scheduling module maintains an IP queue for each data source, and each IP address in the IP queue is sorted in descending order based on priority, and each IP address comes from an IP pool;

[0008] Wherein, the monitoring module monitors the running status of the crawler and periodically checks the running status information of the crawler;

[0009] Step S3: The data cleaning module cleans the valid data using a data cleaning method corresponding to the data type to obtain structured information, and the structured information is stored in the data persistence module.

[0010] According to the method of the first aspect of the present invention, in step S1, the execution logic of the task module is single-threaded, wherein:

[0011] The task module enters the data from multiple data sources into the task list through a script;

[0012] Setting a data collection template according to the data type of the data from the multiple data sources, wherein if the data type of the data from the multiple data sources matches the data collection template in the template library, calling the data collection template from the template library, otherwise generating a new data collection template for the data from the multiple data sources, and storing the new data collection template in the template library;

[0013] The task list is read, corresponding data collection tasks are generated according to data features of the data from multiple data sources, and the data collection tasks are written into the task queue.

[0014] According to the method of the first aspect of the present invention, in step S2, the execution logic of the data acquisition module is multi-threaded concurrent, wherein:

[0015] The data collection module obtains multiple data collection tasks from the task queue in a multi-threaded concurrent manner, and enables multiple docker container crawlers in batches;

[0016] Each of the crawlers selects the matching data collection template for the corresponding data collection task, and calls the IP scheduling module to obtain the IP to crawl the valid data;

[0017] For each data source of the data from multiple data sources, the crawler selects the IP with the highest priority from the IP queue maintained by each data source to perform the data crawling operation.

[0018] According to the method of the first aspect of the present invention, in step S2:

[0019] After the IP with the highest priority is called by the crawler, the IP scheduling module reduces the priority of the IP by one level and puts it back into the IP queue to which it belongs;

[0020] The IP scheduling module determines whether the crawler successfully performs the data crawling operation by calling the IP with the highest priority;

[0021] If so, maintain the priority of the IP address when it is put back into the IP queue;

[0022] If not, the priority of the IP is reduced by m levels, where m=k*P, k represents the coefficient, and its value range is [0.15-0.35], and P represents the initial priority of the IP.

[0023] According to the method of the first aspect of the present invention, in step S2, when the IP scheduling module constructs the IP queue for each data source, it extracts several IP addresses corresponding to the data source from the IP pool, assigns the same initial priority P to the several IPs, and the initial sorting of the several IPs is random sorting.

[0024] According to the method of the first aspect of the present invention, in step S3, the data cleaning module cleans the valid data specifically including:

[0025] When the data type of the valid data is a license, extract the license name, compare the license name with the standard license name in the license library, determine whether the license is included, and if not, extract the license name, license content, and license issuance date to form the structured information;

[0026] When the data type of the valid data is vulnerability, extract the CVE vulnerability number, compare the CVE vulnerability number with the standard number in the vulnerability number library, determine whether the CVE vulnerability number is included, and if not, extract the vulnerability content and URL to form the structured information;

[0027] When the data type of the valid data is source code, the source code package name is normalized, and source code related information is extracted, including project name, version name, and file hash, to form the structured information;

[0028] When the data type of the valid data is a binary file, a hash value of the binary file is calculated, the binary file is hashed and renamed, and relevant information of the binary file, including the project name, version name, and the hash value, is extracted to form the structured information.

[0029] According to the method of the first aspect of the present invention, in step S3, the data persistence module stores the structured information in a Mysql database, stores the source code package in a corresponding path according to its component ID, and stores the binary file in a corresponding path with its hash value as ID.

[0030] The second aspect of the present invention provides a data acquisition system for multiple data sources. The data acquisition system crawls data from multiple data sources, and the data acquisition system includes a task module, a data acquisition module, an IP scheduling module, a monitoring module, a data cleaning module, and a data persistence module; wherein:

[0031] The task module generates a data collection task according to the data from the multiple data sources and maintains a task queue, and simultaneously obtains the data type of the data from the multiple data sources, and configures a data collection template based on the data type;

[0032] The data collection module obtains the data collection task from the task queue, selects the configured data collection template, uses the IP address from the IP scheduling module, and crawls valid data from the data from multiple data sources through a crawler;

[0033] The IP scheduling module maintains an IP queue for each data source, and each IP address in the IP queue is sorted in descending order based on priority, and each IP address comes from an IP pool;

[0034] Wherein, the monitoring module monitors the running status of the crawler and periodically checks the running status information of the crawler;

[0035] The data cleaning module cleans the valid data using a data cleaning method corresponding to the data type to obtain structured information, and the structured information is stored in the data persistence module.

[0036] According to the system of the second aspect of the present invention, the execution logic of the task module is single-threaded, wherein:

[0037] The task module enters the data from multiple data sources into the task list through a script;

[0038] Setting a data collection template according to the data type of the data from the multiple data sources, wherein if the data type of the data from the multiple data sources matches the data collection template in the template library, calling the data collection template from the template library, otherwise generating a new data collection template for the data from the multiple data sources, and storing the new data collection template in the template library;

[0039] The task list is read, corresponding data collection tasks are generated according to data features of the data from multiple data sources, and the data collection tasks are written into the task queue.

[0040] According to the system of the second aspect of the present invention, the execution logic of the data acquisition module is multi-threaded concurrent, wherein:

[0041] The data collection module obtains multiple data collection tasks from the task queue in a multi-threaded concurrent manner, and enables multiple docker container crawlers in batches;

[0042] Each of the crawlers selects the matching data collection template for the corresponding data collection task, and calls the IP scheduling module to obtain the IP to crawl the valid data;

[0043] For each data source of the data from multiple data sources, the crawler selects the IP with the highest priority from the IP queue maintained by each data source to perform the data crawling operation.

[0044] According to the system of the second aspect of the present invention, after the IP with the highest priority is called by the crawler, the IP scheduling module reduces the priority of the IP by one level and puts it back into the IP queue to which it belongs; the IP scheduling module determines whether the crawler successfully calls the IP with the highest priority to perform the data crawling operation; if so, maintains the priority of the IP address when it is put back into the IP queue; if not, reduces the priority of the IP by m levels, m=k*P, k represents a coefficient, and the value range is [0.15-0.35], and P represents the initial priority of the IP.

[0045] According to the system of the second aspect of the present invention, when the IP scheduling module constructs the IP queue for each data source, it extracts several IP addresses corresponding to the data source from the IP pool, assigns the same initial priority P to the several IPs, and the initial sorting of the several IPs is random sorting.

[0046] According to the system of the second aspect of the present invention, the data cleaning module cleans the valid data specifically including:

[0047] When the data type of the valid data is a license, extract the license name, compare the license name with the standard license name in the license library, determine whether the license is included, and if not, extract the license name, license content, and license issuance date to form the structured information;

[0048] When the data type of the valid data is vulnerability, extract the CVE vulnerability number, compare the CVE vulnerability number with the standard number in the vulnerability number library, determine whether the CVE vulnerability number is included, and if not, extract the vulnerability content and URL to form the structured information;

[0049] When the data type of the valid data is source code, the source code package name is normalized, and source code related information is extracted, including project name, version name, and file hash, to form the structured information;

[0050] When the data type of the valid data is a binary file, a hash value of the binary file is calculated, the binary file is hashed and renamed, and relevant information of the binary file, including the project name, version name, and the hash value, is extracted to form the structured information.

[0051] According to the system of the second aspect of the present invention, the data persistence module stores the structured information in a Mysql database, stores the source code package in a corresponding path according to its component ID, and stores the binary file in a corresponding path with its hash value as ID.

[0052] The third aspect of the present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the data collection method for multiple data sources described in the first aspect of the present disclosure are implemented.

[0053] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data collection method for multiple data sources described in the first aspect of the present disclosure.

[0054] In summary, the technical solution proposed in the present invention can effectively improve the utilization rate of IP in the scenario of multiple data sources, thereby improving the collection efficiency. At the same time, based on the template method, the present invention effectively reduces the problems of high development cost, long development cycle and changeable demand caused by the addition of new data sources through template reuse and template incremental modification, and significantly improves the data collection efficiency. In order to cope with the problem of large data volume, the system is based on the use of distributed technology, which can integrate multiple node resources and ensure that even in the face of massive data, the hardware resources are always sufficient when crawling. Using a distributed approach, it is effective to execute concurrently on multiple nodes, which greatly improves the efficiency of crawling and saves the time cost required for data collection. The present invention can effectively collect various data such as component source code, binary and its related licenses, vulnerabilities, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 Schematic diagram of a data collection process for multiple data sources according to an embodiment of the present invention.

[0057] Figure 2 Schematic diagram of the IP pool priority scheduling process according to an embodiment of the present invention.

[0058] Figure 3 A schematic diagram of a template calling process according to an embodiment of the present invention.

[0059] Figure 4 Schematic diagram of a distributed collection process according to an embodiment of the present invention.

[0060] Figure 5 Schematic diagram of the task module execution process according to an embodiment of the present invention.

[0061] Figure 6 It is a schematic diagram of the execution process of the data acquisition module according to an embodiment of the present invention.

[0062] Figure 7 Schematic diagram of the execution process of the IP scheduling module according to an embodiment of the present invention.

[0063] Figure 8 Schematic diagram of the execution process of the monitoring module according to an embodiment of the present invention.

[0064] Fig. 9 Schematic diagram of the execution process of the data cleaning module according to an embodiment of the present invention.

[0065] Fig.10 The figure is a schematic diagram of the execution process of the data persistence module according to an embodiment of the present invention.

[0066] Fig.11 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0068] With the continuous development of the open source movement, the number of open source components is expanding rapidly. At the same time, various hosting platforms and related information platforms, such as CVE, NVD, GitHub, etc., are also developing rapidly. The growth in the number of platforms and projects has brought considerable challenges to data collection.

[0069] In order to cope with the scenario of massive multi-data source collection, the present invention proposes a data collection solution for multiple data sources, which can meet the crawling problem of massive multi-data sources with high efficiency and high quality. It can effectively improve the utilization rate of IP in the multi-data source scenario, thereby improving the collection efficiency. At the same time, based on the template method, the present invention effectively reduces the problems of high development cost, long development cycle and changeable demand caused by the addition of new data sources through template reuse and template incremental modification, and significantly improves the data collection efficiency. In order to cope with the problem of large data volume, the system is based on the use of distributed technology, which can integrate multiple node resources and ensure that even in the face of massive data, the hardware resources are always sufficient when crawling. Using a distributed approach, it is effectively executed concurrently on multiple nodes, which greatly improves the efficiency of crawling and saves the time cost required for data collection. The present invention can effectively collect component source code, binary and its related licenses, vulnerabilities and other data.

[0070] When performing data collection, the data source itself often has an anti-crawling mechanism to limit the data collection system from collecting data. The present invention uses a proxy IP pool to collect multiple IPs and put them into the proxy pool. When performing data collection, the IP is extracted from the IP pool and the extracted IP is used for collection. The present invention takes into account the scenario of multiple data sources. Due to the existence of the anti-crawling mechanism, some IPs will be invalidated. However, the IP that is invalidated in data source A may be valid in data source B. Therefore, the differences between different data sources should be considered during scheduling. In addition, the concept of priority is used during scheduling; if an IP is invalidated, it is avoided that it has the same probability as the non-invalidated IP to be selected the next time.

[0071] The first aspect of the present invention provides a data collection method for multiple data sources. The method crawls data from multiple data sources based on a data collection system, and the data collection system includes a task module, a data collection module, an IP scheduling module, a monitoring module, a data cleaning module, and a data persistence module; Figure 1 As shown, the method specifically includes:

[0072] Step S1, the task module generates a data collection task according to the data from the multiple data sources and maintains a task queue, and at the same time obtains the data type of the data from the multiple data sources, and configures a data collection template based on the data type;

[0073] Step S2, the data collection module obtains the data collection task from the task queue, selects the configured data collection template, uses the IP address from the IP scheduling module, and crawls valid data from the data from multiple data sources through a crawler;

[0074] The IP scheduling module maintains an IP queue for each data source, and each IP address in the IP queue is sorted in descending order based on priority, and each IP address comes from an IP pool;

[0075] Wherein, the monitoring module monitors the running status of the crawler and periodically checks the running status information of the crawler;

[0076] Step S3: The data cleaning module cleans the valid data using a data cleaning method corresponding to the data type to obtain structured information, and the structured information is stored in the data persistence module.

[0077] Specifically, Figure 2 As shown, the present invention uses a shared IP pool to save all IP-related meta-information, and maintains different priority queues for different data sources to ensure different scheduling priorities. When updating the priority, each IP is used once, and its priority will be reduced by a small step, such as from 20 to 19; each IP is found to be invalid during use, and its priority will be reduced by a large step, such as from 20 to 10. In this way, it can be guaranteed that the invalid IP will not be used for a period of time in the future. After a period of time, the invalid IP is called again, and if it is valid, it will continue to be used. If it is invalid, the priority will be further reduced. The reason for this is that the failure of the IP is often due to the limitation of the anti-crawling mechanism or the fluctuation of the agent network. These problems can be solved by standing still for a period of time.

[0078] Specifically, in a multi-data source scenario, there are many data sources and their formats are often not uniform. When collecting data, it is often necessary to use different crawling methods to parse different data sources. Figure 3As shown, the present invention adopts a template method to precipitate a general parsing method into a template. When in use, a specific data source is parsed based on a general template, so that when a new data source is added, not only can the development workload be reduced, but also the development efficiency is greatly improved. Specifically, first determine the three types of basic templates: source code acquisition template, binary acquisition template, and text acquisition template. When encountering new data, determine whether the acquisition template needs to be customized. If customization is required, modify it based on the existing template, or create a new template, and collect data through the newly added template. If the template already exists and does not need to be customized, further perform template matching. The strategy of template reuse effectively improves development efficiency.

[0079] Specifically, the present invention is directed to distributed collection, such as Figure 4 As shown, a single node is used to generate a data collection task first, and multiple nodes read the task and execute it. This method of decoupling task generation and execution enables the data collection system to combine the advantages of multiple resource nodes and effectively exert the capabilities of multiple nodes, so that the invention can be effectively applied to scenarios with massive data.

[0080] Specifically, the task module is mainly used to generate different targeted tasks for multiple data sources, such as component data, vulnerability data, etc., and put these tasks into the task queue. The data acquisition module obtains tasks from the task queue, selects the corresponding acquisition template according to the characteristics of the task data source, generates a specific crawler, and crawls the corresponding data. The crawled data is input into the data cleaning module to obtain useful data information after data cleaning. The data persistence module saves the data information to the persistent medium (file system and MySQL database). When executing the data acquisition task, the administrator needs to set the template and generate the task according to the characteristics of the data source through the task module and put it into the task queue. Then the data acquisition module will read the task and obtain the corresponding template, and use the crawler to crawl the data source data according to the template. When the crawler crawls the data, it will select the IP from the IP pool using the IP scheduling method oriented to the data source for directional crawling. When the crawler crawls the data, the monitoring module will monitor the status of the crawler in real time. After the crawler crawls the data source data, the data cleaning module will clean the data and hand over the cleaned data to the data persistence module for persistence and storage in the file system and database.

[0081] According to the method of the first aspect of the present invention, in step S1, the execution logic of the task module is single-threaded, wherein:

[0082] The task module enters the data from multiple data sources into the task list through a script;

[0083] Setting a data collection template according to the data type of the data from the multiple data sources, wherein if the data type of the data from the multiple data sources matches the data collection template in the template library, calling the data collection template from the template library, otherwise generating a new data collection template for the data from the multiple data sources, and storing the new data collection template in the template library;

[0084] The task list is read, corresponding data collection tasks are generated according to data features of the data from multiple data sources, and the data collection tasks are written into the task queue.

[0085] Specifically, the task module is used to generate tasks from the task list and write the tasks into the task queue for consumption by the data acquisition module. Figure 5 As shown: the system administrator enters the information of the data source to be crawled into the task list through the script; the system administrator sets the template or updates the template according to the data source information; the task module reads the task list and generates corresponding tasks according to the characteristics of different data sources; and the generated tasks are written into the task queue.

[0086] The entire data collection system separates the execution and generation logic of data crawling tasks through the task queue method. The advantage of this is that the task module is single-threaded, so there will be no duplication and competition when generating tasks. The execution logic of data crawling tasks is massively concurrent, which can maximize the efficiency of data crawling. This method of decoupling task generation and execution enables the data collection system to adapt to scenarios with massive data.

[0087] According to the method of the first aspect of the present invention, in step S2, the execution logic of the data acquisition module is multi-threaded concurrent, wherein:

[0088] The data collection module obtains multiple data collection tasks from the task queue in a multi-threaded concurrent manner, and enables multiple docker container crawlers in batches;

[0089] Each of the crawlers selects the matching data collection template for the corresponding data collection task, and calls the IP scheduling module to obtain the IP to crawl the valid data;

[0090] For each data source of the data from multiple data sources, the crawler selects the IP with the highest priority from the IP queue maintained by each data source to perform the data crawling operation.

[0091] Specifically, the data collection module is used to perform data collection tasks. Based on k8s and docker technologies, this module concurrently starts and runs crawlers on the cluster, and can efficiently crawl massive amounts of data. Figure 6As shown, first start the docker container crawler, connect and configure the Redis core database, obtain the required core parameter information such as username and password from the database, and automatically generate a configuration file; start the daemon process to obtain tasks (mainly github, image repository and CVE website tasks); request the k8s (abbreviation of kuberneters) service to batch start multiple docker container crawlers; the crawler selects different templates according to the task type; calls the IP scheduling module to obtain the IP; uses the IP and template to crawl data and obtain data from multiple data sources.

[0092] According to the method of the first aspect of the present invention, in step S2:

[0093] After the IP with the highest priority is called by the crawler, the IP scheduling module reduces the priority of the IP by one level and puts it back into the IP queue to which it belongs;

[0094] The IP scheduling module determines whether the crawler successfully performs the data crawling operation by calling the IP with the highest priority;

[0095] If so, maintain the priority of the IP address when it is put back into the IP queue;

[0096] If not, the priority of the IP is reduced by m levels, where m=k*P, k represents the coefficient, and its value range is [0.15-0.35], and P represents the initial priority of the IP.

[0097] According to the method of the first aspect of the present invention, in step S2, when the IP scheduling module constructs the IP queue for each data source, it extracts several IP addresses corresponding to the data source from the IP pool, assigns the same initial priority P to the several IPs, and the initial sorting of the several IPs is random sorting.

[0098] Specifically, the IP scheduling module is used to obtain IP based on the IP pool, such as Figure 7 As shown: Determine whether the data source is a new data source; if it is a new data source, create a priority scheduling queue for this data source, set the IP priority in the queue to 100, and the order is random. Return the IP at the head of the queue and reduce the priority by 1; if it is not a new data source, return the IP at the head of the queue and reduce the priority by 1; determine whether the IP used is valid for the data source. If the IP is invalid, reduce the priority by 20.

[0099] Specifically, the monitoring module, such as Figure 8 As shown, the crawler's running status will be monitored, crawler status information will be pulled regularly, and displayed visually.

[0100] According to the method of the first aspect of the present invention, in step S3, the data cleaning module cleans the valid data specifically including:

[0101] When the data type of the valid data is a license, extract the license name, compare the license name with the standard license name in the license library, determine whether the license is included, and if not, extract the license name, license content, and license issuance date to form the structured information;

[0102] When the data type of the valid data is vulnerability, extract the CVE vulnerability number, compare the CVE vulnerability number with the standard number in the vulnerability number library, determine whether the CVE vulnerability number is included, and if not, extract the vulnerability content and URL to form the structured information;

[0103] When the data type of the valid data is source code, the source code package name is normalized, and source code related information is extracted, including project name, version name, and file hash, to form the structured information;

[0104] When the data type of the valid data is a binary file, a hash value of the binary file is calculated, the binary file is hashed and renamed, and relevant information of the binary file, including the project name, version name, and the hash value, is extracted to form the structured information.

[0105] Specifically, the data cleaning module is used to clean the crawled data from multiple data sources. It normalizes the storage format and associates the collected basic information such as project source code, language, version, manufacturer, source, etc., establishes associations and mappings such as project-component-version-file-function (parameter), establishes an association with the project code, and finally establishes a unified and standard data format to obtain unified visual data. Data cleaning includes two functions: one is to analyze the data of the data source, sort out the conversion rules and workflows; the other is to analyze the data components, extract valid information, and remove dirty data. Fig. 9 As shown: Determine the data type; where:

[0106] If the data type is license, then: extract the license name, compare the license name with the license names in the library to determine whether the license has been included; if not included, extract relevant structured information such as the license name, license content, and license issuance date.

[0107] If the data type is vulnerability, then: extract the CVE vulnerability number, and compare the number with the number in the library to determine whether the vulnerability has been included; if it has not been included, extract the CVE content, URL and other related structured information.

[0108] If the data type is source code, then: normalize the source code package name, such as "2.0-1" is normalized to "2.0_1"; extract source code related information, such as project name, version name, file hash and other structured information.

[0109] If the data type is a binary file, then: calculate the binary file hash, hash the file, and rename the file; extract binary-related information, such as project name, version name, file hash, and other structured information.

[0110] According to the method of the first aspect of the present invention, in step S3, the data persistence module stores the structured information in a Mysql database, stores the source code package in a corresponding path according to its component ID, and stores the binary file in a corresponding path with its hash value as ID.

[0111] Specifically, the data persistence module is used to persist the cleaned data. The main storage systems are the file system and the MySQL database. Fig.10 As shown in the figure, structured information, such as project name, developer information, version information, file hash, resource files, dependent components and other related information are saved to mysql. The source code package is saved according to the component id and saved to the corresponding path. The binary file is saved to the corresponding path with the file hash value as the id.

[0112] The second aspect of the present invention provides a data acquisition system for multiple data sources. The data acquisition system crawls data from multiple data sources, and the data acquisition system includes a task module, a data acquisition module, an IP scheduling module, a monitoring module, a data cleaning module, and a data persistence module; wherein:

[0113] The task module generates a data collection task according to the data from the multiple data sources and maintains a task queue, and simultaneously obtains the data type of the data from the multiple data sources, and configures a data collection template based on the data type;

[0114] The data collection module obtains the data collection task from the task queue, selects the configured data collection template, uses the IP address from the IP scheduling module, and crawls valid data from the data from multiple data sources through a crawler;

[0115] The IP scheduling module maintains an IP queue for each data source, and each IP address in the IP queue is sorted in descending order based on priority, and each IP address comes from an IP pool;

[0116] Wherein, the monitoring module monitors the running status of the crawler and periodically checks the running status information of the crawler;

[0117] The data cleaning module cleans the valid data using a data cleaning method corresponding to the data type to obtain structured information, and the structured information is stored in the data persistence module.

[0118] According to the system of the second aspect of the present invention, the execution logic of the task module is single-threaded, wherein:

[0119] The task module enters the data from multiple data sources into the task list through a script;

[0120] Setting a data collection template according to the data type of the data from the multiple data sources, wherein if the data type of the data from the multiple data sources matches the data collection template in the template library, calling the data collection template from the template library, otherwise generating a new data collection template for the data from the multiple data sources, and storing the new data collection template in the template library;

[0121] The task list is read, corresponding data collection tasks are generated according to data features of the data from multiple data sources, and the data collection tasks are written into the task queue.

[0122] According to the system of the second aspect of the present invention, the execution logic of the data acquisition module is multi-threaded concurrent, wherein:

[0123] The data collection module obtains multiple data collection tasks from the task queue in a multi-threaded concurrent manner, and enables multiple docker container crawlers in batches;

[0124] Each of the crawlers selects the matching data collection template for the corresponding data collection task, and calls the IP scheduling module to obtain the IP to crawl the valid data;

[0125] For each data source of the data from multiple data sources, the crawler selects the IP with the highest priority from the IP queue maintained by each data source to perform the data crawling operation.

[0126] According to the system of the second aspect of the present invention, after the IP with the highest priority is called by the crawler, the IP scheduling module reduces the priority of the IP by one level and puts it back into the IP queue to which it belongs; the IP scheduling module determines whether the crawler successfully calls the IP with the highest priority to perform the data crawling operation; if so, maintains the priority of the IP address when it is put back into the IP queue; if not, reduces the priority of the IP by m levels, m=k*P, k represents a coefficient, and the value range is [0.15-0.35], and P represents the initial priority of the IP.

[0127] According to the system of the second aspect of the present invention, when the IP scheduling module constructs the IP queue for each data source, it extracts several IP addresses corresponding to the data source from the IP pool, assigns the same initial priority P to the several IPs, and the initial sorting of the several IPs is random sorting.

[0128] According to the system of the second aspect of the present invention, the data cleaning module cleans the valid data specifically including:

[0129] When the data type of the valid data is a license, extract the license name, compare the license name with the standard license name in the license library, determine whether the license is included, and if not, extract the license name, license content, and license issuance date to form the structured information;

[0130] When the data type of the valid data is vulnerability, extract the CVE vulnerability number, compare the CVE vulnerability number with the standard number in the vulnerability number library, determine whether the CVE vulnerability number is included, and if not, extract the vulnerability content and URL to form the structured information;

[0131] When the data type of the valid data is source code, the source code package name is normalized, and source code related information is extracted, including project name, version name, and file hash, to form the structured information;

[0132] When the data type of the valid data is a binary file, a hash value of the binary file is calculated, the binary file is hashed and renamed, and relevant information of the binary file, including the project name, version name, and the hash value, is extracted to form the structured information.

[0133] According to the system of the second aspect of the present invention, the data persistence module stores the structured information in a Mysql database, stores the source code package in a corresponding path according to its component ID, and stores the binary file in a corresponding path with its hash value as ID.

[0134] The third aspect of the present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the data collection method for multiple data sources described in the first aspect of the present disclosure are implemented.

[0135] Fig.11 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Fig.11 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0136] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data collection method for multiple data sources described in the first aspect of the present disclosure.

[0138] In summary, the technical solution proposed in the present invention can effectively improve the utilization rate of IP in the scenario of multiple data sources, thereby improving the collection efficiency. At the same time, based on the template method, the present invention effectively reduces the problems of high development cost, long development cycle and changeable demand caused by the addition of new data sources through template reuse and template incremental modification, and significantly improves the data collection efficiency. In order to cope with the problem of large data volume, the system is based on the use of distributed technology, which can integrate multiple node resources and ensure that even in the face of massive data, the hardware resources are always sufficient when crawling. Using a distributed approach, it is effective to execute concurrently on multiple nodes, which greatly improves the efficiency of crawling and saves the time cost required for data collection. The present invention can effectively collect various data such as component source code, binary and its related licenses, vulnerabilities, etc.

[0139] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A data collection method for multiple data sources, Features: The method crawls data from multiple data sources based on a data acquisition system, wherein the data acquisition system includes a task module, a data acquisition module, an IP scheduling module, a monitoring module, a data cleaning module and a data persistence module; The method specifically comprises: Step S1, the task module generates a data collection task according to the data from the multiple data sources and maintains a task queue, and at the same time obtains the data type of the data from the multiple data sources, and configures a data collection template based on the data type; Step S2, the data collection module obtains the data collection task from the task queue, selects the configured data collection template, uses the IP address from the IP scheduling module, and crawls valid data from the data from multiple data sources through a crawler; The IP scheduling module maintains an IP queue for each data source, and each IP address in the IP queue is sorted in descending order based on priority, and each IP address comes from an IP pool; Wherein, the monitoring module periodically checks the running status information of the crawler to monitor the running status of the crawler; Wherein, in step S2, the execution logic of the data acquisition module is multi-threaded concurrent, wherein: The data collection module obtains multiple data collection tasks from the task queue in a multi-threaded concurrent manner, and enables multiple docker container crawlers in batches; Each of the crawlers selects the matching data collection template for the corresponding data collection task, and calls the IP scheduling module to obtain the IP address to crawl the valid data; Wherein, for each data source of the data from the multiple data sources, the crawler selects the IP address with the highest priority from the IP queue maintained by each data source to perform the data crawling operation; Wherein, in step S2: After the IP address with the highest priority is called by the crawler, the IP scheduling module reduces the priority of the IP address by one level and puts it back into the IP queue to which it belongs; The IP scheduling module determines whether the crawler successfully performs the data crawling operation by calling the IP address with the highest priority; If so, maintain the priority of the IP address when it is put back into the IP queue; If not, the priority of the IP address is reduced by m levels, where m = k*P, k represents a coefficient in the range of [0.15-0.35], and P represents the initial priority of the IP address; Step S3: the data cleaning module cleans the valid data using a data cleaning method corresponding to the data type to obtain structured information, and the structured information is stored in the data persistence module; Wherein, in the step S3, the data cleaning module cleans the valid data specifically including: When the data type of the valid data is a license, extract the license name, compare the license name with the standard license name in the license library, determine whether the license is included, and if not, extract the license name, license content, and license issuance date to form the structured information; When the data type of the valid data is vulnerability, extract the CVE vulnerability number, compare the CVE vulnerability number with the standard number in the vulnerability number library, determine whether the CVE vulnerability number is included, and if not, extract the vulnerability content and URL to form the structured information; When the data type of the valid data is source code, the source code package name is normalized, and source code related information is extracted, including project name, version name, and file hash, to form the structured information; When the data type of the valid data is a binary file, a hash value of the binary file is calculated, the binary file is hashed and renamed, and relevant information of the binary file, including the project name, version name, and the hash value, is extracted to form the structured information.

2. A data collection method for multiple data sources according to claim 1, It is characterized in that In step S1, the execution logic of the task module is single-threaded, wherein: The task module enters the data from multiple data sources into a task list; Setting a data collection template according to the data type of the data from the multiple data sources, wherein if the data type of the data from the multiple data sources matches the data collection template in the template library, calling the data collection template from the template library, otherwise generating a new data collection template for the data from the multiple data sources, and storing the new data collection template in the template library; The task list is read, corresponding data collection tasks are generated according to data features of the data from multiple data sources, and the data collection tasks are written into the task queue.

3. A data collection method for multiple data sources according to claim 2, It is characterized in that In step S2, when constructing the IP queue for each data source, the IP scheduling module extracts several IP addresses corresponding to the data source from the IP pool, assigns the same initial priority P to the several IP addresses, and the initial sorting of the several IP addresses is random.

4. A data collection method for multiple data sources according to claim 3, It is characterized in that In the step S3, the data persistence module stores the structured information in a Mysql database, stores the source code package in a corresponding path according to its component ID, and stores the binary file in a corresponding path with its hash value as ID.

5. A data collection system for multiple data sources, It is characterized in that The data acquisition system crawls data from multiple data sources, and the data acquisition system includes a task module, a data acquisition module, an IP scheduling module, a monitoring module, a data cleaning module and a data persistence module; wherein: The task module generates a data collection task according to the data from the multiple data sources and maintains a task queue, and simultaneously obtains the data type of the data from the multiple data sources, and configures a data collection template based on the data type; The data collection module obtains the data collection task from the task queue, selects the configured data collection template, uses the IP address from the IP scheduling module, and crawls valid data from the data from multiple data sources through a crawler; The IP scheduling module maintains an IP queue for each data source, and each IP address in the IP queue is sorted in descending order based on priority, and each IP address comes from an IP pool; Wherein, the monitoring module periodically checks the running status information of the crawler to monitor the running status of the crawler; The execution logic of the data acquisition module is multi-threaded concurrent, where: The data collection module obtains multiple data collection tasks from the task queue in a multi-threaded concurrent manner, and enables multiple docker container crawlers in batches; Each of the crawlers selects the matching data collection template for the corresponding data collection task, and calls the IP scheduling module to obtain the IP address to crawl the valid data; Wherein, for each data source of the data from the multiple data sources, the crawler selects the IP address with the highest priority from the IP queue maintained by each data source to perform the data crawling operation; in: After the IP address with the highest priority is called by the crawler, the IP scheduling module reduces the priority of the IP address by one level and puts it back into the IP queue to which it belongs; The IP scheduling module determines whether the crawler successfully performs the data crawling operation by calling the IP address with the highest priority; If so, maintain the priority of the IP address when it is put back into the IP queue; If not, the priority of the IP address is reduced by m levels, where m = k*P, k represents a coefficient in the range of [0.15-0.35], and P represents the initial priority of the IP address; The data cleaning module cleans the valid data using a data cleaning method corresponding to the data type to obtain structured information, and the structured information is stored in the data persistence module; The data cleaning module cleans the valid data specifically including: When the data type of the valid data is a license, extract the license name, compare the license name with the standard license name in the license library, determine whether the license is included, and if not, extract the license name, license content, and license issuance date to form the structured information; When the data type of the valid data is vulnerability, extract the CVE vulnerability number, compare the CVE vulnerability number with the standard number in the vulnerability number library, determine whether the CVE vulnerability number is included, and if not, extract the vulnerability content and URL to form the structured information; When the data type of the valid data is source code, the source code package name is normalized, and source code related information is extracted, including project name, version name, and file hash, to form the structured information; When the data type of the valid data is a binary file, a hash value of the binary file is calculated, the binary file is hashed and renamed, and relevant information of the binary file, including the project name, version name, and the hash value, is extracted to form the structured information.

6. An electronic device, It is characterized in that The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the data collection method for multiple data sources described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data collection method for multiple data sources described in any one of claims 1 to 4 are implemented.

Citation Information

Cited By

  • Multi-data-source intelligent student psychological data acquisition and analysis system

    CN120407661A

  • An intelligent student psychological data collection and analysis system with multiple data sources

    CN120407661B