Crawling data processing method and system based on data stream

Abstract

The present invention discloses a crawling data processing method and system based on data flow, wherein the method comprises: crawling target data based on key information pieces to generate a plurality of data items, and transmitting the data items to a first data pipeline; receiving the data items via the first data pipeline, and inputting the data items into corresponding data cleaning functions for cleaning processing according to the categories of the data items, and transmitting the data items that meet the requirements after cleaning to a second data pipeline; creating a plurality of different types of data storage functions, receiving the data items via the second data pipeline, and inputting the data items into corresponding data storage functions for storage query processing according to the categories of the data items, and updating the database according to the query processing results; the above data processing method has a clear logical structure and is easy to expand, and can realize the rapid construction of a data collection project, and is not prone to errors when storing qualified data in the database.

Description

Technical Field

[0001] The present invention relates to the technical field of crawling data processing, and in particular to a crawling data processing method and system based on data stream. Background Art

[0002] With the development of artificial intelligence technology, more and more functions require a large amount of data as support. A large part of enterprise users use crawler tools to collect data and use big data to analyze data. Crawler technology is to crawl data from web pages or device information through certain rules and methods. However, the quality of data collected by crawler tools is far from meeting the requirements for use. Therefore, the data needs to go through a lot of cleaning and warehousing procedures. Often an enterprise or a project needs to collect data information of dozens or even hundreds of dimensions. In the prior art, such multi-task data collection programs generally adopt a parallel processing method, that is, each processing module (such as cleaning, warehousing, etc.) is independent of each other, and each processing module is connected to the database. After any processing module completes its own task, it places the target data in the database. For example, after the cleaning module takes out the data from the database and cleans it, it puts the data that meets the requirements into the database, and the warehousing module extracts the data put in by the cleaning module from the database for processing. For this traditional way of processing crawled data, when there are many types of data, the framework construction work for data collection and processing is relatively large, the management difficulty is also large, and it is easy to have logical confusion and duplicate warehousing. Summary of the invention

[0003] The purpose of the present invention is to provide a data crawling data processing method and system based on data flow, which can quickly build a data collection and processing program framework, has a clear logical structure and is not prone to errors.

[0004] In order to achieve the above object, the present invention discloses a crawling data processing method based on data stream, which comprises:

[0005] Create several key information pieces belonging to different dimensions for crawling data;

[0006] Using a crawler tool to crawl target data based on the key information to generate a plurality of data items, each data item containing a piece of target data, and transmitting the data items to a first data pipeline;

[0007] Create several different types of data cleaning functions;

[0008] The data item is received through the first data pipeline, and is input into a corresponding data cleaning function according to the category of the data item for cleaning, and the cleaned data item that meets the requirements is transmitted to the second data pipeline;

[0009] Create several different types of data storage functions;

[0010] The data item is received via the second data pipeline, and is input into a corresponding data entry function for entry query processing according to the category of the data item, and the database is updated according to the query processing result.

[0011] Preferably, each of the key information pieces includes several data fields, and the field names of the data fields representing the same content in the key information pieces of different dimensions are the same, and the table name and table unique index corresponding to each of the key information pieces are integrated in the same information table for unified management.

[0012] Preferably, when a data item is newly added to the database, all data items of the same category as the newly added data item in the database are sorted as a whole.

[0013] Preferably, the method for overall sorting the data items includes:

[0014] When the data item is processed by the data storage function, the data item that meets the storage condition is transmitted to the third data pipeline;

[0015] Using a data marking function to receive the data item from the third data pipeline, marking the data item, and writing the characteristic name of the data item into redis;

[0016] The data sorting function is used to read the corresponding feature name from Redis, and the tags of the same type of data items in the database are sorted based on the read feature name.

[0017] The present invention also discloses a crawling data processing system based on data stream, which comprises:

[0018] The data preparation module is used to create several key information items belonging to different dimensions for crawling data;

[0019] A data collection module, which is used to crawl the target data based on the key information piece by using a crawler tool to generate a plurality of data items, each of which contains a piece of target data, and transmit the data items to the first data pipeline;

[0020] A data cleaning module, which is used to create several different types of data cleaning functions, receive the data items through the first data pipeline, input the data items into corresponding data cleaning functions according to the categories of the data items for cleaning, and transmit the cleaned data items that meet the requirements to the second data pipeline;

[0021] The data entry module is used to create several different types of data entry functions, receive the data item through the second data pipeline, input the data item into the corresponding data entry function for entry query processing according to the category of the data item, and update the database according to the query processing results.

[0022] Preferably, each of the key information items includes several data fields, and the field names of the data fields representing the same content in the key information items of different dimensions are the same. The data preparation module also integrates the table name and table unique index corresponding to each of the key information items into the same information table for unified management.

[0023] Preferably, it further comprises a data post-processing module, wherein when a data item is newly added to the database, the data post-processing module is used to sort all data items of the same category as the newly added data item in the database as a whole.

[0024] Preferably, the data post-processing module includes a marking module and a sorting module; the marking module is used to receive the data item from the third data pipeline using a data marking function, mark the data item, and write the characteristic name of the data item into redis; the third data pipeline is used to receive data items that meet the storage conditions; the sorting module is used to read the corresponding characteristic name from Redis using a data sorting function, and sort the tags of similar data items in the database based on the read characteristic names.

[0025] The present invention also discloses another crawling data processing system based on data stream, which comprises:

[0026] one or more processors;

[0027] Memory;

[0028] And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include instructions for executing the data stream-based crawling data processing method as described above.

[0029] The present invention also discloses a computer-readable storage medium, which includes a computer program, and the computer program can be executed by a processor to complete the data flow-based crawling data processing method as described above.

[0030] Compared with the prior art, the above technical scheme of the present invention uses the idea of ​​data flow to design the framework of the processing program for processing crawled data, that is, the various processing processes are connected in series, and only the final warehousing process is communicated with the database. In this way, the data to be processed flows from the collection stage to the cleaning stage in a sequential flow manner, and from the cleaning stage to the warehousing stage, and finally the data that meets the requirements is updated to the database in the warehousing stage; it can be seen that the above data processing method has a clear logical structure and is easy to expand, so that a data collection project can be quickly built, and it is not easy to make mistakes when storing qualified data in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the crawling data processing method in an embodiment of the present invention.

[0032] Figure 2 This is a flow chart of the crawling data processing method in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0034] This embodiment discloses a crawling data processing method based on data flow, which is used to crawl data from web pages or other devices through crawler tools. Compared with the parallel alternating processing method in the prior art, this embodiment adopts a layer-by-layer progressive flow method to process data. Specifically, Figure 1 and Figure 2 , the data processing method comprises:

[0035] S1. Data preparation stage: according to project needs, create several key information items belonging to different dimensions for crawling data;

[0036] S2. Entering the data collection stage: using a crawler tool, based on the key information, crawling the target data from the target web page or other device to generate a number of data items (ie, data containers), each data item containing a target data, and transmitting the data item to the first data pipeline;

[0037] S3. Enter the data cleaning phase: First, create several different types of data cleaning functions;

[0038] S4. Then, the data item is received through the first data pipeline, and input into the corresponding data cleaning function according to the category of the data item for cleaning, and the cleaned data item that meets the requirements is transmitted to the second data pipeline;

[0039] S5. Enter the data storage phase: First, create several different types of data storage functions;

[0040] S6. Then, the data item is received through the second data pipeline, and according to the category of the data item, it is input into the corresponding data storage function for storage query processing, and the database is updated according to the query processing result. That is, the database is queried whether there is an object identical to the data in the current data item. If not, the creation time and update time of the current database are initialized, and then the data insertion operation is performed; if the data object already exists, the field values ​​of the new data and the old data are updated if they are inconsistent, and directly skipped if they are consistent. After the updated dictionary is generated, the updated data is added, and then the data update operation is performed.

[0041] The data processing method in this embodiment uses the idea of ​​data flow to design a framework for a processing program for processing crawled data, that is, Figure 1 , connect each processing flow in series, and only connect the last warehousing flow to the database. In this way, the data to be processed flows from the collection stage to the cleaning stage in a sequential manner, and from the cleaning stage to the warehousing stage. Finally, the data that meets the requirements is updated to the database in the warehousing stage. It can be seen that the above data processing method has a clear logical structure and is easy to expand. It can realize the rapid construction of a data collection project, and it is not easy to make mistakes when storing qualified data in the database.

[0042] Furthermore, each of the key information pieces includes several data fields, and the field names of the data fields representing the same content (such as release time, etc.) in the key information pieces of different dimensions are the same, and the table name and table unique index corresponding to each of the key information pieces are integrated in the same information table for unified management to facilitate subsequent unified calls.

[0043] Furthermore, the data processing method in this embodiment also includes a data post-processing stage, that is, when a data item is newly added to the database, all data items of the same category as the newly added data item in the database are sorted as a whole to facilitate subsequent calls.

[0044] Specifically, the method for overall sorting the data items includes:

[0045] First, when the data item is processed by the data storage function, the data item that meets the storage condition is transmitted to the third data pipeline;

[0046] Then, the data item is received from the third data pipeline using a data tagging function, and the data item is tagged, and the characteristic name of the data item is written into redis (a remote dictionary service, which is an open source log-type, Key-Value database written in ANSI C language, supports the network, can be based on memory or can be persistent, and provides APIs in multiple languages); for example, if the data stored in the database is component A, and the version number is 1.0, the characteristic name "component A" is written into redis;

[0047] Next, the data sorting function is used to read the corresponding feature name from Redis, and the tags of the same type of data items in the database are sorted based on the read feature name. For example, if "component A" is read, all component A data are queried in the database. If three data are found, namely component A (version 1.0), component A (version 2.0), and component A (version 3.0), among which component A (version 1.0) and component A (version 2.0) are existing data, the tag sequence number of component A (version 1.0) is 2, and the tag sequence number of component A (version 2.0) is 1 (representing the latest), then, after component A (version 3.0) enters, through the processing of the data sorting function, the tag sequence number of component A (version 1.0) is 3, the tag sequence number of component A (version 2.0) is 2, and the tag sequence number of component A (version 3.0) is 1 (representing the latest).

[0048] In another preferred embodiment of the present invention, a crawling data processing system based on data stream is also disclosed, which includes the following functional modules:

[0049] The data preparation module is used to create several key information items belonging to different dimensions for crawling data;

[0050] A data collection module, which is used to crawl the target data based on the key information piece by using a crawler tool to generate a plurality of data items, each of which contains a piece of target data, and transmit the data items to the first data pipeline;

[0051] A data cleaning module, which is used to create several different types of data cleaning functions, receive the data items through the first data pipeline, input the data items into corresponding data cleaning functions according to the categories of the data items for cleaning, and transmit the cleaned data items that meet the requirements to the second data pipeline;

[0052] The data entry module is used to create several different types of data entry functions, receive the data item through the second data pipeline, input the data item into the corresponding data entry function for entry query processing according to the category of the data item, and update the database according to the query processing results.

[0053] Furthermore, each of the key information pieces includes several data fields, and the field names of the data fields representing the same content in the key information pieces of different dimensions are the same. The data preparation module also integrates the table name and table unique index corresponding to each of the key information pieces into the same information table for unified management.

[0054] Furthermore, the processing system in this embodiment also includes a data post-processing module, and the data post-processing module is used to sort all data items of the same category as the newly added data item in the database when a data item is newly added to the database.

[0055] Specifically, the data post-processing module includes a marking module and a sorting module; the marking module is used to adopt a data marking function to receive the data item from the third data pipeline, mark the data item, and write the characteristic name of the data item into redis; the third data pipeline is used to receive data items that meet the storage conditions; the sorting module is used to adopt a data sorting function to read the corresponding characteristic name from Redis, and sort the tags of similar data items in the database based on the read characteristic name.

[0056] The present invention also discloses another crawling data processing system based on data stream, which includes one or more processors, a memory and one or more programs, wherein one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the program includes instructions for executing the crawling data processing method based on data stream as described above. The processor can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits for executing related programs to implement the functions required to be performed by the modules in the crawling data processing system based on data stream of the embodiment of the present application, or to execute the crawling data processing method based on data stream of the method embodiment of the present application.

[0057] The present invention also discloses a computer-readable storage medium, which includes a computer program, and the computer program can be executed by a processor to complete the crawling data processing method based on data flow as described above. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0058] The embodiment of the present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes the above-mentioned crawling data processing method based on data stream.

[0059] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.

Claims

1. A crawling data processing method based on data stream, characterized in that: include: Create several key information pieces belonging to different dimensions for crawling data; Using a crawler tool to crawl target data based on the key information to generate a plurality of data items, each data item containing a piece of target data, and transmitting the data items to a first data pipeline; Create several different types of data cleaning functions; The data item is received through the first data pipeline, and is input into a corresponding data cleaning function according to the category of the data item for cleaning, and the cleaned data item that meets the requirements is transmitted to the second data pipeline; Create several different types of data storage functions; The data item is received via the second data pipeline, and is input into a corresponding data storage function for storage query processing according to the category of the data item, and the database is updated according to the query processing result; when a data item is newly stored in the database, all data items of the same category as the newly added data item in the database are sorted as a whole; The method for overall sorting the data items includes: When the data item is processed by the data storage function, the data item that meets the storage condition is transmitted to the third data pipeline; Using a data marking function to receive the data item from the third data pipeline, marking the data item, and writing the characteristic name of the data item into redis; The data sorting function is used to read the corresponding feature name from Redis, and the tags of the same type of data items in the database are sorted based on the read feature name.

2. The crawling data processing method based on data stream according to claim 1 is characterized in that: Each of the key information pieces includes several data fields, and the field names of the data fields representing the same content in the key information pieces of different dimensions are the same, and the table name and table unique index corresponding to each of the key information pieces are integrated in the same information table for unified management.

3. A crawling data processing system based on data stream, characterized in that: include: The data preparation module is used to create several key information items belonging to different dimensions for crawling data; A data collection module, which is used to crawl the target data based on the key information piece by using a crawler tool to generate a plurality of data items, each of which contains a piece of target data, and transmit the data items to the first data pipeline; A data cleaning module, which is used to create several different types of data cleaning functions, receive the data items through the first data pipeline, input the data items into corresponding data cleaning functions according to the categories of the data items for cleaning, and transmit the cleaned data items that meet the requirements to the second data pipeline; A data entry module, which is used to create several different types of data entry functions, receive the data item through the second data pipeline, input the data item into the corresponding data entry function according to the category of the data item for entry query processing, and update the database according to the query processing result; A data post-processing module, which is used to sort all data items of the same category as the newly added data item in the database when a data item is newly added to the database; The data post-processing module includes a marking module and a sorting module; the marking module is used to receive the data item from the third data pipeline using a data marking function, mark the data item, and write the characteristic name of the data item into redis; the third data pipeline is used to receive data items that meet the storage conditions; the sorting module is used to read the corresponding characteristic name from Redis using a data sorting function, and sort the tags of similar data items in the database based on the read characteristic names.

4. The crawling data processing system based on data stream according to claim 3 is characterized in that: Each of the key information pieces includes several data fields, and the field names of the data fields representing the same content in the key information pieces of different dimensions are the same. The data preparation module also integrates the table name and table unique index corresponding to each of the key information pieces into the same information table for unified management.

5. A crawling data processing system based on data stream, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include instructions for executing the data stream-based crawling data processing method as described in any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that: It includes a computer program, which can be executed by a processor to complete the data stream-based crawling data processing method as described in any one of claims 1 to 2.

Images