Data crawling method, system and device and storage medium

Abstract

The invention provides a data crawling method, system and device and a storage medium, and the method comprises the steps: obtaining site information of a target website, and configuring a corresponding crawler script according to the site information; generating a corresponding crawling task according to the crawler script; executing the crawling task to obtain target data in the target website; storing the target data, and generating a target data index; and exporting the target data by utilizing a data export interface according to the target data index. According to the method, the corresponding crawler script can be created only through the site information, then the crawling task is executed through the crawler script, data wanted by a user can be obtained, and the data obtaining cost is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and in particular to a data crawling method, system, device and storage medium. Background Art

[0002] Knowledge Graph, also known as knowledge domain visualization or knowledge domain mapping map, is a series of various graphics that show the knowledge development process and structural relationship. It uses visualization technology to describe knowledge resources and their carriers, and mine, analyze, construct, draw and display knowledge and their mutual connections. With the development of network technology, more and more companies are keen to build knowledge graphs for intelligent search, text analysis and machine reading. However, the data provided by the company itself often cannot meet the requirements of knowledge graph construction, and the existing data crawling methods often have high time and economic costs, which are not suitable for the development needs of enterprises. Summary of the invention

[0003] In view of this, the purpose of the present invention is to overcome the deficiencies in the prior art and to provide a data crawling method, system, device and storage medium.

[0004] The present invention provides the following technical solutions:

[0005] In a first aspect, the present application provides a data crawling method, comprising:

[0006] Obtain the site information of the target website and configure the corresponding crawler script according to the site information;

[0007] Generate a corresponding crawling task according to the crawler script;

[0008] Execute the crawling task to obtain target data from the target website;

[0009] The target data is saved and a target data index is generated;

[0010] According to the target data index, the target data is exported using a data export interface.

[0011] In one implementation, the step of obtaining the site information of the target website and configuring a corresponding crawler script according to the site information includes:

[0012] Obtaining site information of the target website, wherein the site information includes a site ID, a site name, a site description, and a site address;

[0013] Determine script information of the crawler script according to the site information, the script information including crawling start time, interval type and interval time;

[0014] The crawler script is configured according to the script information.

[0015] In one implementation, generating a corresponding crawling task according to the crawler script includes:

[0016] Generate a corresponding time expression according to the crawling start time, the interval type and the interval time;

[0017] The task scheduling service interface is called, and the crawling task is generated using the site address and the time expression.

[0018] In one implementation, the storing the target data includes:

[0019] Construct a sensitive word tab, input a target sensitive word in the sensitive word tab, and generate a sensitive word replacement strategy according to the target sensitive word;

[0020] Initial data is obtained, and the target sensitive words contained in the initial data are replaced by the sensitive word replacement strategy to obtain the target data and save the target data.

[0021] In one implementation, generating a target data index includes:

[0022] The interface of the search server Elasticsearch is called, and the target data index is generated using the target data.

[0023] In one implementation, the step of obtaining target data from the target website includes:

[0024] Determine whether the target website is an external network server;

[0025] If it is the external network server, select the corresponding ferrying rule to formulate the ferrying task;

[0026] Calling the ferry task to convert the target data of the external network server into a data compression package;

[0027] The data compression package is transmitted to an intranet server, and the data compression package is restored to obtain the target data.

[0028] In one implementation, after acquiring the target data in the target website, the method includes:

[0029] The target data is used to replace data on the crawled pages of the target website to obtain extracted pages, and an extracted identifier corresponding to the crawled pages is generated.

[0030] In a second aspect, the present application provides a data crawling system, comprising:

[0031] An acquisition module is used to acquire the site information of the target website and configure a corresponding crawler script according to the site information;

[0032] A generation module, used to generate corresponding crawling tasks according to the crawler script;

[0033] A crawling module, used to execute the crawling task and obtain the target data in the target website;

[0034] A saving module, used to save the target data and generate a target data index;

[0035] The export module is used to export the target data according to the target data index using a data export interface.

[0036] In a third aspect, the present application provides an electronic device comprising a memory and at least one processor, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the data crawling method as described in the first aspect.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the data crawling method as described in the first aspect is implemented.

[0038] The embodiments of the present invention have the following beneficial effects:

[0039] The data crawling method provided by the present invention, through the site information set in advance, the user only needs to create a corresponding crawler script according to the site information, and then execute the crawling task through the crawler script to obtain the data the user wants, which greatly reduces the cost of data acquisition.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of a data crawling method flow is shown;

[0043] Figure 2 A schematic diagram of a crawler script configuration method flow is shown;

[0044] Figure 3 A schematic diagram of a target data acquisition method flow chart is shown;

[0045] Figure 4 A schematic diagram of the data crawling system framework is shown.

[0046] Description of main component symbols:

[0047] 400. Data crawling system; 401. Acquisition module; 402. Generation module; 403. Crawling module; 404. Saving module; 405. Exporting module. DETAILED DESCRIPTION

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0049] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of the template are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0051] Example 1

[0052] See also Figure 1 , Figure 1 A data crawling method flow chart provided in this embodiment, which can be used to crawl data from web pages of various websites, includes:

[0053] S101, obtaining the site information of the target website, and configuring a corresponding crawler script according to the site information.

[0054] A web crawler is a program or script that automatically crawls the World Wide Web according to certain rules. Therefore, crawler technology is often used to obtain various data. However, for ordinary users, if they want to use web crawlers skillfully, they need to first connect to the CSS language or XPATH language and formulate selection rules based on regular expressions, which takes a lot of time for users.

[0055] In order to save users' learning time and costs, crawler engineers can write crawler scripts for relevant sites in advance and build them into the user's system. Users only need to configure and call the scripts when using them.

[0056] See also Figure 2 , step S101 includes:

[0057] S1011. Acquire site information of the target website, where the site information includes a site ID, a site name, a site description, and a site address.

[0058] Site information can be stored in a database. Users can save the site information of frequently used target websites into the database and generate a corresponding site list. The site list includes the site name and site identifier for user convenience. When crawling data later, it can be directly obtained from the database. Site information can also be temporarily entered by the user. Site information generally includes site ID, site name, site description, site address, etc.

[0059] S1012. Determine script information of the crawler script according to the site information, where the script information includes crawling start time, interval type, and interval time.

[0060] Script information generally includes crawling start time, interval type, interval time, last crawling time and crawling status, etc. Script information is generally used to control when crawling starts and ends, the time interval between each crawl, etc.

[0061] S1013. Configure the crawler script according to the script information.

[0062] After configuring the crawler script according to the script information, you can then crawl data from the target site.

[0063] S102: Generate a corresponding crawling task according to the crawler script.

[0064] After the crawler script is configured, the corresponding crawling task can be generated, that is, according to the script information such as the crawling start time, the crawler script is controlled to start crawling data. However, the crawling task needs to be performed through the corresponding scheduling task service interface, and the scheduling task service interface only supports cron expressions, that is, time expressions, in terms of time rules. Therefore, the time information in the script information needs to be converted into the corresponding time expression.

[0065] Common time expressions usually consist of six space-separated numbers (time type is yyyy-MM-ddHH:mm:ss, and the setting range is after the current time node), interval time (setting range is number), interval type (setting range is year, month, day, week), so it is necessary to generate a time expression based on the crawling time, interval type, and interval time, and then call the task scheduling service interface, pass in the scheduling interface address and time expression to generate the final crawling task.

[0066] S103: Execute the crawling task to obtain target data from the target website.

[0067] When executing a crawling task to obtain target data, since some of the data may be sensitive words, it is easy to cause this part of the data to be unreadable or to appear garbled. Therefore, in order to ensure that the crawled data can be read and displayed normally, you can first formulate a corresponding sensitive word replacement strategy.

[0068] See also Figure 3 , step S103 includes:

[0069] S1031 . Construct a sensitive word tab, input a target sensitive word in the sensitive word tab, and generate a sensitive word replacement strategy according to the target sensitive word.

[0070] The sensitive word tab is the sensitive word input or collection page. After the user clicks the sensitive word tab to enter the page, he can create a new sensitive word category or select an existing sensitive word category, fill in sensitive words, replacement words and synonyms in the sensitive word category, and form a sensitive word replacement strategy.

[0071] S1032: Acquire initial data, and use the sensitive word replacement strategy to replace the target sensitive word contained in the initial data to obtain the target data and save it.

[0072] The background generates a policy entity based on the sensitive word replacement strategy passed in and saves it to the database. You can then query the database to obtain the set policy.

[0073] After setting up the sensitive word replacement strategy, each time the crawling task crawls a portion of the initial data, the sensitive words in the initial data can be replaced by the sensitive word replacement strategy with corresponding replacement words or synonyms to obtain the target data that can be read and browsed normally.

[0074] In one implementation, for data that has been crawled, in order to avoid repeated crawling, the user can add corresponding identifiers to the crawled web pages.

[0075] Because the HTML content of the source page of the target website is stored during the crawling process, after entering the details page of the target website for the first time, the system will first determine whether the details page has been processed based on the data replacement status recorded in the database. If not, the original HTML will be taken out and the resource files in it will be replaced according to the identifier agreed with the crawler engineer to avoid the inability to access the resources in the original page in the event of a network outage. After completion, the replacement status of the web page is set to replaced and the data is updated in the database.

[0076] When you enter the web page for the second time, you can also judge whether the details page has been processed based on the replacement status, so as to reduce the query time. Finally, the replaced HTML is returned to the front-end browser page to render and display the original web page. Due to the lack of the original page style, there will be slight differences from the original page, but it will not affect the use.

[0077] In one embodiment, the method further comprises:

[0078] Determine whether the target website is an external network server;

[0079] If it is the external network server, select the corresponding ferrying rule to formulate the ferrying task;

[0080] Calling the ferry task to convert the target data of the external network server into a data compression package;

[0081] The data compression package is transmitted to an intranet server, and the data compression package is restored to obtain the target data.

[0082] If the user's data acquisition website is an intranet and the target website is also an intranet, the target data can be acquired using conventional crawling methods. If the target website is an external server, conventional crawling methods may not be able to normally acquire the target data.

[0083] At this time, you can create a scheduled task through the Java program according to the external website and ferry rules selected by the user. When the scheduled task is executed, the background queries the site for the last ferry time recorded in the database (the execution time will be stored in the database each time a ferry task is performed, and the incremental data will be filtered out later by this time), and all SQL data and resources after this time are queried, and compressed into a package through the Java program. Because the compressed package may be very large, it is cut into several small compressed packages, and the external interface of the ferry machine is called for data input.

[0084] The intranet determines the identifier of the compressed package based on the data obtained from the output port and merges it into the original compressed package to obtain the target data.

[0085] S104: Save the target data and generate a target data index.

[0086] After obtaining the compressed package corresponding to the target data, decompress it into the corresponding SQL file and resource file, and store it in the database and server file directory. Finally, call the intranet elasticsearch service interface to create an intranet index, so that users can quickly browse data using the system on the intranet.

[0087] S105. Export the target data using a data export interface according to the target data index.

[0088] You can set up corresponding export functions in the system, such as setting category export buttons, single article export buttons, batch selection buttons, etc. Users can export target data by clicking the corresponding export button.

[0089] For example: users can first manually enter the title, source and other query conditions to filter out article data, then click the button, and then check the articles to be exported to complete the selection of the export method. The background will then filter out the corresponding data based on the data source selected by the user. The system will query the specific data content to be exported based on the user's selected filtering conditions or checked articles, and generate word or excel format documents according to the selected export type through the word or excel operation interface provided in the third-party jar package poi package to meet the format requirements of users' daily office work.

[0090] This embodiment has built-in site rules in advance. Users only need to select the built-in site they want to crawl and set the corresponding crawling time, generate the corresponding crawling task to complete the creation of the crawler, and then use the crawler to crawl data, thereby solving the problem of high cost for users to obtain data.

[0091] Example 2

[0092] See also Figure 4The present application also provides a data crawling system 400, including:

[0093] The acquisition module 401 is used to acquire the site information of the target website and configure the corresponding crawler script according to the site information;

[0094] A generation module 402, used to generate a corresponding crawling task according to the crawler script;

[0095] A crawling module 403 is used to execute the crawling task and obtain the target data in the target website;

[0096] A saving module 404 is used to save the target data and generate a target data index;

[0097] The export module 405 is used to export the target data according to the target data index using a data export interface.

[0098] It can be understood that the implementation method of the data crawling method described in the above embodiment 1 is also applicable to this embodiment, so it will not be repeated here.

[0099] Example 3

[0100] The embodiment of the present application also provides a computer device, for example, the computer device can be, but is not limited to, a desktop computer, a notebook, etc., and its existence form is not limited, mainly depending on whether it needs to support the interface display function of the browser web page, etc. Exemplarily, the computer device includes a memory and at least one processor, the memory stores a computer program, and the processor is used to execute the computer program to implement the data crawling method described in the above embodiment 1.

[0101] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU) and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. At least one of the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0102] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.

[0103] Furthermore, the memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data (such as iteration data, version data, etc.) created according to the use of the computer device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0104] Example 4

[0105] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the data crawling method described in the above embodiment 1.

[0106] It can be understood that the implementation method of the data crawling method described in the above embodiment 1 is also applicable to this embodiment, so it will not be repeated here.

[0107] The computer-readable storage medium may be either a non-volatile storage medium or a volatile storage medium. For example, the computer-readable storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0108] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0110] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0111] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

[0112] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0113] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0114] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A data crawling method, characterized in that: include: Obtain the site information of the target website and configure the corresponding crawler script according to the site information; Generate a corresponding crawling task according to the crawler script; Execute the crawling task to obtain target data from the target website; The target data is saved and a target data index is generated; According to the target data index, the target data is exported using a data export interface.

2. The data crawling method according to claim 1, characterized in that: The step of obtaining the site information of the target website and configuring a corresponding crawler script according to the site information includes: Obtaining site information of the target website, wherein the site information includes a site ID, a site name, a site description, and a site address; Determine script information of the crawler script according to the site information, the script information including crawling start time, interval type and interval time; The crawler script is configured according to the script information.

3. The data crawling method according to claim 2, characterized in that: Generating a corresponding crawling task according to the crawler script includes: Generate a corresponding time expression according to the crawling start time, the interval type and the interval time; The task scheduling service interface is called, and the crawling task is generated using the site address and the time expression.

4. The data crawling method according to claim 1, characterized in that: The step of saving the target data comprises: Construct a sensitive word tab, input a target sensitive word in the sensitive word tab, and generate a sensitive word replacement strategy according to the target sensitive word; Initial data is obtained, and the target sensitive words contained in the initial data are replaced by the sensitive word replacement strategy to obtain the target data and save the target data.

5. The data crawling method according to claim 4, characterized in that: The generating of the target data index comprises: The interface of the search server Elasticsearch is called, and the target data index is generated using the target data.

6. The data crawling method according to claim 1, characterized in that: The step of acquiring target data from the target website includes: Determine whether the target website is an external network server; If it is the external network server, select the corresponding ferrying rule to formulate the ferrying task; Calling the ferry task to convert the target data of the external network server into a data compression package; The data compression package is transmitted to an intranet server, and the data compression package is restored to obtain the target data.

7. The data crawling method according to claim 1, characterized in that: After the target data in the target website is obtained, the method includes: The target data is used to replace data on the crawled pages of the target website to obtain extracted pages, and an extracted identifier corresponding to the crawled pages is generated.

8. A data crawling system, characterized in that: include: An acquisition module is used to acquire the site information of the target website and configure a corresponding crawler script according to the site information; A generation module, used to generate corresponding crawling tasks according to the crawler script; A crawling module, used to execute the crawling task and obtain the target data in the target website; A saving module, used to save the target data and generate a target data index; The export module is used to export the target data according to the target data index using a data export interface.

9. An electronic device, characterized in that: The method comprises a memory and at least one processor, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the data crawling method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the data crawling method according to any one of claims 1 to 7 is implemented.

Images