A data crawling method, system, device, and storage medium

Abstract

This invention provides a data crawling method, system, device, and storage medium. The method includes: obtaining site information of a target website; configuring a corresponding crawler script based on the site information; generating a corresponding crawling task based on the crawler script; executing the crawling task to obtain target data from the target website; saving the target data and generating a target data index; and exporting the target data using a data export interface based on the target data index. This invention allows users to create a corresponding crawler script simply by using the site information, and then execute the crawling task using the crawler script to obtain the data they want, greatly reducing the cost of data acquisition.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a data crawling method, system, device and storage medium. Background Technology

[0002] A knowledge graph, also known as a knowledge domain visualization or knowledge domain mapping map, is a series of various graphs that display the development process and structural relationships of knowledge. It uses visualization techniques to describe knowledge resources and their carriers, and to mine, analyze, construct, draw, and display knowledge and the interrelationships between them. With the development of network technology, more and more enterprises are keen to build knowledge graphs for intelligent search, text analysis, and machine reading. However, the data provided by enterprises themselves often cannot meet the requirements for building knowledge graphs, and existing data crawling methods are often time- and economically costly, making them unsuitable for the development needs of enterprises. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a data crawling method, system, device and storage medium.

[0004] This invention provides the following technical solution:

[0005] Firstly, this application provides a data crawling method, including:

[0006] Obtain the site information of the target website, and configure the corresponding crawler script based on the site information;

[0007] Generate corresponding crawling tasks based on the crawler script;

[0008] Execute the crawling task to obtain target data from the target website;

[0009] The target data is saved, and a target data index is generated;

[0010] Based on the target data index, the target data is exported using the data export interface.

[0011] In one embodiment, obtaining the site information of the target website and configuring the corresponding crawler script based on the site information includes:

[0012] Obtain the site information of the target website, including site ID, site name, site description, and site address;

[0013] The script information of the crawler script is determined based on the site information, and the script information includes crawl start time, interval type and interval time;

[0014] Configure the crawler script according to the script information.

[0015] In one embodiment, generating the corresponding crawling task based on the crawler script includes:

[0016] Generate a corresponding time expression based on the crawl start time, the interval type, and the interval time;

[0017] The task scheduling service interface is invoked, and the crawling task is generated using the site address and the time expression.

[0018] In one embodiment, saving the target data includes:

[0019] Construct a sensitive word tab, input target sensitive words into the sensitive word tab, and generate a sensitive word replacement strategy based on the target sensitive words;

[0020] Obtain initial data, and use the sensitive word replacement strategy to replace the target sensitive words contained in the initial data to obtain the target data and save it.

[0021] In one implementation, generating the target data index includes:

[0022] Call the Elasticsearch search server's API to generate the target data index using the target data.

[0023] In one embodiment, obtaining the target data from the target website includes:

[0024] Determine whether the target website is an external server;

[0025] If it is an external network server, then select the corresponding ferry rule to formulate the ferry task;

[0026] The ferry task is invoked to convert the target data on the external network server into a data compressed package;

[0027] The compressed data package is transmitted to an intranet server, and the compressed data package is restored to obtain the target data.

[0028] In one implementation, after obtaining the target data from the target website, the process includes:

[0029] The target data is used to replace the data of the crawled pages of the target website to obtain the extracted pages, and an extracted identifier corresponding to the crawled pages is generated.

[0030] Secondly, this application provides a data crawling system, including:

[0031] The acquisition module is used to acquire site information of the target website and configure the corresponding crawler script based on the site information.

[0032] The generation module is used to generate corresponding crawling tasks based on the crawler script;

[0033] The crawling module is used to execute the crawling task and obtain target data from the target website;

[0034] A storage module is used to store the target data and generate a target data index;

[0035] The export module is used to export the target data based on the target data index using a data export interface.

[0036] Thirdly, this application provides an electronic device including a memory and at least one processor, the memory storing a computer program, and the processor executing the computer program to implement the data crawling method as described in the first aspect.

[0037] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed, implements the data crawling method as described in the first aspect.

[0038] The embodiments of the present invention have the following beneficial effects:

[0039] The data crawling method provided by this invention allows users to create corresponding crawler scripts simply by using pre-set site information. Then, by executing the crawler scripts, users can obtain the data they want, greatly reducing the cost of data acquisition.

[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A schematic diagram of a data crawling method is shown;

[0043] Figure 2 This diagram illustrates a method for configuring a web crawler script.

[0044] Figure 3 A schematic diagram of the target data acquisition method is shown;

[0045] Figure 4 A schematic diagram of the framework structure of a data crawling system is shown.

[0046] Explanation of key component symbols:

[0047] 400. Data crawling system; 401. Acquisition module; 402. Generation module; 403. Crawling module; 404. Saving module; 405. Export module. Detailed Implementation

[0048] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0049] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the template description is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0051] Example 1

[0052] See Figure 1 , Figure 1 This is a schematic diagram of a data crawling method provided in this embodiment. This method can be used to crawl data from web pages of various websites. The method includes:

[0053] S101. Obtain the site information of the target website and configure the corresponding crawler script according to the site information.

[0054] A web crawler is a program or script that automatically retrieves information from the World Wide Web according to certain rules. Therefore, web crawling technology is often used to obtain various types of data. However, for ordinary users, to use web crawlers proficiently, they need to first connect to CSS or XPath languages ​​and formulate selection rules based on regular expressions, which requires a significant investment of time.

[0055] To save users' learning time and costs, web crawling engineers can pre-write the relevant website crawling scripts and embed them in the user's system. Users only need to configure and call the scripts when needed.

[0056] See Figure 2 Step S101 includes:

[0057] S1011. Obtain the site information of the target website, including site ID, site name, site description and site address.

[0058] Site information can be stored in a database. Users can save information about frequently used target websites to the database, generating a corresponding site list. This list includes site names and site identifiers for user convenience. When data needs to be crawled later, it can be retrieved directly from the database. Alternatively, site information can be entered temporarily by the user. Site information typically includes site ID, site name, site description, and site address, etc.

[0059] S1012. Determine the script information of the crawler script based on the site information. The script information includes the crawl start time, interval type, and interval time.

[0060] Script information typically includes crawl start time, interval type, interval time, last crawl time, crawl status, etc. Script information is generally used to control when crawling starts and ends, the time interval between two crawls, etc.

[0061] S1013. Configure the crawler script according to the script information.

[0062] After configuring the crawler script according to the script information, you can then crawl data from the target site.

[0063] S102. Generate the corresponding crawling task according to the crawler script.

[0064] Once the crawler script is configured, the corresponding crawling task can be generated. This means that the crawler script will start crawling data based on the set start time and other script information. However, the crawling task needs to be processed through a corresponding scheduling service interface. This interface only supports cron expressions (time expressions) for timing rules. Therefore, the time information in the script needs to be converted into the corresponding time expression.

[0065] Common time expressions typically consist of six space-separated numbers (time type yyyy-MM-ddHH:mm:ss, setting range after the current time node), interval time (setting range is numbers), and interval type (setting range is year, month, day, week). Therefore, it is necessary to generate a time expression based on the crawling time, interval type, and interval time, and then call the task scheduling service interface, passing in the scheduling interface address and the time expression to generate the final crawling task.

[0066] S103. Execute the crawling task to obtain target data from the target website.

[0067] When performing a web scraping task to obtain target data, some data may contain sensitive words, which may cause the data to be unreadable or result in garbled characters. Therefore, in order to ensure that the scraped data can be read and displayed normally, a corresponding sensitive word replacement strategy can be formulated first.

[0068] See Figure 3 Step S103 includes:

[0069] S1031. Construct a sensitive word tab, input the target sensitive word into the sensitive word tab, and generate a sensitive word replacement strategy based on the target sensitive word.

[0070] The Sensitive Words tab is the page for inputting or collecting sensitive words. After clicking on the Sensitive Words tab, users can enter the page and create a new sensitive word category or select an existing sensitive word category. In the sensitive word category, they can fill in sensitive words, replacement words, and synonyms to form a sensitive word replacement strategy.

[0071] S1032. Obtain initial data, and use the sensitive word replacement strategy to replace the target sensitive words contained in the initial data to obtain the target data and save it.

[0072] The backend generates a policy entity based on the input sensitive word replacement strategy and saves it to the database. The configured strategy can be retrieved later by querying the database.

[0073] After setting up the sensitive word replacement strategy, each time the crawling task crawls a portion of the initial data, the sensitive words in the initial data can be replaced with corresponding replacement words or synonyms to obtain target data that can be read and browsed normally.

[0074] In one implementation, to avoid repeated crawling of data that has already been crawled, users can add corresponding identifiers to the crawled web pages.

[0075] Because the HTML content of the target website's source page is stored during the crawling process, when the system first enters the target website's details page, it will first determine whether the details page has been processed based on the data replacement status recorded in the database. If not, it will retrieve the original HTML and replace the resource files in it according to the identifiers agreed upon with the crawler engineers to avoid being unable to access the resources in the original page when the network is disconnected. After completion, the replacement status of the webpage will be set to replaced and the data will be updated in the database.

[0076] When the webpage is accessed again, the replacement status can be used to determine whether the details page has been processed. This allows for a reduction in search time by determining whether the page has been replaced. Finally, the replaced HTML is returned to the front-end browser for rendering and displaying the original webpage. Due to the lack of the original page style, there will be slight differences from the original page, but this will not affect usability.

[0077] In one embodiment, the method further includes:

[0078] Determine whether the target website is an external server;

[0079] If it is an external network server, then select the corresponding ferry rule to formulate the ferry task;

[0080] The ferry task is invoked to convert the target data on the external network server into a data compressed package;

[0081] The compressed data package is transmitted to an intranet server, and the compressed data package is restored to obtain the target data.

[0082] If both the user's data acquisition website and the target website are on an intranet, then conventional crawling methods can be used to acquire the target data. However, if the target website is an external server, conventional crawling methods may not be able to acquire the target data.

[0083] At this point, a scheduled task can be created using a Java program based on the user-selected external website and the ferry rules. When the scheduled task's execution time arrives, the backend queries the database for the last recorded ferry time (each ferry task's execution time is stored in the database, and incremental data is subsequently filtered using this time). It then retrieves all SQL data and resources after that time, compresses them into a single compressed file using a Java program (which, due to the potentially large size of the compressed file, is split into several smaller compressed files), and calls the ferry machine's external interface for data input.

[0084] The internal network uses the data obtained from the output port to determine the identifier of the compressed package and merge it into the original compressed package to obtain the target data.

[0085] S104. Save the target data and generate a target data index.

[0086] After obtaining the compressed package corresponding to the target data, it is decompressed into the corresponding SQL file and resource file, and stored in the database and server file directory. Finally, the internal Elasticsearch service interface is called to create an internal index, making it convenient for users to quickly browse data on the internal network.

[0087] S105. Based on the target data index, export the target data using the data export interface.

[0088] The system allows users to set corresponding export functions, such as export buttons for categories, single articles, and batch selection. Users can export the target data by clicking the corresponding export button.

[0089] For example, users can first filter article data by manually entering search criteria such as title and source, then click the button, and then select the articles to be exported to complete the export method selection. The background then filters the corresponding data according to the data source selected by the user. The system will query the specific data content to be exported based on the filter criteria or the selected articles selected by the user, and generate Word or Excel format documents according to the selected export type through the Word or Excel operation interface provided in the third-party JAR package POI program package, to meet the format needs of users' daily office work.

[0090] This embodiment uses pre-built site rules, so users only need to select the built-in sites they want to crawl and set the corresponding crawling time to generate the corresponding crawling task to complete the creation of the crawler. Then, the crawler can be used to crawl data, thus solving the problem of high data acquisition costs for users.

[0091] Example 2

[0092] See Figure 4This application also provides a data crawling system 400, comprising:

[0093] The acquisition module 401 is used to acquire site information of the target website and configure the corresponding crawler script according to the site information;

[0094] The generation module 402 is used to generate corresponding crawling tasks based on the crawler script;

[0095] The crawling module 403 is used to execute the crawling task and obtain target data from the target website;

[0096] The storage module 404 is used to store the target data and generate a target data index;

[0097] The export module 405 is used to export the target data according to the target data index using the data export interface.

[0098] It is understood that the implementation method of the data crawling method described in Embodiment 1 above is also applicable to this embodiment, so it will not be described again here.

[0099] Example 3

[0100] This application also provides a computer device, which may be, but is not limited to, a desktop computer, a laptop, etc. Its form is not limited, mainly depending on whether it needs to support the interface display function of a web browser. Exemplarily, the computer device includes a memory and at least one processor. The memory stores a computer program, and the processor executes the computer program to implement the data crawling method described in Embodiment 1 above.

[0101] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0102] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). The memory stores computer programs, and the processor, upon receiving execution instructions, can execute the computer programs accordingly.

[0103] Furthermore, the memory may include a stored program area and a stored data area, wherein the stored program area may store the operating system and application programs required for at least one function; the stored data area may store data created based on the use of the computer device (such as iterative data, version data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0104] Example 4

[0105] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to execute the data crawling method described in Embodiment 1 above.

[0106] It is understood that the implementation method of the data crawling method described in Embodiment 1 above is also applicable to this embodiment, so it will not be described again here.

[0107] The computer-readable storage medium can be either a non-volatile storage medium or a volatile storage medium. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0109] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0110] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0112] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0113] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0114] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A data crawling method, characterized in that, The method comprises the following steps: acquiring site information of a target website, and configuring a corresponding crawler script according to the site information; generating a corresponding crawling task according to the crawler script; executing the crawling task to acquire target data in the target website; saving the target data and generating a target data index; exporting the target data by using a data export interface according to the target data index; the acquiring of the target data in the target website comprises the following steps: judging whether the target website is an external network server; if the target website is the external network server, formulating a ferry task according to a corresponding ferry rule according to an external website site; when reaching an execution time point of the ferry task, querying a last ferry time according to an external website site, screening out incremental data according to the last ferry time, calling the ferry task, and converting the incremental data of the external network server into a data compression package; transmitting the incremental data compression package to an internal network server, restoring the data compression package, and obtaining the target data; after the acquiring of the target data in the target website, the method comprises the following steps: replacing data in a crawled page of the target website by using the target data, obtaining an extracted page, and generating an extracted identifier corresponding to the crawled page; after entering a detail page of the target website for the first time, judging whether the detail page has been processed according to a data replacement state recorded in a database, if not, taking out original html, and replacing a resource file according to an identifier agreed with a crawler engineer; judging whether the detail page has been processed according to a replacement state of the resource file when entering the detail page for the second time.

2. The data crawling method of claim 1, wherein, The acquiring of the site information of the target website and the configuring of the corresponding crawler script according to the site information comprise the following steps: acquiring site information of the target website, wherein the site information comprises a site ID, a site name, a site description and a site address; determining script information of the crawler script according to the site information, wherein the script information comprises a crawling start time, an interval type and an interval time; configuring the crawler script according to the script information.

3. The data crawling method of claim 2, wherein, The generating of the corresponding crawling task according to the crawler script comprises the following steps: generating a corresponding time expression according to the crawling start time, the interval type and the interval time; calling a task scheduling service interface, and generating the crawling task by using the site address and the time expression.

4. The data crawling method of claim 1, wherein, The saving of the target data comprises the following steps: constructing a sensitive word tab, inputting a target sensitive word in the sensitive word tab, and generating a sensitive word replacement strategy according to the target sensitive word; acquiring initial data, replacing the target sensitive word contained in the initial data by using the sensitive word replacement strategy to obtain the target data, and saving the target data.

5. The data crawling method of claim 4, wherein, The generating of the target data index comprises the following steps: calling an interface of a search server Elasticsearch, and generating the target data index by using the target data.

6. A data crawling system, comprising: The method comprises the following steps: an acquiring module, configured to acquire site information of a target website, and configure a corresponding crawler script according to the site information; a generating module, configured to generate a corresponding crawling task according to the crawler script; The crawling module is configured to execute the crawling task and obtain target data in the target website; The saving module is configured to save the target data and generate a target data index; The exporting module is configured to export the target data by using a data exporting interface according to the target data index; The obtaining of the target data in the target website comprises: determining whether the target website is an external network server; if the target website is the external network server, formulating a ferry task according to a corresponding ferry rule selected according to an external website; when reaching an execution time point of the ferry task, querying a last ferry time according to an external website, screening out incremental data according to the last ferry time, calling the ferry task, and converting the incremental data of the external network server into a data compression package; transmitting the incremental data compression package to an internal network server, restoring the data compression package, and obtaining the target data; after the obtaining of the target data in the target website, comprising: replacing data of a crawled page of the target website by using the target data, obtaining an extracted page, and generating an extracted identifier corresponding to the crawled page; after entering a detail page of the target website for the first time, determining whether the detail page has been processed according to a data replacement state recorded in a database, if not, obtaining an original html, and replacing a resource file according to an identifier agreed with a crawler engineer; when entering the detail page for the second time, determining whether the detail page has been processed according to a replacement state of the resource file.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed to implement the data crawling method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed to implement the data crawling method according to any one of claims 1 to 5.

Images