Website resource crawling method and device, computer equipment and storage medium
A website and resource technology, applied in the direction of network data indexing, network data retrieval, special data processing applications, etc., can solve the problems of reducing the development efficiency of R&D personnel, reducing the enthusiasm of R&D personnel, time-consuming and labor-intensive, etc., to save labor costs and time costs , Improve flexibility and effectiveness, and ensure accuracy
Active Publication Date: 2019-09-27
KINGSOFT
View PDF5 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
However, the problem is that R&D personnel can still configure a small number of data sources, and for large data needs, mechanical repetitive activities will also reduce the enthusiasm of R&D personnel, and this development method is time-consuming and laborious, which seriously reduces Improve the development efficiency of R&D personnel
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.
[0044] The crawling method, device, computer equipment and computer-readable storage medium of the embodiments of the present invention will be described below with reference to the accompanying drawings.
[0045] figure 1 It is a flowchart of a method for crawling website resources according to an embodiment of the present invention. It should be noted that the website resource crawling method in the embodiment of the present invention can be applied to the website resource crawling device in the embodiment of the present inven...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a website resource crawling method and device, computer equipment and a storage medium. The method comprises the following steps: determining a flow chart designed by a user, wherein the flow chart comprises a plurality of nodes and connection relationships among the nodes, and each node corresponds to one control; generating a crawling configuration rule for the target website based on a connection relationship between the control corresponding to the node in the flow chart and the node in the flow chart; and crawling corresponding resources in the target website according to the crawling configuration rule to obtain corresponding crawling result information. According to the method, a user can design the corresponding flow chart according to the requirements of the user, the crawler rule configuration process is proceeded based on the flow chart, the configuration flexibility, effectiveness and crawling accuracy are improved, and the labor cost and the time cost can be effectively saved.
Description
technical field [0001] The invention relates to the field of computer applications, in particular to a method, device, computer equipment and storage medium for crawling website resources. Background technique [0002] With the rapid development of Internet technology, there are massive amounts of data on the Internet. In order to conveniently provide users with search functions, search engines often need to search and analyze massive amounts of data on the Internet. The emergence of crawler technology has effectively improved search efficiency. Crawler technology mainly extracts effective information by identifying, crawling, and cleaning specific resources. With the development of the times, crawler technology will also develop rapidly and be applied to more application fields to improve the utilization rate of data and promote the development of society. [0003] In related technologies, resources on the crawled webpage are identified and crawled mainly by manually view...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/951
CPCG06F16/951
Inventor 孙加亮
Owner KINGSOFT