Method and system of web page link library updating
A web link and update method technology, applied in the Internet field, can solve the problems of long update time and low update efficiency of web link library, and achieve the effect of improving update efficiency
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0027] Please refer to the attached figure 1 , which discloses a flowchart of the first method for updating a web page link library according to an embodiment of the present invention. In this embodiment, each link in the web page link library is sorted according to a corresponding order of crawling.
[0028] The web page link library in the prior art includes fixed-length files and variable-length files, each link is stored in the variable-length file, and the grabbing status, position and length of each link in the variable-length file are stored in the fixed-length file. in the file. The variable-length file stores variable-length information such as links. The fixed-length file is composed of class or structure objects. For example, if the structure defined for the fixed-length file is ClinkData, then the fixed-length file is composed of one ClinkData object followed by one ClinkData object. If some link information needs to be added, a parameter can be directly added to...
Embodiment 2
[0051] see figure 2 , which is a flow chart of the second method for updating a webpage link library disclosed in an embodiment of the present invention, each link in the webpage link library in this embodiment is sorted according to the corresponding crawling order, and the method may include:
[0052] Step S201: Obtain the link to be updated including the initial link and the new link;
[0053] Step S202: Mapping the initial link of the webpage in the webpage link library and the initial crawling state into the memory;
[0054] In fact, the fixed-length file in the web page link library is mapped into the memory, and after being mapped into the memory, the content of the file in the memory is the same as that of the fixed-length file. The purpose of doing this is that only when the fixed-length file is mapped into the memory, can each link object in the fixed-length file be operated like an array, and when the initial link in the fixed-length file is updated, there will be...
Embodiment 3
[0081] see image 3 , is a structural schematic diagram of the first web page link library update system disclosed in the embodiment of the present invention, each link in the web page link library in the system is sorted according to the order of capture, and the system may include: an acquisition module 301, a judgment Module 302, the first update module 303 and the second update module 304, wherein:
[0082] The obtaining module 301 is configured to obtain a link to be updated including an initial link and a new link;
[0083] The judging module 302 is configured to judge whether the link to be updated belongs to the webpage link library;
[0084] Specifically, if the crawling order variable type is a static variable, then after a link object is generated, the value of the grabbing order variable of the object is zero; if the links in the webpage link library are arranged in a positive order If the variable value of the crawl order of the smallest link is 1, then the judg...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More