Balanced scheduling system and balanced scheduling method based on site quota
A scheduling system and scheduling method technology, applied in the Internet field, can solve problems such as occupying processing capacity and other websites cannot process in time, and achieve the effect of taking into account both efficiency and timeliness
Inactive Publication Date: 2013-01-09
BEIJING QIHOO TECH CO LTD +1
View PDF4 Cites 4 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
If the number of servers is insufficient, the pages of some websites with a large amount of data will take up most of the processing power, causing other websites to be unable to process them in time
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0044] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0045] The flow of the balanced scheduling method based on site quota in this embodiment is as follows: figure 1 shown, including:
[0046] Step S110, obtaining the scheduling tasks in the domain name queue of the site. Scheduling tasks can be obtained from the domain name queue of the whole network search system.
[0047] In step S120, the server designated for the scheduling task downloads a corresponding number of pages from t...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a balanced scheduling system based on the site quota and relates to the technical field of the internet. The system comprises a scheduling task acquisition module used for acquiring scheduling tasks in a domain name queue of a site, a scheduling module used for downloading pages with corresponding number from a server appointed by the scheduling tasks according to the number of preconfigured once schedulable pages. The scheduling task acquisition module is used for acquiring the scheduling tasks in the domain name queue of the site according to the preset task priority. A balanced scheduling method based on the site quota is further disclosed. By means of the balanced scheduling system and the balanced scheduling method based on the site quota, all sites can acquire certain downloading chance under any condition. Simultaneously, different quota limits can be set according to actual condition to give consideration to both efficiency and timeliness and meet requirements of different search products. The system and the method also lay the foundation on unified processing of whole network search and vertical search.
Description
technical field [0001] The invention relates to the technical field of the Internet, in particular to a system and method for balanced scheduling based on site quotas. Background technique [0002] For search engines, crawling pages from the Internet is the first step in their process. However, the number of pages accumulated on the Internet is huge, and the number of newly updated and newly generated pages every day is also huge. How to obtain these pages in a timely manner is the primary problem faced by search engines. In order to crawl these massive pages in a timely manner, reasonable and effective scheduling must be carried out. Therefore, which scheduling algorithm to use is very important. [0003] Currently, when the webpage search schedules webpage crawling, newly discovered pages are queued in order. For the history page, the rescheduling frequency is determined according to the update frequency of the history page. [0004] In web search, download delays are...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): H04L12/865H04L29/12H04L47/6275
Inventor 卢宏林
Owner BEIJING QIHOO TECH CO LTD