Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

An information processing method and server

An information processing method and server technology, applied in the direction of digital transmission system, data exchange network, electrical components, etc., can solve the problems of fast system capture rate, no solution, and inability to automatically adjust

Active Publication Date: 2020-09-11
TENCENT TECH (BEIJING) CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] 1. The crawling rate and frequency of webpage data are pre-configured in the script program, and cannot be automatically adjusted according to the number of tasks during the data capture process; 2. The setting of the crawling rate is based on the configured project information , different projects may configure the same domain name, which may cause the crawling rate of the system under the same domain name to be too fast, resulting in the proxy network interconnection protocol (IP) being blocked; however, for the above problems, there is no effective solution

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • An information processing method and server
  • An information processing method and server
  • An information processing method and server

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0089] An embodiment of the present invention provides an information processing method. figure 1 It is a schematic diagram of the composition and structure of the server in Embodiment 1 of the present invention; figure 1 As shown, the server includes: project starter 101, token pool 102, task pool 103, rate controller 104, token generator 105, scheduling queue 106, scheduler 107, processing queue 108 and processor 109; ,

[0090] The project launcher 101 is used to add task information to be fetched into the task pool 103; the task information includes: address information, task identification and initial capture frequency of the task; extracting the task information Domain name information, generate a token according to the domain name information, and add the token to the token pool 102; the token includes: domain name information, initial crawl rate, last token generation time, domain name IP and proxy IP;

[0091] The task pool 103 is used to store task information;

...

Embodiment 2

[0126] The embodiment of the invention also provides a server. Figure 4 It is a schematic diagram of the third composition structure of the server in the embodiment of the present invention; Figure 4 As shown, the server includes: a configuration unit 110, an item pool 111, an item initiator 101, a token pool 102, a task pool 103, a rate controller 104, a frequency controller 112, a token generator 105, a scheduling queue 106, Scheduler 107, processing queue 108 and processor 109; Wherein,

[0127] The configuration unit 110 is configured to configure the captured items, generate corresponding item information based on the configured captured items, and send the item information to the item pool 111;

[0128] The project pool 111 is used to store project information; the project information includes project scripts and project status; the project status includes installed status and uninstalled status;

[0129]The project launcher 101 is configured to scan project informat...

Embodiment 3

[0153] The embodiment of the invention also provides a server. Figure 5 It is a schematic diagram of the fourth composition structure of the server in the embodiment of the present invention; Figure 5As shown, the server includes: a configuration unit 110, an item pool 111, an item initiator 101, a token pool 102, a task pool 103, a rate controller 104, a frequency controller 112, a token generator 105, a scheduling queue 106, Scheduler 107, processing queue 108, processor 109 and domain name resolver 113; Wherein,

[0154] The configuration unit 110 is configured to configure the captured items, generate corresponding item information based on the configured captured items, and send the item information to the item pool 111;

[0155] The project pool 111 is used to store project information; the project information includes project scripts and project status; the project status includes installed status and uninstalled status;

[0156] The project launcher 101 is configur...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the present invention discloses an information processing method and server, including: adding the task information to be captured into the task pool; extracting the domain name information of the task information, generating a token according to the domain name information, and Add the token to the token pool; determine the capture rate corresponding to the domain name information according to the amount of task information that can be scheduled under the domain name information in the token pool; scan the token pool, according to the The capture rate determines the number of tokens corresponding to the domain name information, and when the number of tokens meets the preset conditions, sends the corresponding number of tokens to the scheduling queue; obtains tokens from the scheduling queue, according to the The domain name information corresponding to the token selects the first task information that meets the second preset condition in the task pool, and adds the first task information to the processing queue; extracts the first task information from the processing queue Task information, executing corresponding data capture according to the first task information.

Description

technical field [0001] The present invention relates to information processing technology, in particular to an information processing method and server. Background technique [0002] With the rapid development of Internet technology, web pages have become the carrier of massive information. One of the methods currently used to extract information from webpages is a web crawler, which specifically extracts specified webpage content through a script program configured for webpage data extraction. [0003] In the process of implementing the technical solutions of the embodiments of the present application, the inventors of the present application at least found the following technical problems in the related art: [0004] 1. The crawling rate and frequency of webpage data are pre-configured in the script program, and cannot be automatically adjusted according to the number of tasks during the data capture process; 2. The setting of the crawling rate is based on the configured ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/12H04L12/873H04L47/52
CPCH04L47/527H04L61/5061H04L61/4511
Inventor 黄文飞吴一飞施驭李振洋
Owner TENCENT TECH (BEIJING) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com