Method and device for grabbing external web pages by proxy for users in cloud environment
An external web page and cloud environment technology, applied in the Internet field, can solve problems such as many restrictions, not supporting user-defined requirements, and inability to use resource auditing by users, so as to improve the efficiency of use and facilitate management.
Active Publication Date: 2017-12-22
BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0002] At present, it is very common to access an external web page or submit a request to an external address in the proxy cloud environment. Users can use the sdk (Software Development Kit, software development kit) provided by the FetchURL service (web crawling service provided by the cloud environment) Access to external web pages, however, this access method has the following problems: (1) SDK has many restrictions and does not support user's custom requirements; (2) cannot audit the user's resources; (3) does not support https access
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0022] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention. On the contrary, the embodiments of the present invention include all changes, modifications and equivalents coming within the spirit and scope of the appended claims.
[0023] In the description of the present invention, it should be understood that the terms "first", "second" and so on are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance. In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "connected" and "connect...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present invention proposes a method and device for grabbing external webpages on behalf of users in a cloud environment, wherein the method includes the following steps: receiving a request from a user to grab a webpage; verifying the user's identity; counting the user's quota to determine whether the user is in In the blacklist; parse the request to determine whether to use the http or https protocol to crawl the webpage; and if the user is authenticated and not in the blacklist, proxy the user to crawl the webpage using the http or https protocol as required. According to the method of the embodiment of the present invention, through the user identity verification and quota blacklist verification in the proxy cloud environment, users who meet the identity verification and quota blacklist verification use http or https protocol to grab web pages, which not only meets the requirements of users to obtain external resources At the same time, statistics on the resources used by users can be made to facilitate management, and users with large usage can be restricted to improve usage efficiency.
Description
technical field [0001] The invention relates to the technical field of the Internet, in particular to a method and a device for grabbing external webpages on behalf of users in a cloud environment. Background technique [0002] At present, it is very common to access an external web page or submit a request to an external address in the proxy cloud environment. Users can use the sdk (Software Development Kit, software development kit) provided by the FetchURL service (web crawling service provided by the cloud environment) Access to external web pages, however, this access method has the following problems: (1) SDK has many restrictions and does not support user's custom requirements; (2) cannot audit the user's resources; (3) does not support https access. Contents of the invention [0003] The present invention aims to solve at least one of the above-mentioned technical problems. [0004] For this reason, an object of the present invention is to propose a method of pro...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08H04L29/06G06F17/30
Inventor 高巍何磊李彬孟
Owner BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD