Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Distributed crawler method based on Kubentes cluster and storage device

A storage device and distributed technology, applied in the field of reptiles, can solve problems such as complex and cumbersome operations, and achieve the effect of improving resource utilization efficiency and making crawler deployment simple, convenient and easy to operate

Pending Publication Date: 2022-03-01
福建博思数字科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0012] To this end, it is necessary to provide a distributed crawler method based on the Kubenetes cluster to solve the technical problem that the crawler code needs to be uploaded to each remote machine one by one when there is a code update for the existing crawler, and the operation is complicated and cumbersome. The technical solution is as follows:

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed crawler method based on Kubentes cluster and storage device
  • Distributed crawler method based on Kubentes cluster and storage device
  • Distributed crawler method based on Kubentes cluster and storage device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0063]In order to describe in detail the possible application scenarios, technical principles, specific solutions that can be implemented, goals and effects that can be achieved, etc., the following will be described in detail in conjunction with the listed specific embodiments and accompanying drawings. The embodiments described herein are only used to illustrate the technical solutions of the present application more clearly, so they are only examples, and cannot be used to limit the protection scope of the present application.

[0064] Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The word "embodiment" appearing in various positions in the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or relationship with other embodiments. In principle, in thi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of crawlers, in particular to a distributed crawler method based on a Kubentes cluster and storage equipment. The distributed crawler method based on the Kubentes cluster comprises the following steps: constructing a code warehouse; constructing a product library; constructing an automatic deployment process; automatically deploying a mirror image file of a product library to each node of Kubentes according to the deployment process; and starting a crawler task for task collection. By means of the method, complex environment configuration needed by the crawler does not need to be configured one by one, the environment is packaged in a mirror image package, crawler programs do not need to be upgraded one by one in deployment upgrading, and the crawler programs are deployed to all crawler machines through a product library according to an automatic deployment process. And when updating exists, crawler deployment is simple, convenient and easy to operate.

Description

technical field [0001] The invention relates to the technical field of reptiles, in particular to a Kubenetes cluster-based distributed crawler method and storage device. Background technique [0002] A web crawler (also known as a web spider, a web robot, and more often referred to as a web chaser in the FOAF community) is a program or script that automatically grabs information on the World Wide Web according to certain rules. [0003] With the rapid development of the network, the World Wide Web has become the carrier of a large amount of information, how to effectively extract and use this information has become a huge challenge. Search Engine (Search Engine), such as the traditional general search engine AltaVista, Yahoo! And Google, etc., as a tool to assist people in retrieving information, become the entrance and guide for users to access the World Wide Web. However, these universal search engines also have certain limitations, such as: [0004] (1) Users in diffe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/951G06F16/27
CPCG06F16/951G06F16/27
Inventor 张汀勇叶章明侯祥钦陈天宇郑为洛林杰王敏力卞江新
Owner 福建博思数字科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com