Unlock instant, AI-driven research and patent intelligence for your innovation.

Web crawler method and device based on artificial intelligence, equipment and medium

A web crawler and artificial intelligence technology, applied in the field of artificial intelligence, can solve problems such as crawler settings, less data content, and low crawling efficiency

Pending Publication Date: 2022-04-22
PING AN TECH (SHENZHEN) CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the prior art, the required data content is often crawled by directly using the crawler program. However, in the crawling process, no specific settings are made for the depth of the crawler, resulting in the depth of the crawled web pages being too deep. To a large number of web pages, but the real data content is very little, resulting in low crawling efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Web crawler method and device based on artificial intelligence, equipment and medium
  • Web crawler method and device based on artificial intelligence, equipment and medium
  • Web crawler method and device based on artificial intelligence, equipment and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0057]In order to understand the purpose, features and advantages of the present application more clearly, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. Many specific details are set forth in the following description to facilitate a full understanding of the present application, and the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0058] In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a web crawler method and device based on artificial intelligence, electronic equipment and a storage medium, the web crawler method based on artificial intelligence comprises the steps that crawler parameters are configured based on an intelligent search engine, and the crawler parameters comprise a target website needing to be grabbed and a target field in a webpage; generating a first crawler program based on the crawler parameters and a preset crawler template; optimizing the first crawler program according to a machine learning algorithm to generate a second crawler program; determining the crawling depth of the second crawler program according to a preset crawler index to obtain a third crawler program; and performing data crawling based on the third crawler program to obtain a crawler log. According to the method, the crawling depth of the crawler program can be determined under the condition of ensuring the accuracy of the crawler data, so that the crawling efficiency of the data is improved.

Description

technical field [0001] The present application relates to the technical field of artificial intelligence, and in particular to an artificial intelligence-based web crawler method, device, electronic equipment, and storage medium. Background technique [0002] Ansible is an open source product, which is used to automatically execute resource configuration management and application deployment. In the process of deploying cloud platforms in batches, in order to ensure the high operating performance of cloud platforms, Ansible and crawler technology are generally used in combination. The crawler technology realizes the parameter capture required by Ansible, so that Ansible can perform batch deployment based on the captured parameter information. [0003] In the prior art, the required data content is often crawled by directly using the crawler program. However, in the crawling process, no specific settings are made for the depth of the crawler, resulting in the depth of the cra...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/951G06K9/62G06N20/00
CPCG06F16/951G06N20/00G06F18/24323G06F18/214
Inventor 黄日华
Owner PING AN TECH (SHENZHEN) CO LTD