Data extraction method and device, and computer readable storage medium
A data extraction and data retrieval technology, applied in the field of big data processing, can solve the problems of inability to crawl a large amount of data, increase the development difficulty and development time, and spend a lot of time for development engineers, so as to meet the needs of a large amount of data crawling and reduce development costs. Effects of difficulty and development time
Inactive Publication Date: 2019-03-29
中电科嘉兴新型智慧城市科技发展有限公司
View PDF4 Cites 4 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
However, as enterprises continue to increase their demand for data, for example, when they need to crawl data from multiple scattered web pages or websites, traditional crawlers cannot satisfy
At this time, it is necessary to write a set of specific crawler rules for each web page, which requires a lot of time for development engineers, greatly increases the difficulty and time of development, and cannot meet the needs of a large amount of crawling data.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0043] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude one or Presence or addition of multiple other features, integers, steps, operations, elements, components and / or collections thereof. It should also be understood that the terminology used in the de...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The embodiment of the invention discloses a data extraction method and device and a computer-readable storage medium. The method comprises the following steps: URLs of a plurality of pages to be crawled are requested through an http mode, and a plurality of pages to be crawled are rendered through a rendering server; a plurality of pages to be crawled after rendering are downloaded to obtain a plurality of page data; The data of multiple pages are analyzed intelligently to obtain the position of data to be crawled. A target data is extracted according to that position of the data to be crawl.The embodiment of the invention can quickly realize the crawling of a large amount of web page data, solves the problem that the crawling rules need to be manually written for each page in the prior art, reduces the development difficulty and the development time, and meets the requirement of the crawling of a large amount of data.
Description
technical field [0001] The present invention relates to the technical field of big data processing, in particular to a data extraction method, device and computer-readable storage medium. Background technique [0002] With the rise of big data technology, enterprises and individuals have an increasing demand for data. When obtaining data, non-professionals often use manual copying and pasting. When the amount of data is particularly small, manual operation can solve the problem, but when the amount of data becomes large, manual operation cannot meet the demand. For enterprises, crawler technology is usually used for data acquisition. [0003] At present, most crawler technologies and frameworks require developers to have better technology, and each page of the target website needs to be accurately analyzed, so as to find the location of each data in the page, and then extract the data through web page analysis technology . When the data required by the enterprise is limi...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & AuthorityApplications(China)
IPC IPC(8): G06F16/951G06N3/04
CPCG06N3/045
Inventor李栋陈昊姜雪平黄飞沈涵拯
Owner中电科嘉兴新型智慧城市科技发展有限公司