Crawler retrieval and big data intelligent recommendation optimization processing method based on open source framework

Abstract

The invention relates to a crawler retrieval and big data intelligent recommendation optimization processing method based on an open source framework. The method comprises the steps: carrying out theresource crawler through the open source framework, and obtaining a needed target service resource; performing word segmentation on the obtained target service resources according to an NPL word segmentation technology to realize information word segmentation matching; and performing information screening and recommendation according to preset keywords, fields and weight values. The crawler retrieval and big data intelligent recommendation optimization processing method based on the open source framework is adopted; the network resource collection capability and the intelligent recommendationalgorithm function of the target user are improved; a web crawler technology is realized by combining an open source HttpClient technology and a python algorithm packet, so that part of labor investment and time cost are greatly reduced or even directly saved, and crawler resource management has relatively high flexibility and expandability; and intelligent recommendation algorithm scheduling is executed for the target user, so that on-demand filtering is realized and effective information is screened out.

Description

technical field [0001] The invention relates to the field of big data platforms, in particular to the field of resource acquisition, and specifically refers to a method for crawler retrieval and big data intelligent recommendation optimization processing based on an open source framework. Background technique [0002] With the advent of the era of network big data and the rapid development of business data of various enterprises and companies, in order to deal with the ever-increasing huge data, for the research and analysis of big data, various units need to continuously invest a lot of manpower, material resources and time costs, which are specifically reflected in the following aspects point: [0003] 1) Continuously invest a lot of human resources and time costs in order to obtain data resources; [0004] 2) Due to the continuous growth of the acquired data, it is necessary to extract and analyze effective resources to invest too much labor cost; [0005] 3) Due to the...

AI Technical Summary

Problems solved by technology

[0003] 1) Continuously invest a lot of human resources and time costs in order to obtain data resources;

[0004] 2) Due to the continuous growth of the acquired data, it is necessary to extract and analyze effective resources to invest too much labor cost;

[0008] PHP: Advantages: Low cost, strong expansion and implantability Disadvantages: For multi-threading, asynchronous support is not good enough concurrent processing

[0009] Java: Advantages: Perfect web crawler ecosystem Disadvantages: The Java language itself is cumbersome, with a large amount of code, and the cost of reconstruction is high

[0010] C / C++: Advantages: almost the strongest operating efficiency and performance Disadvantages: labor costs, low development efficiency and heavy post-maintenance

[0012] 1) Relying on inherent technology, the crawled resources are limited, and the impact of anti-crawlers will reduce the amount of resource acquisition

[0013] 2) Diversity, variability, and complexity of website structure increase the difficulty of data crawling

[0014] Crawled data is less effective, or contains irrelevant data such as website source code structure or advertisement images, etc.

Images