Rapid search method for mass images based on Hadoop platform

Abstract

The invention relates to the field of computer big data processing, in particular to a rapid search method for mass images based on a Hadoop platform. The method comprises the steps that 1, a Hadoop cluster platform is built; 2, a security scheme is set; 3, single-image storage processing is performed; 4, file preprocessing and combining are performed; 5, an image index is built; 6, a client side adopts image names and creating time as parameters for initiating an access request, Name Node operation is performed to obtain a minute time period where the images are located and Blocks information corresponding to a combined file, and the minute time period and the Blocks information are returned back to the client side. The problem that Name Node memory consumption is excessive and the search efficiency is low during Hadoop search of mass images can be well solved, the Name Node load during search is effectively reduced, the Name Node performance is improved, and therefore wider application range of the Hadoop platform is achieved.

Description

technical field [0001] The invention relates to the field of computer big data processing, in particular to a fast retrieval method for massive pictures based on the Hadoop platform. Background technique [0002] With the popularization and wide application of the Internet, e-commerce platforms and social networks have also continued to develop, and the number of pictures used for product display or social sharing has exploded. On these e-commerce sites and social networking sites, the information expression of pictures far exceeds the description of text information, so these e-commerce sites and social networking sites pay more attention to the quality of pictures. From the analysis of Taobao, in the traffic of the entire business platform, visits to pictures are as high as 91.5%. Tencent photo album users also upload 1.1 billion pictures every week, and the current total number of pictures is nearly 70 billion, with a total capacity of up to 15PB. Since a large number o...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the performance bottleneck problem in the retrieval of massive images, the present invention proposes a Hadoop-based massive image retrieval method, realizes the merging of small images through Sequence, and sets the offset of a single Sequence File during the merging process to quickly locate the index DataNode and Fileld that store image blocks solve the problem of massive image data expansion and fast retrieval

Images