Rapid search method for mass images based on Hadoop platform

A Hadoop cluster and platform technology, applied in the field of fast retrieval of massive images, to solve low retrieval efficiency, solve excessive memory consumption, and achieve the effect of performance

Inactive Publication Date: 2017-06-16
SHANDONG BUSINESS INST
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Aiming at the performance bottleneck problem in the retrieval of massive images, the present invention proposes a Hadoop-based massive image retrieval method, realizes the merging of small images through Sequence, and sets the offset of a single Sequence File during the merging process to quickly locate the index DataNode and Fileld that store image blocks solve the problem of massive image data expansion and fast retrieval

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Rapid search method for mass images based on Hadoop platform
  • Rapid search method for mass images based on Hadoop platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0018] First: Deploy the Hadoop cluster. After the system is deployed, check the network to ensure that the machines in the cluster can communicate with each other. Install SSH and configure SSH password-free login. Add the IP host mapping relationship to the end of the etc / hosts file to install the Java environment. Add export JAVA_HOME= / usr / jdk1.6.0 at the end of conf / hadoop-env.sh, add testA to the master file, add test1, test2, and test3 to the slaves file, and modify the conf / core-site.xml file.

[0019] Second: Install Redis. Download Redis, copy it to the corresponding directory, install, compile and start the service.

[0020] Third: Install HAProxy. Download haproxy, copy it to the corresponding directory, compile and install.

[0021] Fourth: The client first initiates a write data request to the NameNode. After filtering by the load balancing module, it first arrives at the application server and waits in line to enter the HDFS storage system. After the request...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the field of computer big data processing, in particular to a rapid search method for mass images based on a Hadoop platform. The method comprises the steps that 1, a Hadoop cluster platform is built; 2, a security scheme is set; 3, single-image storage processing is performed; 4, file preprocessing and combining are performed; 5, an image index is built; 6, a client side adopts image names and creating time as parameters for initiating an access request, Name Node operation is performed to obtain a minute time period where the images are located and Blocks information corresponding to a combined file, and the minute time period and the Blocks information are returned back to the client side. The problem that Name Node memory consumption is excessive and the search efficiency is low during Hadoop search of mass images can be well solved, the Name Node load during search is effectively reduced, the Name Node performance is improved, and therefore wider application range of the Hadoop platform is achieved.

Description

technical field [0001] The invention relates to the field of computer big data processing, in particular to a fast retrieval method for massive pictures based on the Hadoop platform. Background technique [0002] With the popularization and wide application of the Internet, e-commerce platforms and social networks have also continued to develop, and the number of pictures used for product display or social sharing has exploded. On these e-commerce sites and social networking sites, the information expression of pictures far exceeds the description of text information, so these e-commerce sites and social networking sites pay more attention to the quality of pictures. From the analysis of Taobao, in the traffic of the entire business platform, visits to pictures are as high as 91.5%. Tencent photo album users also upload 1.1 billion pictures every week, and the current total number of pictures is nearly 70 billion, with a total capacity of up to 15PB. Since a large number o...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 孙玉林徐宝华贾春朴张福元陈守森
Owner SHANDONG BUSINESS INST
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap