Unlock instant, AI-driven research and patent intelligence for your innovation.

A file processing method and device

A file processing and file technology, applied in file systems, file access structures, electronic digital data processing, etc., can solve the problems of long time consumption, high memory access pressure on Namenode nodes, and low file reading efficiency, so as to relieve work pressure, The effect of improving processing speed and access efficiency

Active Publication Date: 2020-03-20
HANDAN BRANCH OF CHINA MOBILE GRP HEBEI COMPANYLIMITED
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In this way, when accessing a stored file, the Namenode needs to read all the metadata and select the metadata information of the accessed file. The memory access pressure of the Namenode node is high, and it is necessary to scan all the metadata of the Namenode node when judging whether a small file exists. It takes a long time and the file reading efficiency is not high

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A file processing method and device
  • A file processing method and device
  • A file processing method and device

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0048] see figure 1, which shows the first flow chart of the first embodiment of the file processing method of the present invention, the flow may include:

[0049] Step 100: Get at least two files.

[0050] Here, the Namenode node of the Hadoop cluster can be used to obtain the file. For example, the Namenode node of the Hadoop cluster obtains the file through ftp downloading. At this time, the obtained file is stored in the local storage directory of the Hadoop cluster.

[0051] In this step, the source of the obtained file is not limited, for example, the file may be obtained from an interface machine.

[0052] Step 101: Merge the files meeting the file merging conditions to obtain a merged file.

[0053] Specifically, the acquired files are classified; among the files of each category, the files whose capacity is smaller than the capacity threshold are marked as the files to be merged in the corresponding category; the sum of the capacities of the files to be merged in a...

no. 2 example

[0091] see Figure 4 , which shows a schematic diagram of the composition and structure of the file processing device in the embodiment of the present invention, the device includes: an acquisition module 200, a derivation module 201 and a storage module 202;

[0092] Obtaining module 200, configured to obtain at least two files.

[0093] The deriving module 201 is used for merging the files satisfying the file merging condition to obtain the merged file.

[0094] The storage module 202 is configured to store the merged file in the form of BloomMapFile.

[0095] Specifically, the deriving module 201 is configured to classify the acquired files; among the files of each category, mark the files whose capacity is smaller than the capacity threshold as the files to be merged in the corresponding category; When the sum of the capacity of the files to be merged reaches the capacity threshold, use the BloomFilter to merge the files to be merged in the corresponding category to obta...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Embodiments of the present invention disclose a file processing method. The method comprises: acquiring at least two files; when the acquired files meet a file merge condition, merging each file that meets the file merge condition, and obtaining a merged file; and storing the merged file in the form of BloomMapFile. The embodiments of the present invention further disclose a file processing apparatus.

Description

technical field [0001] The invention relates to data service technology, in particular to a file processing method and device. Background technique [0002] With the rapid development of Internet technology and the continuous increase of digital information, the storage of information has become one of the most concerned focuses; at present, the storage of such file data is mainly managed by deploying distributed file systems. There are many distributed file systems, such as Google File System (GFS), Hadoop Distributed File System (HDFS), Lustre, Fast Distributed File System (FDFS), etc. Among them, HDFS is one of the most important components in Hadoop. As a distributed file system, HDFS has attracted more and more attention in its development speed and application fields. [0003] File storage in HDFS will inevitably generate corresponding metadata. The existing technical solutions store metadata on Namenode nodes. When accessing stored files, Namenode needs to read all m...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/13G06F16/182
CPCG06F16/13G06F16/182
Inventor 张琳陈保符刘婕
Owner HANDAN BRANCH OF CHINA MOBILE GRP HEBEI COMPANYLIMITED