Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and device for updating aggregation objects

An object and object header technology, applied in the field of updating aggregated objects, can solve the problems of analyzing metadata, taking a long time to read data, low disk space usage, and high pressure of disk writing data, so as to improve disk space usage and time-consuming Short, reducing the effect of writing pressure

Inactive Publication Date: 2017-05-24
ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
View PDF5 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The user experience is slow to read and write small files, and the disk space consumption is fast
In addition, the operation of a large number of small files will also cause serious system problems, because each small file needs to call the Rados interface once to perform a full write process, and the disk write data pressure is high. size, such as 4KB alignment, resulting in low disk space usage
Reading multiple small files frequently calls the read interface, and it takes a long time to analyze metadata and read data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for updating aggregation objects
  • Method and device for updating aggregation objects
  • Method and device for updating aggregation objects

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0039] Such as figure 1 As shown, a method of updating an aggregate object includes the following steps:

[0040] Step S101: Determine the source files that meet the conditions.

[0041] Step S102: Write the qualified source files into a large file to form an aggregation object.

[0042] Step S103: Save the aggregation object to disk.

Embodiment 2

[0044] Such as figure 2 As shown, a method of updating an aggregate object includes the following steps:

[0045] Step S201: Predefine thresholds required for source file aggregation, including predefined source file thresholds and aggregation object thresholds.

[0046] Step S202: Judging the size of the aggregation object, if the size of the aggregation object is greater than the threshold of the aggregation object, proceed to step S206 and directly write it to the disk; if the size of the aggregation object file is smaller than the threshold of the aggregation object, proceed to step S203 to read the source file , write the source file to the aggregate object.

[0047] Step S204: judging the source file that meets the conditions; it includes, judging the size of the source file, if the size of the source file is greater than the source file threshold, then proceed to step S206, directly write it to the disk, if the source file size is smaller than the source file threshol...

Embodiment 3

[0051] Such as image 3 As shown, a method of updating an aggregate object includes the following steps:

[0052] Step S301: Predefine thresholds required for source file aggregation, including predefined source file thresholds and time thresholds.

[0053] Step S302: Read the source file.

[0054] Step S303: Determine whether the source file size reaches the source file threshold, if the source file size is greater than the source file threshold, then proceed directly to step S306, and write it to disk, if the source file size is smaller than the source file threshold, then proceed to step S304, which will meet Conditional source files are written to one large file forming aggregate objects.

[0055] Step S305: Before saving the aggregation object to the disk, it also includes, according to the time of the last write to disk, judging whether the time interval reaches the time threshold, if so, go to step S306, and write the aggregation object to the disk for storage, if not...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the field of file storage, and discloses a method and a device for updating aggregation objects. By the aid of the method and the device, severe system problems due to a large quantity of small file operation in the prior art can be solved. The method includes judging eligible source files; writing the eligible source files into a large file to form aggregation objects; storing the aggregation objects into magnetic disks; predefining source file thresholds, aggregation object thresholds and time thresholds required by source file aggregation. The method and the device have the advantages that metadata information in the same aggregation objects can be updated in a unified manner, accordingly, writing pressures on clusters can be relieved, clients and interaction frequencies can be reduced, the small file writing efficiency can be improved, frequent small file operation can be prevented, small files can be written at one step and can be repeatedly read, and the space utilization rates of the magnetic disks can be increased; the sequential reading performance can be improved, reading interfaces do not need to be frequently invoked when the multiple small files are about to be read, and metadata analysis and data reading are short in time consumption.

Description

technical field [0001] The invention relates to the field of file storage, in particular to a method and device for updating aggregated objects. Background technique [0002] In some application scenarios, such as social shopping sites, radio and television, and online videos, text, pictures, music, etc. will be generated, and there are a large number of operations on small files. These scenarios are characterized by a large number of small files, generally below 1M in size, fast flow of small files, and frequent operations, that is, write once and read multiple times. The user experience is that small files are read and written slowly, and disk space is consumed quickly. In addition, the operation of a large number of small files will also cause serious system problems, because each small file needs to call the Rados interface once to perform a full write process, and the disk write data pressure is high. Sizes, such as 4KB alignment, result in low disk space usage. Read...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/1847
Inventor 李旭东李长春李晓伟
Owner ZHENGZHOU YUNHAI INFORMATION TECH CO LTD