Unlock instant, AI-driven research and patent intelligence for your innovation.

Batch file storage method and system

A file storage and file technology, applied in the field of batch file storage methods and systems, can solve the problems of high implementation and operation costs, clearing old file performance defects, etc., achieve low implementation and operation and maintenance costs, optimal disk space maintenance algorithms, The effect of low system complexity

Pending Publication Date: 2020-12-01
西安光启智能技术有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Defects in FastDFS: performance defects in clearing old files when storing a large number of small files
[0012] High implementation and operating costs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Batch file storage method and system
  • Batch file storage method and system
  • Batch file storage method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0048] figure 1 It is a flow chart of a batch file storage method of the present invention. like figure 1 As shown, a batch file storage method, comprising steps: S11, setting the number of bytes of the file to be stored; S12, registering the service node; S13, searching for available service nodes; S14, gathering the file to be stored; S15. Store the aggregation file in an available service node. Store aggregated files to available service nodes, which is the process of saving small files. By setting the number of bytes of the unit file to be stored, customizing the size of small files, registering service nodes and using available service nodes, small files are aggregated and then saved, thus making full use of disk space; small disk fragments; implementation and operation and maintenance costs Inexpensive; the system has low complexity, and can be generally applied to the storage of a large number of small files generated by AI systems, Internet of Things and other syste...

Embodiment 2

[0086] Figure 7 It is a structural schematic diagram of a batch file storage system of the present invention. like Figure 7 As shown, a batch file storage system includes a client, a registration center, and a service node. The client maintains a list of service nodes and the path of the unit file to be stored in the system; the registration center implements the service node The service process of registering and searching service nodes; the service nodes are used to register, maintain and cancel themselves with the registration center, save the unit files to be stored, and aggregate the unit files to be stored into aggregated files, and the system performs the above-mentioned batch file storage method. The client uses the proxy mechanism to encapsulate the calling logic between the client application and the background, and uses an efficient IO communication framework for Socket communication. The client dynamically maintains a list of server nodes and implements a dyna...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a batch file storage method and system, and the method comprises the steps: setting the number of bytes of a to-be-stored unit file; registering the service node; searching foravailable service nodes; converging the unit files to be stored; and storing the aggregation file into the available service node. By adopting the batch file storage method and system, the writing efficiency of continuous high-concurrency massive small files is optimal; the disk space is fully utilized; the disk fragments are minimum; the implementation, operation and maintenance cost is low; thesystem is low in complexity and can be universally suitable for storing a large number of small files generated by an AI system, the Internet of Things and other systems, the file content refers to various information data capable of being statically stored, the content is voice, short videos, texts, pictures and other types of information, and the information organization form is partially structured or unstructured information. Wherein the information comprises picture information, sensor data information, position information, sound information, short video information and the like collected by the Internet of Things.

Description

【Technical field】 [0001] The invention relates to the technical field of image storage, in particular to a batch file storage method and system. 【Background technique】 [0002] With the development of science and technology, the application fields of artificial intelligence (AI) are constantly expanding. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. During the generation process of AI video, a large number of small pictures will be generated, and the massive small pictures need to maintain a continuous and stable storage and access state under high concurrency. [0003] There are currently two solutions for processing the massive small images generated in AI videos: FastDFS and Ceph. [0004] FastDFS solution: FastDFS is a lightweight distributed file storage system, suitable for writing a large number of small files and reading and accessing high-c...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/182G06F16/11
CPCG06F16/11G06F16/182
Inventor 刘若鹏栾琳季春霖晁汪邦
Owner 西安光启智能技术有限公司