A storage method based on HDFS distributed file system and a use method

Abstract

A storage method based on an HDFS distributed file system and a use method are provided. The storage method includes: an integrated content storage file block structure, including: a content index table block, a sample volume data block, a basic information table block, a three-dimensional volume matrix block and a header information backup block; a file buffer pool based on the integrated contentstorage file block structure, including: a user queue, a user data queue and an HDFS connection pool; a file cache pool access process. The technology of the invention is built on the distributed file system, reduces the data storage cost, is easy to expand, supports the storage expansion of the non-stop state, and improves the safety of the redundant storage of the data storage. Through distributed data access, the concurrent access capability is greatly improved. Compared with a conventional centralized storage technology, the technology has better read-write performance and meets the online high concurrent access requirements. This technology is deployed on cloud platform and can quickly build application sharing platform to meet the distributed performance requirements of mobile application development for cloud storage.

Description

technical field [0001] The invention belongs to the field of medical image processing, and in particular relates to a storage method and a use method based on an HDFS distributed file system. Background technique [0002] With the rapid development of medical imaging technology, medical images have become an important basis for medical clinical diagnosis. These data are currently stored in the PACS (Picture Archiving and Communication System) system, using high-performance, large-capacity network storage arrays, tape libraries and other storage media. PACS follows the DICOM3.0 international standard, which is the organization and communication standard of medical images. [0003] At present, the PACS system has gradually developed from a single machine and departments to the whole hospital and regions, realizing the film-free hospital. Regionalization is the current main research goal of government health departments and medical institutions, but building a large shared me...

AI Technical Summary

Problems solved by technology

[0004] 1) High construction costs: the amount of image data reaches TB and PB levels, the cost of using traditional storage architectures (such as FC SAN / iSCSI) is high, and the flexibility of heterogeneous integration and expansion is poor;

[0005] 2) There is a bottleneck in the transmission bandwidth: Even the high-performance FC SAN network bandwidth and processing capacity are difficult to meet the fast processing and transmission requirements of PB series;

[0006] 3) Limited usability: Large-scale hospital PACS systems often use the "online-nearline-offline" storage mode. Most of the offline data is stored in the tape library. The usability is poor, and the data cannot be obtained in real time;

[0007] 4) Lack of an integrated application sharing platform: Medical image collaboration, such as Web DICOM terminals, image consultation, image referral, distance education, digital film storage and other services basically adopt a "point-to-point" model, lacking integration, cross-platform, Highly available regional medical imaging collaborative application software, data sharing is difficult, such as transfer, remote medical treatment, etc. can not transfer data online

[0012] 3) Data redundancy is high, and by default each data is backed up on 3 servers;

At present, Hadoop also provides corresponding solutions for small files, such as Hadoop Archive file archiving, SequenceFile files, etc., but these methods cannot fully meet the application requirements of medical DICOM sequence images, such as lack of content index and single random access

[0017] 2) HDFS is not suitable for real-time application problems

The concept of HDFS design is not suitable for real-time applications. During the data writing process, each data block needs to be copied at least 3 times. The writing performance is much lower than the reading performance. Therefore, the writing performance of HDFS is not real-time and not suitable for multi-task concurrency It is not suitable for PACS real-time applications that need to quickly obtain image resources and write diagnostic reports

At the same time, every time HDFS is accessed, the client needs to establish a connection, open, close and disconnect the connection. For a sequence of hundreds of images that are frequently read, compared with the local file system, the reading efficiency will be significantly reduced;

[0018] 3) Low efficiency of random reading and writing of HDFS file content

If you access small files, you must jump from one Datanode to another Datanode, which greatly reduces the read performance

Images