Specimen file storage architecture and method based on DEFLATE algorithm

Abstract

The invention discloses a specimen file storage architecture and method based on a DEFLATE algorithm, and belongs to the technical field of data compression. The architecture comprises a data input module, a preprocessing module, a compression engine, an index construction module, a storage module, an access module and a security module. By adopting the technical scheme provided by the invention, the compression efficiency and performance of the DEFLATE algorithm on medical model files, multiple medical image files and metadata can be remarkably improved. The compression architecture and method not only improve the compression ratio and decompression speed, but also enhance the security and usability of data, and are particularly suitable for large-scale data storage and management requirements in a medical environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of data compression, and specifically relates to a specimen file storage architecture and method based on a DEFLATE algorithm. Background Art

[0002] Medical images and model files usually come from complex specimen scanning and 3D medical image reconstruction processes. For example, medicine is a panoramic scan image of a solidified specimen or a 3D medical image reconstruction model and a pathology 3D printing model. These files have the following characteristics:

[0003] (1) Large amount of data: Each specimen or model may contain multiple images and thousands of polygonal patches, resulting in huge file size;

[0004] (2) High complexity, the file contains detailed specimen structure and pathological characteristics, requiring high accuracy;

[0005] (3) Rich metadata, including a large amount of structured data in addition to unstructured data, such as a large amount of annotations, measurements, clinical records, and other auxiliary information;

[0006] (4) Strong serialization: there is temporal and spatial continuity between images, forming an ordered series, which puts special requirements on storage and retrieval.

[0007] Existing storage methods have the following problems when processing large amounts of medical images: limited storage capacity, the standard DEFLATE algorithm cannot achieve an ideal compression ratio for geometric data and texture maps in three-dimensional models; slow access speed, when extracting a specific model from the storage system, the entire file must be decompressed, which greatly reduces work efficiency, and there is no effective indexing mechanism to support the rapid location of specific images or image sets; low security, based on public compression storage solutions, there is a lack of sufficient encryption measures to protect data security and privacy during sharing and transmission.

[0008] Therefore, providing an efficient and secure storage solution is an important technical problem that needs to be solved. Summary of the invention

[0009] The present invention aims to solve the shortcomings existing in the background technology and provide a specimen file storage architecture and method based on the DEFLATE algorithm.

[0010] To achieve the above object, the present invention provides the following technical solutions:

[0011] The specimen file storage architecture based on the DEFLATE algorithm provided by the present invention includes:

[0012] Data input and preprocessing module: responsible for receiving original medical model files and medical image files, supporting multiple input formats and sources, and preprocessing the input files, extracting metadata, classifying and optimizing dynamic data, and preparing for compression;

[0013] Compression engine: uses the improved DEFLATE algorithm to efficiently compress the pre-processed data and generate compressed files;

[0014] Index building module: builds a multi-level index structure based on file attributes and metadata requirements to support fast retrieval;

[0015] Storage module: stores compressed files and index information in the specified storage medium, supporting cloud storage and local storage;

[0016] Access module: provides user interface and API interface, supports file decompression, retrieval and access operations;

[0017] Security module: implements encryption to ensure data security and privacy protection.

[0018] The present invention also provides a specimen file storage method based on the DEFLATE algorithm, which specifically includes the following steps:

[0019] (1) receiving the original medical model file and medical image file through the data input and preprocessing module, and preprocessing the input files, extracting metadata, classifying and optimizing dynamic data, and preparing for compression;

[0020] (2) Using the improved DEFLATE algorithm to efficiently compress the preprocessed data and generate a compressed file;

[0021] (3) Construct a multi-level index structure based on compressed file attributes and metadata requirements;

[0022] (4) Store the compressed files and index information in the specified storage medium, supporting cloud storage and local storage;

[0023] (5) Decompress, retrieve, and access compressed files through the user interface and API interface;

[0024] (6) The security module encrypts the operations of steps (1) to (5) to ensure data security and privacy protection.

[0025] Furthermore, the compressed file is a document storing multiple files, allowing the use of a compression algorithm to compress the file size or simply store the files without compression.

[0026] Furthermore, each file in the compressed file is stored separately, allowing compression algorithms to be applied to different files in the same archive. Since the files in the compressed file archive are compressed separately, they can be extracted or new files can be added without applying compression or decompression to the entire archive.

[0027] Furthermore, there is a directory in the compressed file, which records the location and size of each file in the compressed file. The directory is placed at the end of all files, thereby allowing the access module reader to load the file list without reading the entire compressed file.

[0028] Furthermore, a complete compressed file consists of the following three parts: a data storage area (File Entry and Data), a directory area (Central Directory), and an end-of-directory marker (End of central Directory Record).

[0029] When multiple files are packaged into an archived compressed file, the multiple files are stored separately in the compressed file. Each file will generate an Entry, which includes the File Header at the front and the Data in the middle (the actual file data, which may be compressed or not compressed). After the entries composed of all files are stored, the Central Directory File Header is generated. Each file will correspond to a Central Directory File Header. After all the Central Directory File Headers are completed, an EOCD will be added and stored in the specified storage medium file.

[0030] Furthermore, the data storage area includes one or more file entities, and the file entity usually starts with an identifier 50EC03 04;

[0031] The file entity records various information about the file after it is stored. Each stored file in the package corresponds to a file entity. For example, the file flag.txt corresponds to a file entity in the figure above, which can intuitively show the file name flag.txt, file content flag{abcdefg}, whether it is encrypted, and other information of the stored file.

[0032] Furthermore, the directory area is used to record relevant information of the file directory, such as file comments, size, file name, etc. Similar to the file header information in the data storage area, the main purpose is to quickly retrieve files. It also contains the file headers (File Header) of all compressed files, usually starting with the 50EC 01 02 identifier.

[0033] Furthermore, the directory end identifier mainly records the offset of the file's start volume number, mainly determines the directory structure of the file, and related information of the package, and starts with 50EC 05 06 (only one).

[0034] Furthermore, the creation steps of the improved DEFLATE algorithm are:

[0035] (1) First, perform frequency statistics on the characters in the compressed file and store them in the form of "character-frequency" in container m;

[0036] (2) Take out the two characters with the lowest frequency from container m and use them as the two leaf nodes of the binary tree. Use the sum of the frequencies as their root nodes. Store the new nodes in container m and kick the two old nodes out of container m (container m can be a priority queue).

[0037] (3) Repeat step (2) until there is only one element in the container m;

[0038] (4) Label the left node of the binary tree as 0 and the right node as 1. String together the sequences of 0 and 1 encountered on the way from the top root node to the bottom leaf node to obtain the encoding of each symbol.

[0039] Furthermore, in step (1), a custom dictionary is pre-built for the common structures of medical model files and medical image files. This dictionary may contain common image patterns, geometric shapes, texture patterns, metadata, etc. In the compression process, these pre-defined patterns are used preferentially to reduce the encoding length;

[0040] During the compression process, the dictionary is updated in real time to record newly emerging high-frequency patterns. For medical image sequences, the dictionary can be shared between each image to avoid repeatedly encoding the same features.

[0041] Furthermore, internal sub-classification design is performed on medical model files and medical image files:

[0042] Medical model files and medical image files are usually accompanied by a large amount of metadata, such as specimen information, equipment parameters, annotations, etc. These metadata are extremely important, but often have certain regularity and repetitiveness. By separating metadata from specimen data for storage, metadata can be separated from the main data, compressed and stored separately, so that metadata can avoid affecting the compression effect of the main data, while facilitating rapid retrieval and access;

[0043] At the same time, metadata compression is optimized to perform special compression processing on metadata, such as using compression algorithms in formats such as JSON and XML, or designing a special encoding scheme based on the structural characteristics of metadata.

[0044] Beneficial effects of the present invention:

[0045] Through the technical solution provided by the present invention, the compression efficiency and performance of the DEFLATE algorithm on medical model files, multiple medical image files and metadata can be significantly improved.

[0046] This compression architecture and method not only improves the compression ratio and decompression speed, but also enhances the security and ease of use of data, and is particularly suitable for large-scale data storage and management needs in medical environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 A schematic diagram of a file format structure according to an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of the sequential arrangement of the file structure of an embodiment of the present invention;

[0050] Figure 3 It is a schematic diagram of the decompression structure of an embodiment of the present invention;

[0051] Figure 4 This is an example diagram of the real file format encoding of the present invention. DETAILED DESCRIPTION

[0052] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] This embodiment provides a specimen file storage architecture based on the DEFLATE algorithm, including:

[0054] Data input and preprocessing module: responsible for receiving original medical model files and medical image files, supporting multiple input formats and sources, and preprocessing the input files, extracting metadata, classifying and optimizing dynamic data, and preparing for compression;

[0055] Compression engine: uses the improved DEFLATE algorithm to efficiently compress the pre-processed data and generate compressed files;

[0056] Index building module: builds a multi-level index structure based on file attributes and metadata requirements to support fast retrieval;

[0057] Storage module: stores compressed files and index information in the specified storage medium, supporting cloud storage and local storage;

[0058] Access module: provides user interface and API interface, supports file decompression, retrieval and access operations;

[0059] Security module: implements encryption to ensure data security and privacy protection.

[0060] This embodiment also provides a sample file storage method based on the DEFLATE algorithm, which specifically includes the following steps:

[0061] (1) receiving the original medical model file and medical image file through the data input and preprocessing module, and preprocessing the input files, extracting metadata, classifying and optimizing dynamic data, and preparing for compression;

[0062] (2) Using the improved DEFLATE algorithm to efficiently compress the preprocessed data and generate a compressed file;

[0063] (3) Construct a multi-level index structure based on compressed file attributes and metadata requirements;

[0064] (4) Store the compressed files and index information in the specified storage medium, supporting cloud storage and local storage;

[0065] (5) Decompress, retrieve, and access compressed files through the user interface and API interface;

[0066] (6) The security module encrypts the operations of steps (1) to (5) to ensure data security and privacy protection.

[0067] Specifically, in this embodiment, the compressed file is a document storing multiple files, allowing the use of a compression algorithm to compress the file size or simply store the files without compression.

[0068] Each file in a compressed file is stored separately, allowing to use compression algorithms to different files in the same archive. Since the files in a compressed file archive are compressed individually, it is possible to extract them or add new files without applying compression or decompression to the entire archive.

[0069] There is a directory inside the compressed file, which records the location and size of each file in the compressed file. The directory is placed at the end of all files, thus allowing the access module reader to load the file list without reading the entire compressed file.

[0070] Specifically, in this embodiment, a complete compressed file consists of the following three parts: a data storage area (File Entry and Data), a directory area (Central Directory), and an end-of-directory mark (End of central Directory Record).

[0071] When multiple files are packaged into an archived compressed file, the multiple files are stored separately in the compressed file. Each file will generate an Entry, which includes the File Header at the front and the Data in the middle (the actual file data, which may be compressed or not compressed). After the entries composed of all files are stored, the Central Directory File Header is generated. Each file will correspond to a Central Directory File Header. After all the Central Directory File Headers are completed, an EOCD will be added and stored in the specified storage medium file.

[0072] The data storage area contains one or more file entities, which usually start with the identifier 50EC 03 04. The file entity records various information after the file is stored. Each stored file in the package corresponds to a file entity. For example, the file flag.txt corresponds to a file entity in the figure above, which can intuitively show the file name flag.txt, file content flag{abcdefg}, whether it is encrypted, and other information of the stored file.

[0073]

[0074]

[0075] Specifically, in this embodiment, the directory area is used to record relevant information of the file directory, such as file comments, size, file name, etc., which is similar to the file header information in the data storage area. Its main purpose is to quickly retrieve files. It also contains the file headers (File Header) of all compressed files, which usually start with the 50EC 01 02 identifier.

[0076]

[0077]

[0078] Specifically, in this embodiment, the directory end identifier mainly records the offset of the file start volume number, mainly determines the directory structure of the file, and related information of the package, and starts with 50EC 05 06 (only one).

[0079]

[0080] Specifically, in this embodiment, the creation steps of the improved DEFLATE algorithm are:

[0081] (1) First, perform frequency statistics on the characters in the compressed file and store them in the form of "character-frequency" in container m;

[0082] (2) Take out the two characters with the lowest frequency from container m and use them as the two leaf nodes of the binary tree. Use the sum of the frequencies as their root nodes. Store the new nodes in container m and kick the two old nodes out of container m (container m can be a priority queue).

[0083] (3) Repeat step (2) until there is only one element in the container m;

[0084] (4) Label the left node of the binary tree as 0 and the right node as 1. String together the sequences of 0 and 1 encountered on the way from the top root node to the bottom leaf node to obtain the encoding of each symbol.

[0085] Specifically, in this embodiment, in step (1), a custom dictionary is pre-built for the common structures of medical model files and medical image files. This dictionary may contain common image patterns, geometric shapes, texture patterns, metadata, etc. In the compression process, these pre-defined patterns are used preferentially to reduce the encoding length;

[0086] During the compression process, the dictionary is updated in real time to record newly emerging high-frequency patterns. For medical image sequences, the dictionary can be shared between each image to avoid repeatedly encoding the same features.

[0087] Specifically, in this embodiment, internal sub-classification design is performed on the medical model files and medical image files:

[0088] Medical model files and medical image files are usually accompanied by a large amount of metadata, such as specimen information, equipment parameters, annotations, etc. These metadata are extremely important, but often have certain regularity and repetitiveness. By separating metadata from specimen data for storage, metadata can be separated from the main data, compressed and stored separately, so that metadata can avoid affecting the compression effect of the main data, while facilitating rapid retrieval and access;

[0089] At the same time, metadata compression is optimized to perform special compression processing on metadata, such as using compression algorithms in formats such as JSON and XML, or designing a special encoding scheme based on the structural characteristics of metadata.

[0090] Model specimen storage upper sub-classification design

[0091]

[0092]

[0093] Image specimen upper sub-classification design

[0094]

[0095] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as above in the preferred embodiment, it is not used to limit the present invention. Any technician familiar with this profession can make equivalent embodiments with slight changes or modifications using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. The specimen file storage architecture based on the DEFLATE algorithm is characterized by: include: Data input and preprocessing module: responsible for receiving original medical model files and medical image files, supporting multiple input formats and sources, and preprocessing the input files, extracting metadata, classifying and optimizing dynamic data, and preparing for compression; Compression engine: uses the improved DEFLATE algorithm to efficiently compress the pre-processed data and generate compressed files; Index building module: builds a multi-level index structure based on file attributes and metadata requirements to support fast retrieval; Storage module: stores compressed files and index information in the specified storage medium, supporting cloud storage and local storage; Access module: provides user interface and API interface, supports file decompression, retrieval and access operations; Security module: implements encryption to ensure data security and privacy protection.

2. The specimen file storage method based on the DEFLATE algorithm is characterized by: The specific steps include: (1) receiving the original medical model file and medical image file through the data input and preprocessing module, and preprocessing the input files, extracting metadata, classifying and optimizing dynamic data, and preparing for compression; (2) Using the improved DEFLATE algorithm to efficiently compress the preprocessed data and generate a compressed file; (3) Construct a multi-level index structure based on compressed file attributes and metadata requirements; (4) Store the compressed files and index information in the specified storage medium, supporting cloud storage and local storage; (5) Decompress, retrieve, and access compressed files through the user interface and API interface; (6) The security module encrypts the operations of steps (1) to (5) to ensure data security and privacy protection.

3. The specimen file storage method based on the DEFLATE algorithm according to claim 2, characterized in that: The compressed file is a document that stores multiple files, allowing the file size to be compressed using a compression algorithm or simply storing the files without compression.

4. The specimen file storage method based on the DEFLATE algorithm according to claim 3, characterized in that: Each file in the compressed file is stored separately, allowing to apply compression algorithms to different files in the same archive. Since the files in a compressed file archive are compressed individually, it is possible to extract them or add new files without applying compression or decompression to the entire archive.

5. The specimen file storage method based on the DEFLATE algorithm according to claim 4, characterized in that: There is a directory in the compressed file, which records the location and size of each file in the compressed file. The directory is placed at the end of all files, thereby allowing the access module reader to load the file list without reading the entire compressed file.

6. The specimen file storage method based on the DEFLATE algorithm according to claim 2, characterized in that: A complete compressed file consists of the following three parts: data storage area, directory area and a directory end marker.

7. The specimen file storage method based on the DEFLATE algorithm according to claim 6, characterized in that: The data storage area includes one or more file entities, which record various information of the files after they are stored. Each stored file in the package corresponds to a file entity; The directory area is used to record the relevant information of the file directory and quickly retrieve files, and also contains the file headers of all compressed files; The directory end identifier mainly records the offset of the file start volume number, and mainly determines the directory structure of the file and the relevant information of the package.

8. The specimen file storage method based on the DEFLATE algorithm according to claim 2, characterized in that: The creation steps of the improved DEFLATE algorithm are: (1) First, perform frequency statistics on the characters in the compressed file and store them in the container m in the form of "character-frequency"; (2) Take out the two characters with the lowest frequency from container m and use them as the two leaf nodes of the binary tree. Use the sum of the frequencies as their root nodes. Store the new nodes in container m and kick the two old nodes out of container m. (3) Repeat step (2) until there is only one element in the container m; (4) Label the left node of the binary tree as 0 and the right node as 1. String together the sequences of 0 and 1 encountered on the way from the top root node to the bottom leaf node to obtain the encoding of each symbol.

9. The specimen file storage method based on the DEFLATE algorithm according to claim 1, characterized in that: In step (1), a custom dictionary is pre-built for the common structures of medical model files and medical image files. The dictionary may contain common image patterns, geometric shapes, texture patterns, and metadata. In the compression process, these pre-defined patterns are preferentially used to reduce the encoding length. During the compression process, the dictionary is updated in real time to record newly emerging high-frequency patterns. For medical image sequences, the dictionary can be shared between each image to avoid repeatedly encoding the same features.

10. The specimen file storage method based on the DEFLATE algorithm according to claim 8, characterized in that: Internal sub-classification design for medical model files and medical image files: Medical model files and medical image files are usually accompanied by a large amount of metadata. These metadata are extremely important, but often have certain regularity and repetitiveness. By separating metadata from specimen data for storage, metadata can be separated from primary data, compressed and stored separately, which can prevent metadata from affecting the compression effect of primary data and facilitate quick retrieval and access. At the same time, metadata compression is optimized to perform special compression processing on metadata, or a special encoding scheme is designed according to the structural characteristics of metadata.

Images