Data storage method based on lossless compression algorithm

Abstract

The application relates to the technical field of data storage, and discloses a data storage method based on a lossless compression algorithm, which receives data to be stored through a data interface, automatically identifies the data format and carries out feature analysis by using a data analysis tool to obtain the size, type and complexity of the data; through deep analysis of the data features, the application can flexibly select a suitable Huffman coding processing mode, optimize the compression process according to different data types and complexity; in addition, the application introduces an upper and lower context modeling optimization mechanism, dynamically adjusts the coding mode, and further improves the compression ratio and storage efficiency; through a block storage strategy and in combination with metadata management, the efficient storage and reliability verification of the data are ensured; the application not only solves the problem of low efficiency of the existing compression algorithm, but also improves the integrity and reliability of data storage, and is suitable for the storage requirements of large-scale data and diversified data.

Description

Technical Field

[0001] This invention relates to the technical field of data storage, and in particular to a data storage method based on a lossless compression algorithm. Background Technology

[0002] With the continuous advancement of information technology and the popularization of Internet applications, the demand for data generation and storage has exploded. In traditional data storage systems, data is often stored in its raw form, resulting in wasted storage space and low access efficiency. Therefore, how to effectively compress data to reduce storage space has become a key issue in the field of information technology.

[0003] In recent years, lossless data compression technology has gradually become one of the core technologies for solving this problem. Lossless compression technology can minimize the storage requirements of data through algorithms while ensuring data integrity and recoverability. Although lossless compression technology has solved the storage space problem to a certain extent, existing compression methods often suffer from insufficient efficiency, slow processing speed, and unsatisfactory compression ratio when faced with large-scale data, highly complex data, and diverse data formats. Existing technologies mostly rely on a single compression algorithm and cannot flexibly select the optimal compression strategy according to data characteristics, resulting in limited compression effects.

[0004] Current lossless compression technologies typically employ static or fixed encoding methods. This approach fails to fully leverage its advantages when dealing with different types of data. For example, while standard Huffman coding performs well in compressing static text, its compression efficiency and speed are often unsatisfactory when faced with complex structured or multimedia data. Furthermore, existing context modeling optimization methods are often limited to predefined models and cannot dynamically adjust to changes in different data types, thus limiting the effectiveness of the optimization process. More importantly, existing storage methods generally neglect metadata management during data storage, especially the recording of critical information such as data integrity and compression ratio, which is particularly important in distributed storage and big data application scenarios. Therefore, existing technologies have certain shortcomings when facing the requirements of data diversity, complexity, and storage reliability. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] Therefore, the present application solves the technical problem that the existing lossless data compression technology generally faces the problems of compression efficiency, storage reliability and diversified data processing.

[0008] To solve the above technical problems, the present application provides the following technical solutions: receiving data to be stored through a data interface, automatically identifying the data format and performing feature analysis using a data analysis tool, obtaining the data size, data type and complexity of the data;

[0009] Selecting a suitable Huffman encoding processing to receive the data according to the data type and the complexity, obtaining intermediate compressed data;

[0010] Inputting the intermediate compressed data into a context modeling function for encoding optimization to generate final compressed data;

[0011] Storing the final compressed data to a designated storage medium, and using a block storage strategy to divide the final compressed data into fixed-size data blocks, generating metadata for each data block, and then updating the state to record the storage information of all data;

[0012] The metadata at least contains block number, compression rate, timestamp and integrity check code.

[0013] As a preferred scheme of the data storage method based on the lossless compression algorithm, obtaining the data size, data type and complexity of the data includes:

[0014] Receiving multi-source data to be stored through an HTTP protocol data interface or an FTP protocol data interface, the multi-source data at least contains JSON, CSV, XML, image files and audio files;

[0015] Using the data analysis tool to automatically identify the data format and output the data type, the data analysis tool is file extension, MIME type, JSON library and CSV library, wherein the MIME type of the data is checked, the mimetypes library or the python-magic library is used to identify the data format, for text data, the content is read by parsing, and if the parsing is successful, the type is recorded;

[0016] Storing the received multi-source data as a byte array, calculating the number of bytes in the byte length function to obtain the data size D, and the mathematical expression formula is:

[0017] D=len(data)

[0018] Wherein, D represents the size of the data, i.e. the number of bytes, data is the data, and len() is the byte length function.

[0019] The complexity of the statistical data is obtained by the ratio of the structural information amount of the data to the data size D, and the calculation formula of the complexity is:

[0020]

[0021] Wherein, C represents the complexity, S represents the structural information amount of the data, and D is the data size.

[0022] As a preferred scheme of the data storage method based on the lossless compression algorithm, the structural information amount of the data at least includes the number of key-value pairs or the number of features in the data.

[0023] As a preferred scheme of the data storage method based on the lossless compression algorithm, if the output data type is text, a standard Huffman encoding is selected.

[0024] If the output data type is image or audio, an adaptive Huffman encoding is selected.

[0025] The to-be-processed data is then transmitted into the selected Huffman encoding for encoding processing, and intermediate compressed data D H is output.

[0026] As a preferred scheme of the data storage method based on the lossless compression algorithm, the intermediate compressed data D H is output, which includes:

[0027] The data is traversed, the occurrence frequency of each symbol is counted, and a symbol frequency table is established.

[0028] A Huffman tree is constructed according to the symbol frequency table, and a corresponding encoding table is generated.

[0029] The to-be-processed data is encoded using the encoding table, and intermediate compressed data D H is obtained, and the mathematical expression formula is:

[0030] D H = Huffman_encode(data)

[0031] Wherein, the efficiency of the Huffman encoding is directly related to the symbol frequency, and the higher the frequency of the symbol, the shorter the encoding, and the symbol includes characters and bytes, and Huffman_encode determines the corresponding Huffman encoding processing mode.

[0032] As a preferred scheme of the data storage method based on the lossless compression algorithm, the intermediate compressed data D HThe input is fed into a context modeling function, the frequency of symbol occurrence is counted, a dynamically updated frequency table is maintained, the context model is updated according to the frequency table, a plus-one smoothing technique is used to avoid zero frequency problem, the frequency distribution is adjusted using maximum likelihood estimation, dynamic adjustment is made at each encoding, encoding optimization is performed, including:

[0033]

[0034] where O represents the optimized compression ratio, and p(i) is the probability of occurrence of symbol i;

[0035] The probability distribution is updated to improve coding efficiency, reduce redundancy, and generate final compressed data D F according to the results of context modeling

[0036] D F = Context_Modeling(D H )

[0037] where D F is the final compressed data, and Context_Modeling() is the context modeling function.

[0038] As a preferred scheme of the data storage method based on the lossless compression algorithm according to the present application, the final compressed data D F is stored in a designated storage medium, the medium state is obtained by API call, and the medium state is selected according to the comparison result with the threshold value, wherein the medium state at least contains available space, storage speed and access demand, when the threshold value is less than the available space in the medium state, the final compressed data D F is written into the selected available space;

[0039] The stored final compressed data D F is divided into fixed-size data blocks;

[0040] Metadata is generated for each data block, which at least contains block number, compression ratio, timestamp and integrity check code;

[0041] Each data block is written, and the write pointer is updated to ensure sequential writing;

[0042] State update is performed to record the storage information of all data;

[0043] Index is established to quickly retrieve and access the stored data.

[0044] As a preferred scheme of the data storage method based on the lossless compression algorithm, the corresponding metadata is generated when each data block is divided, and the SHA256 hash value of each data block is calculated to generate an integrity check code, check the integrity of the data block, and record the check value, and the mathematical expression formula is:

[0045] Checksum = SHA256(data_block)

[0046] Checksum is the checksum, which is a fixed length of 256-bit output value calculated by the SHA256 hash function, that is, the check value, and data_block is the data block to be checked.

[0047] The hash value of each data block is recorded as part of the metadata to ensure that the data is not tampered with during transmission and storage.

[0048] The beneficial effects of the present application are:

[0049] 1. Through feature analysis, the most suitable processing method can be accurately selected according to different types of data (such as text, image, audio) and the complexity of the data (such as structured and unstructured), the extraction of data size can provide quantitative basis for subsequent storage and compression decision, and the identification of data type and complexity ensures the accurate matching of subsequent compression method, through accurate identification of data characteristics, the efficiency and accuracy of data processing are improved, which provides a solid foundation for subsequent optimized compression and storage, and reduces the possibility of incorrect judgment or inefficient compression.

[0050] 2. The most suitable encoding method can be selected for different data types (such as text, binary file, image) and complexity (such as simple or complex data structure), and the most suitable encoding method can effectively avoid the problem of low compression ratio and slow compression speed caused by improper encoding method, especially when facing complex structure or changing data, it can intelligently adjust dynamically to achieve better compression effect and improve the efficiency and resource utilization of data storage.

[0051] 3. The context modeling function can dynamically analyze the symbol distribution and context relationship in the data, and can adjust the encoding strategy according to the historical characteristics and appearance probability of the data in real time, so as to optimize the compression efficiency, through this optimization, the data redundancy is further reduced, the compression ratio is improved, and at the same time, the compressed data can maintain good data recovery while maintaining high efficient compression, and is suitable for more extensive data types and application scenarios.

[0052] 4、Through the introduction of the block storage strategy, the flexibility and reliability of data storage are improved, the data block storage can effectively reduce the influence of single data block damage or loss on the entire data set, improve the fault tolerance of the system, the independence of each data block makes the data more efficient in storage, transmission and recovery process, by generating metadata (such as block number, compression rate, timestamp and integrity check code) for each data block, accurate storage tracking and management can be provided to ensure the integrity and reliability of data, the record of metadata can accurately track the status of each data block during data recovery, quickly locate and repair damaged or missing parts, so as to ensure the high availability and high reliability of the system, in the distributed storage environment, the combination of block and metadata management can improve the performance and fault tolerance of the storage system, and optimize the allocation and use of storage resources. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0054] Figure 1 The flowchart of the data storage method based on the lossless compression algorithm shown in the present application;

[0055] Figure 2 The compression rate comparison diagram shown in the present application;

[0056] Figure 3 The storage efficiency comparison diagram shown in the present application. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments.

[0058] Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0059] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0060] In the traditional storage scheme, the data type and processing mode are usually determined by manual or preset parameters, which has errors and limitations of manual intervention and is not suitable for diversified and dynamically changing data types; in the prior art, Huffman coding is usually static, that is, based on fixed rules for coding, which cannot be flexibly adjusted according to the changes and characteristics of data, and in the traditional compression algorithm, the encoding process is usually based on a static or preset probability model.

[0061] The embodiment of the present application provides a data storage method based on a lossless compression algorithm, which can automatically select the most suitable Huffman coding variant (such as standard Huffman coding, dynamic Huffman coding or adaptive Huffman coding) by analyzing the data characteristics (such as type and complexity), thereby improving the compression efficiency and adaptability of the algorithm; the adaptive context modeling used can be automatically adjusted according to the real-time changes of the data stream, thereby improving the adaptability and flexibility of compression.

[0062] According to the embodiment of the present application, in combination with Figure 1 the flowchart shown, a data storage method based on a lossless compression algorithm specifically includes the following steps:

[0063] S1, receiving the data to be stored through a data interface, automatically identifying the data format and performing characteristic analysis by using a data analysis tool, obtaining the data size, data type and complexity of the data;

[0064] S2, selecting a suitable Huffman coding to process the received data according to the data type and complexity, obtaining intermediate compressed data;

[0065] S3, inputting the intermediate compressed data into a context modeling function for encoding optimization to generate final compressed data;

[0066] S4, storing the final compressed data to a specified storage medium, dividing the final compressed data into fixed-size data blocks by using a block storage strategy, generating metadata for each data block, and then updating the state to record the storage information of all data;

[0067] The metadata at least includes block number, compression rate, timestamp and integrity check code.

[0068] The implementation process and / or effects of some examples of the present application will be described in more detail below in combination with some preferred or optional examples of the present application.

[0069] Exemplary, in cloud storage platform, data storage is a crucial link, with a large number of multimedia files (such as images, audio, video) and other formats (such as text, log files, database export files) are frequently uploaded and downloaded, how to efficiently store these huge and diverse data, and at the same time ensure the integrity, validity and fast access of data has become a problem that cloud storage service providers need to solve.

[0070] In this embodiment, a data storage method based on lossless compression algorithm is proposed to improve storage efficiency, reduce storage space consumption, and provide integrity protection for stored data. Especially for various types of uploaded data (such as images, audio, text), the method can select appropriate encoding compression mode according to the type and structure characteristics of the data, and through further optimization processing (such as context modeling optimization, block storage, metadata generation) to realize efficient storage and access.

[0071] As an example, the cloud storage platform is set up to provide cloud storage services for a large social media company, and customers upload image, audio and text data through the platform; the types of data to be stored are:

[0072] Image file (JPEG format): 4MB;

[0073] Audio file (MP3 format): 10MB;

[0074] Text file (CSV and JSON format): 1MB (including user data and configuration information);

[0075] Storage medium: distributed storage system of cloud storage, with high availability and elastic expansion capability, each node has a storage space of 50GB, and the system supports high-speed storage and low-latency access;

[0076] The system receives these various types of data through the interface of HTTP protocol, receives the image file (4MB, JPEG format): through the data analysis tool, the system uses the MIME type (image / jpeg) and the file extension (.jpg) to identify the file as an image file, the system records the data size as 4MB, the data is considered as complex data, containing rich pixel information and compression characteristics;

[0077] Receive audio file (10MB, MP3 format): use MIME type (audio / mpeg) and file extension (.mp3) to identify audio file, data size is 10MB, audio data has high complexity, containing multi-band sound waveform data, and there is certain redundancy;

[0078] Receiving text files (CSV and JSON formats, total 1MB): The system automatically identifies as a text file by parsing the file format (CSV library, JSON library), the data size is 1MB, and the structure is relatively simple (stored in the form of key-value pairs);

[0079] Feature analysis:

[0080] Data size (D): D 图像 = 4MB, D 音频 = 10MB, D 文本 = 1MB;

[0081] Data type: image as JPEG, audio as MP3, text as CSV / JSON;

[0082] Complexity (C): Calculate the data complexity, formula:

[0083]

[0084] Where S is the structural information of the data, and D is the data size;

[0085] Image file (JPEG): Image files usually contain a lot of redundant data, assuming the structural information S is 2MB, then the complexity is:

[0086]

[0087] Audio file (MP3): Audio files have frequency band redundancy, assuming the structural information S is 4MB, then the complexity is:

[0088]

[0089] Text file (CSV / JSON): Text data structure is simple, assuming the structural information S is 0.3MB, then the complexity is:

[0090]

[0091] According to the above analysis, the system selects the corresponding Huffman encoding method:

[0092] Image file (JPEG): Select adaptive Huffman encoding, image data has high redundancy (e.g. adjacent pixels are usually similar);

[0093] Audio file (MP3): Select adaptive Huffman encoding, similar frequency bands and samples in audio data will increase redundancy, suitable for using adaptive Huffman encoding;

[0094] Text file (CSV / JSON): Standard Huffman encoding is chosen, as the redundancy of text data is relatively low;

[0095] Huffman encoding implementation process: For each data type, a Huffman tree is built by counting the frequency of symbols, and a corresponding encoding table is generated;

[0096] Image (JPEG): Suppose the frequency of each pixel in a certain image is:

[0097] f 像素1 = 0.4, f 像素2 = 0.3, f 像素3 = 0.2, f 像素4 = 0.1

[0098] A Huffman tree is built by frequency and an encoding table is generated;

[0099] Audio (MP3): Suppose the frequency of frequency bands in an audio signal is:

[0100] f 频段1 = 0.5, f 频段2 = 0.2, f 频段3 = 0.15, f 频段4 = 0.15

[0101] A Huffman tree is built by frequency and an encoding table is generated;

[0102] Text (CSV / JSON): Suppose the frequency of symbols in text data is:

[0103] f 字符1 = 0.3, f 字符2 = 0.2, f 字符3 = 0.25, f 字符4 = 0.25

[0104] Standard Huffman encoding is used for processing;

[0105] For a symbol i, suppose its frequency is f i , the Huffman encoding length L can be represented as:

[0106]

[0107] For example, if the frequency of a symbol is 0.4, its encoding length is:

[0108]

[0109] The intermediate compression data of each file is input into the context modeling function for optimization;

[0110] Audio file optimization: During each encoding of an audio file, the frequency table is updated, the add-one smoothing technique is applied, and the maximum likelihood estimation (MLE) is used to adjust the symbol probability distribution to avoid zero frequency issues.

[0111] For example, in an audio signal, the probability of a certain frequency component is P f = 0.3, and the probability distribution of this frequency may be affected by outdated data. After adjusting by maximum likelihood estimation, the updated probability distribution is:

[0112]

[0113] Image file optimization: The encoding process of an image is optimized by further adjusting the correlation of pixel blocks and the encoding table through context modeling.

[0114] Context modeling is used to model the correlation of each pixel block, adjust the frequency table, and reduce redundancy by dynamically adjusting the context model to further compress data.

[0115] Optimization formula: Assuming the original probability of a certain symbol i is P i , and the optimized probability after context modeling is P ′ i The optimized compression ratio O can be represented as:

[0116]

[0117] The final compressed data is stored in a designated storage medium and uses a block storage strategy.

[0118] Block storage: For example, if the final compressed data size is 15MB, it is divided into 4 data blocks, each of which is 4MB, and the last data block is slightly smaller than 4MB (e.g. 3MB).

[0119] The metadata of each data block includes:

[0120] Block number: 1, 2, 3, 4.

[0121] Compression rate: For each data block, calculate its compression rate, for example, the compression rate is 40%.

[0122] Timestamp: Record the timestamp of each data block, for example, the timestamp is "2024-11-04 14:30".

[0123] Integrity check code: Calculate the integrity check code (SHA256 hash value) of each data block to get the check value:

[0124] Checksum1 = SHA256(data_block_1) = d1a1b1c1...

[0125] For example, the hash value of data block 1 is d1a1b1c1..., and the hash value of data block 2 is e2a2b2c2...

[0126] The final compressed data is stored in a designated storage medium, adopts a block storage strategy, and generates metadata;

[0127] After each data block is written, the storage state (such as available space, storage speed) is updated, and the write pointer position is recorded to ensure that the data is written in order;

[0128] Generate index: including block number, compression rate, timestamp, integrity check code information, for fast retrieval and access;

[0129] Assuming that the data is finally stored in a cloud storage platform, each data block size is 4MB, and the compression rate is 40%, the actual storage space occupied is:

[0130]

[0131] The data is stored by the lossless compression algorithm provided in the above embodiment, which saves about 60% of the storage space, and ensures the integrity and accessibility of the data.

[0132] Preferably, in order to further verify the advantages of the data storage method based on the lossless compression algorithm provided in the embodiment of the application compared with the traditional compression algorithm in terms of data compression rate, storage efficiency, data integrity and storage security, the following experiments are performed in this embodiment:

[0133]

Experimental object

[0134] Traditional method: unable to dynamically select the best encoding method according to the data type and complexity, resulting in low compression efficiency, especially in the processing of audio and video data, the storage process is not intelligent enough to efficiently manage large-scale data, and there is a risk of data loss or tampering during data transmission and storage process;

[0135] The method of the application: by selecting appropriate encoding method (standard Huffman and adaptive Huffman), combined with context modeling optimization, improving the compression rate, through SHA256 hash value generation and metadata management, can ensure the integrity of the data and prevent the data from being tampered with during transmission and storage process, according to the state of the storage medium (such as available space, storage speed, access demand, etc.) to select the appropriate storage location, to ensure the efficiency and scalability of the storage process, and also can dynamically adjust the compression method (such as standard Huffman and adaptive Huffman) according to the complexity of the data, to optimize the compression effect;

[0136]

Experimental environment conditions

[0137] Hardware Environment:

[0138] Processor: Intel i7-9700K 3.6GHz;

[0139] Memory: 32GB RAM;

[0140] Storage: 500GB SSD;

[0141] Network: Gigabit network for data transmission;

[0142] Software Environment:

[0143] Operating System: Windows 10;

[0144] Programming Language: Python 3.8;

[0145] Data Analysis Library: mimetypes, python-magic, pandas;

[0146] Compression Algorithm Library: zlib (for standard Huffman encoding), Python adaptive Huffman library (custom);

[0147] Data Storage Library: Custom storage module (for simulating storage media);

[0148] Experimental Data:

[0149] JSON File: Size 1024KB;

[0150] CSV File: Size 512KB;

[0151] Audio File: Size 2048KB;

[0152] Video File: Size 4096KB;

[0153] Image File: Size 2048KB;

[0154] Text File: Size 256KB;

[0155]

Experimental Process

[0156] Traditional Method (Standard Huffman Encoding):

[0157] Receive each type of data through the data interface, first determine the data format (JSON, CSV, audio);

[0158] Use standard Huffman encoding to compress text data, audio, video and image files;

[0159] After compressing the data, record the compressed data size, compression rate information, and store the data to the analog storage medium;

[0160] The method of the application:

[0161] Receive various types of data through the data interface, and automatically identify the data format (such as JSON, CSV, audio files, video files) using data analysis tools;

[0162] According to the data type (such as text data selecting standard Huffman, audio and video selecting adaptive Huffman), calculate the appropriate encoding mode according to the data complexity;

[0163] Use standard Huffman encoding for JSON files;

[0164] Use adaptive Huffman encoding for audio files;

[0165] Use adaptive Huffman encoding for video files and image files;

[0166] Input the intermediate compressed data into the context modeling function for further optimization, and generate the final compressed data;

[0167] According to the state of the storage medium (available space, storage speed, etc.), the final compressed data is stored in blocks, and metadata (block number, compression rate, timestamp and integrity check code) is generated for each data block;

[0168]

Data analysis

[0169] Referring to Figure 2 , it can be seen that for text data, the compression rate of the method of the application is 0.45, and the compression rate of the traditional method is 0.40, which reduces the storage space by about 12.5%, and occupies less space. For audio data, the compression rate of the method of the application is 0.60, while the compression rate of the traditional method is 0.50, which reduces the storage requirement by about 20%. For image data, the compression rate of the method of the application is 0.55, and the compression rate of the traditional method is 0.45, which reduces the storage requirement by about 22%. For video data, the compression rate of the method of the application is 0.50, and the compression rate of the traditional method is 0.42, which reduces the storage space by about 19%;

[0170] It should be noted that the method of the application has higher efficiency in compressing text data, and is suitable for scenarios that require a large amount of stored text data, such as large-scale document processing or database storage. Audio files are usually large, especially long audio data. The method of the application can effectively reduce the storage space and optimize image storage;

[0171] Referring to Figure 3It can be seen that the storage space occupancy ratio of the method of the present application is 0.50 for text data, 0.45 for audio data, 0.48 for image data, and 0.52 for video data, while the storage space occupancy ratio of the traditional method is 0.60, 0.55, 0.62, and 0.65 respectively, which saves about 16.67% of the storage space for text data, 18.18% of the storage space for audio data, 22.58% of the storage space for image data, and 20% of the storage space for video data.

[0172] It should be further noted that the storage efficiency of audio data is improved by applying the method of the present application, which can store more audio files under the same storage capacity, and at the same time, the storage efficiency is improved, and the storage cost is effectively reduced. Especially for projects that need to store a large amount of image data, the storage efficiency of video data can be effectively improved, and the method is suitable for video monitoring, high-definition video storage, and other video data scenarios that require efficient storage and fast access.

[0173] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limited. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A data storage method based on a lossless compression algorithm, characterized in that, Comprise: Receiving data to be stored through a data interface, automatically identifying the data format and performing feature analysis using a data analysis tool, obtaining the data size, data type and complexity of the data; comprising: Receiving multi-source data to be stored through an HTTP protocol data interface or an FTP protocol data interface, the multi-source data at least containing JSON, CSV, XML, image files, audio files; Using the data analysis tool to automatically identify the data format and output the data type, the data analysis tool is file extension, MIME type, JSON library, CSV library, wherein the MIME type of the data is checked, the data format is identified using the mimetypes library or the python-magic library, for text data, the content is read by parsing, and if the parsing is successful, the type is recorded; Store the received multi-source data as a byte array, calculate the number of bytes in it according to the byte length function, obtain the data size D, and the mathematical expression formula is: Wherein, D represents the size of data, i.e. the number of bytes, data is data, is a byte length function; Statistical data structure characteristics, the complexity of the data is obtained by the ratio of the structure information amount of the data to the data size D, and the calculation formula of the complexity is: Wherein, C represents the complexity, S represents the structure information amount of the data, and D is the data size; According to the data type and the complexity, select the appropriate Huffman coding processing received data to obtain intermediate compressed data; If the output data type is text, select standard Huffman coding; If the output data type is image or audio, select adaptive Huffman coding; The data to be processed is transmitted into the selected Huffman code for encoding processing, and intermediate compressed data is output ; comprising: Traverse the data, count the frequency of each symbol, and establish a symbol frequency table; According to the symbol frequency table, build a Huffman tree and generate a corresponding coding table; encoding the to-be-processed data using the encoding table to obtain intermediate compressed data The mathematical expression formula is: Wherein, the efficiency of Huffman coding is directly related to the symbol frequency, the higher the frequency of the symbol, the shorter the code, the symbol contains characters and bytes, According to the selection of the corresponding Huffman coding processing mode Input the intermediate compressed data into the context modeling function for coding optimization to generate the final compressed data; Store the final compressed data to the specified storage medium, and use the block storage strategy to divide the final compressed data into fixed size data blocks, generate metadata for each data block, and then update the state to record the storage information of all data; Wherein, the metadata at least contains block number, compression rate, timestamp and integrity check code.

2. The data storage method based on lossless compression algorithm according to claim 1, characterized in that, The structure information amount of the data at least contains the number of key-value pairs or the number of features in the data.

3. The data storage method based on lossless compression algorithm according to claim 1, characterized in that, The intermediate compressed data is input into a context modeling function, the frequency of symbol occurrences is counted, a dynamically updated frequency table is maintained, the context model is updated based on the frequency table, a plus one smoothing technique is used to avoid zero frequency problems, the frequency distribution is adjusted using maximum likelihood estimation, dynamic adjustments are made at each encoding, and encoding optimizations are performed, including: where O denotes the optimized compression ratio, is the probability of the symbol i appearing; improve coding efficiency, reduce redundancy, and generate final compressed data according to the results of the context modeling by updating the probability distribution : wherein is the final compressed data, () is a context modeling function.

4. The data storage method based on lossless compression algorithm according to claim 3, characterized in that, storing the final compressed data to a designated storage medium, the designated storage medium is obtained by API call medium state, according to the medium state and threshold comparison results selected, wherein the medium state at least contains available space, storage speed and access requirements, when the threshold is less than the available space in the medium state, then select the final compressed data write to the selected available space; storing the final compressed data into fixed size data blocks; Generate metadata for each data block, the metadata at least contains block number, compression rate, timestamp and integrity check code; Each time a data block is written, the write pointer is updated to ensure sequential writing; Update the state to record the storage information of all data; Establish an index to quickly retrieve and access the stored data.

5. The data storage method based on lossless compression algorithm according to claim 4, characterized in that, Generate corresponding metadata at each data block division, and calculate the SHA256 hash value of each data block to generate the integrity check code, check the data block integrity and record the check value, and the mathematical expression formula is: wherein is a checksum, a fixed-length 256-bit output value computed from the SHA256 hash function, i.e. a check value, is the data block to be checked; Record the hash value of each data block as part of the metadata to ensure that the data is not tampered with during transmission and storage.

Images