Data storage method based on lossless compression algorithm

Abstract

The invention relates to the technical field of data storage, and discloses a lossless compression algorithm-based data storage method, which comprises the following steps of: receiving to-be-stored data through a data interface, automatically identifying a data format by using a data analysis tool, and performing feature analysis to obtain the size, the type and the complexity of the data; through deep analysis of data features, the method can flexibly select a proper Huffman coding processing mode, and optimizes a compression process according to different data types and complexity. In addition, a context modeling optimization mechanism is introduced, the coding mode is dynamically adjusted, and the compression ratio and the storage efficiency are further improved; efficient storage and reliability verification of data are ensured through a block storage strategy in combination with metadata management; the problem that an existing compression algorithm is low in efficiency is solved, the integrity and reliability of data storage are improved, and the method is suitable for storage requirements of large-scale data and diversified data.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data storage method based on a lossless compression algorithm. Background Art

[0002] With the continuous advancement of information technology and the popularization of Internet applications, the generation and storage demand of data has exploded. In traditional data storage systems, data is often stored in its original form, resulting in a waste of storage space and low access efficiency. Therefore, how to effectively compress data to reduce storage space has become a key issue in the field of information technology.

[0003] In recent years, lossless data compression technology has gradually become one of the core technologies to solve this problem. Lossless compression technology can minimize data storage requirements through algorithms while ensuring data integrity and recoverability. Although lossless compression technology has solved the storage space problem to a certain extent, existing compression methods often have problems such as insufficient efficiency, slow processing speed, and unsatisfactory compression ratio when facing large-scale data, high-complexity data, and diverse data formats. Existing technologies mostly rely on a single compression algorithm and cannot flexibly select the optimal compression strategy according to data characteristics, resulting in limited compression effect.

[0004] Current lossless compression technology usually adopts static or fixed encoding methods. This method fails to give full play to its advantages when facing different types of data. For example, although standard Huffman coding performs well when compressing static text, its compression efficiency and speed are often unsatisfactory when facing complex structured data or multimedia data. In addition, the context modeling optimization methods in the existing technology are often limited to predefined models and cannot dynamically adjust and adapt to changes in different data types. Therefore, the effect of the optimization process is limited. More importantly, the existing storage methods generally ignore the metadata management during data storage, especially the recording of key information such as data integrity and compression rate, which is particularly important in distributed storage and big data application scenarios. Therefore, the existing technology has certain shortcomings when facing data diversity, complexity and storage reliability requirements. Summary of the invention

[0005] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract of the specification and the title of the invention of this application to avoid blurring the purpose of this section, the abstract of the specification and the title of the invention, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0006] In view of the above existing problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is that the existing lossless data compression technology generally faces problems in compression efficiency, storage reliability and diversified data processing.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: receiving data to be stored through a data interface, automatically identifying the data format and performing feature analysis using a data analysis tool, and obtaining the data size, data type and complexity of the data;

[0009] Selecting appropriate Huffman coding to process the received data according to the data type and the complexity to obtain intermediate compressed data;

[0010] Inputting the intermediate compressed data into a context modeling function for encoding optimization to generate final compressed data;

[0011] The final compressed data is stored in a designated storage medium, and the final compressed data is divided into data blocks of fixed size using a block storage strategy, metadata is generated for each data block, and then the status is updated to record the storage information of all data;

[0012] The metadata at least includes a block number, a compression ratio, a timestamp and an integrity check code.

[0013] As a preferred solution of the data storage method based on the lossless compression algorithm of the present invention, obtaining the data size, data type and complexity of the data includes:

[0014] Receiving multi-source data to be stored through an HTTP protocol data interface or an FTP protocol data interface, wherein the multi-source data at least includes JSON, CSV, XML, image files, and audio files;

[0015] The data analysis tool is used to automatically identify the data format and output the data type. The data analysis tool is a file extension, MIME type, JSON library, and CSV library. The MIME type of the data is checked, and the data format is identified using the mimetypes library or the python-magic library. For text data, the content is read by parsing. If the parsing is successful, the type is recorded.

[0016] The received multi-source data is stored as a byte array, and the number of bytes therein is calculated according to the byte length function to obtain the data size D, which is mathematically expressed as follows:

[0017] D=len(data)

[0018] Where D represents the size of the data, i.e. the number of bytes, data is the data, and len() is the byte length function;

[0019] The structural characteristics of the statistical data are obtained by the ratio of the structural information amount of the data to the data size D, and the complexity of the data is calculated as follows:

[0020]

[0021] Among them, C represents complexity, S represents the amount of structural information of the data, and D is the data size.

[0022] As a preferred solution of the data storage method based on the lossless compression algorithm described in the present invention, the structural information amount of the data at least includes the number of key-value pairs or the number of features in the data.

[0023] As a preferred solution of the data storage method based on the lossless compression algorithm described in the present invention, if the output data type is text, standard Huffman coding is selected;

[0024] If the output data type is image or audio, adaptive Huffman coding is selected;

[0025] Then the data to be processed is passed into the selected Huffman code for encoding processing, and the intermediate compressed data D is output. H .

[0026] As a preferred solution of the data storage method based on the lossless compression algorithm of the present invention, outputting the intermediate compressed data D H ,include:

[0027] Traverse the data, count the frequency of occurrence of each symbol, and establish a symbol frequency table;

[0028] Constructing a Huffman tree according to the symbol frequency table and generating a corresponding coding table;

[0029] The data to be processed is encoded using the encoding table to obtain intermediate compressed data D H , the mathematical expression formula is:

[0030] D H =Huffman_encode(data)

[0031] Among them, the efficiency of Huffman coding is directly related to the symbol frequency. The higher the frequency of the symbol, the shorter the code. The symbol includes characters and bytes. Huffman_encode is determined by selecting the corresponding Huffman coding processing method.

[0032] As a preferred solution of the data storage method based on the lossless compression algorithm of the present invention, the intermediate compressed data D HInput into the context modeling function, count the frequency of occurrence of symbols, maintain a dynamically updated frequency table, update the context model according to the frequency table, use the plus 1 smoothing technology to avoid the zero frequency problem, use the maximum likelihood estimation to adjust the frequency distribution, dynamically adjust each time encoding, and perform encoding optimization, including:

[0033]

[0034] Where O represents the optimized compression ratio, and p(i) is the probability of the occurrence of symbol i;

[0035] By updating the probability distribution to improve coding efficiency and reduce redundancy, the final compressed data D is generated based on the results of context modeling. F :

[0036] D F =Context_Modeling(D H )

[0037] Among them, D F To finally compress the data, Context_Modeling() is the context modeling function.

[0038] As a preferred solution of the data storage method based on the lossless compression algorithm described in the present invention, the final compressed data D F The specified storage medium is selected based on the comparison result between the medium state and the threshold value obtained by calling the API, wherein the medium state at least includes available space, storage speed and access requirements, and when the threshold value is less than the available space in the medium state, the final compressed data D is selected. F writing to the selected available space;

[0039] The final compressed data D to be stored F Divide into fixed-size data blocks;

[0040] Generate metadata for each data block, the metadata including at least a block number, a compression ratio, a timestamp, and an integrity check code;

[0041] Each time a data block is written, the write pointer is updated to ensure sequential writing;

[0042] Perform status updates to record storage information of all data;

[0043] Create indexes for quick retrieval and access to stored data.

[0044] As a preferred solution of the data storage method based on the lossless compression algorithm described in the present invention, corresponding metadata is generated each time the data block is divided, and the SHA256 hash value is calculated for each data block to generate an integrity check code, check the integrity of the data block and record the check value. The mathematical expression formula is:

[0045] Checksum = SHA256 (data_block)

[0046] Among them, Checksum is the checksum, which is a fixed-length 256-bit output value calculated by the SHA256 hash function, that is, the check value, and data_block is the data block to be checked;

[0047] The hash value of each data block is recorded as part of the metadata to ensure that the data has not been tampered with during transmission and storage.

[0048] Beneficial effects of the present invention:

[0049] 1. Through feature analysis, the most appropriate processing method can be accurately selected according to different types of data (such as text, images, audio) and the complexity of the data (such as structured, unstructured). The extraction of data size can provide a quantitative basis for subsequent storage and compression decisions. The identification of data types and complexity ensures the precise matching of subsequent compression methods. By accurately identifying data features, the efficiency and accuracy of data processing are improved, providing a solid foundation for subsequent optimized compression and storage, and reducing the possibility of misjudgment or inefficient compression.

[0050] 2. Ability to select the most appropriate encoding method for different data types (such as text, binary files, images) and complexities (such as simple or complex data structures). Selecting the most appropriate encoding method can effectively avoid the problems of low compression ratio and slow compression speed caused by improper encoding methods. Especially when facing complex structures or changing data, it can be adjusted dynamically and intelligently to achieve better compression effects and improve data storage efficiency and resource utilization;

[0051] 3. The context modeling function dynamically analyzes the symbol distribution and contextual relationship in the data, and can adjust the encoding strategy in real time according to the historical characteristics and occurrence probability of the data, thereby optimizing the compression efficiency. Through this optimization, data redundancy is further reduced, and the compression ratio is improved. At the same time, the compressed data can maintain high data recovery while maintaining efficient compression, and adapt to a wider range of data types and application scenarios;

[0052] 4. The introduction of block storage strategy improves the flexibility and reliability of data storage. Block storage can effectively reduce the impact of damage or loss of a single data block on the entire data set and improve the fault tolerance of the system. The independence of each data block makes data storage, transmission and recovery more efficient. By generating metadata for each data block (such as block number, compression rate, timestamp and integrity check code), it can provide accurate storage tracking and management to ensure data integrity and reliability. The recording of metadata makes it possible to accurately track the status of each data block during data recovery, quickly locate and repair damaged or missing parts, thereby ensuring high availability and reliability of the system. In a distributed storage environment, the combination of block and metadata management can improve the performance and fault tolerance of the storage system and optimize the allocation and use of storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0054] Figure 1 It is a schematic diagram of the flow of the data storage method based on the lossless compression algorithm shown in the present invention;

[0055] Figure 2 It is a schematic diagram of compression ratio comparison shown in the present invention;

[0056] Figure 3 It is a schematic diagram of storage efficiency comparison shown in the present invention. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0058] Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without making any creative work should fall within the scope of protection of the present invention.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0060] In traditional storage solutions, data types and processing methods are usually determined by manual or preset parameters, which have errors and limitations of human intervention and are not suitable for diverse and dynamically changing data types. In the prior art, Huffman coding is usually static, that is, it is encoded based on fixed rules and cannot be flexibly adjusted according to changes and characteristics of the data. In traditional compression algorithms, the encoding process is usually based on static or preset probability models.

[0061] The embodiment of the present invention provides a data storage method based on a lossless compression algorithm. By analyzing data characteristics (such as type and complexity), the most suitable Huffman coding variant (such as standard Huffman coding, dynamic Huffman coding or adaptive Huffman coding) can be automatically selected, thereby improving the compression efficiency and the adaptability of the algorithm; the adaptive context modeling adopted can automatically adjust according to the real-time changes of the data stream, thereby improving the adaptability and flexibility of the compression.

[0062] According to an embodiment of the present invention, Figure 1 The flowchart shown is a data storage method based on a lossless compression algorithm, which specifically includes the following steps:

[0063] S1. Receive the data to be stored through the data interface, use the data analysis tool to automatically identify the data format and perform feature analysis to obtain the data size, data type and complexity;

[0064] S2, select appropriate Huffman coding to process the received data according to the data type and complexity to obtain intermediate compressed data;

[0065] S3, inputting the intermediate compressed data into the context modeling function for encoding optimization to generate final compressed data;

[0066] S4. Store the final compressed data in a designated storage medium, and use a block storage strategy to divide the final compressed data into data blocks of fixed size, generate metadata for each data block, and then update the status to record the storage information of all data;

[0067] The metadata at least includes a block number, a compression ratio, a timestamp, and an integrity check code.

[0068] The following describes in more detail the implementation process and / or effects of certain embodiments of the present invention in conjunction with some preferred or optional examples of the present invention.

[0069] For example, in a cloud storage platform, data storage is a crucial link. As a large number of multimedia files (such as images, audio, video) and other formats (such as text, log files, database export files) are frequently uploaded and downloaded, how to efficiently store these huge and diverse data while ensuring the integrity, validity and fast access of the data has become an urgent problem that cloud storage service providers need to solve.

[0070] In this embodiment, a data storage method based on a lossless compression algorithm is proposed to improve storage efficiency, reduce storage space consumption, and provide integrity protection for stored data. In particular, for uploaded data of various types (such as images, audio, and text), the method of the present invention can select a suitable encoding compression method according to the type and structural characteristics of the data, and achieve efficient storage and access through further optimization processing (such as context modeling optimization, block storage, and metadata generation).

[0071] As an example, the cloud storage platform is set to provide cloud storage services for a large social media company, and customers upload images, audio and text data through the platform; the types of data to be stored are:

[0072] Image file (JPEG format): 4MB;

[0073] Audio file (MP3 format): 10MB;

[0074] Text files (CSV and JSON format): 1MB (including user data and configuration information);

[0075] Storage medium: Cloud storage's distributed storage system has high availability and elastic expansion capabilities. Each node has a storage space of 50GB. The system supports high-speed storage and low-latency access.

[0076] The system receives these various types of data through an interface using the HTTP protocol, receiving an image file (4MB, JPEG format): Through the data analysis tool, the system uses the MIME type (image / jpeg) and file extension (.jpg) to identify the file as an image file, and the system records the data size as 4MB. The data is considered complex data, containing rich pixel information and compression characteristics;

[0077] Receive audio files (10MB, MP3 format): Use the MIME type (audio / mpeg) and file extension (.mp3) to identify audio files. The data size is 10MB. The audio data is highly complex and contains multi-band sound waveform data, which has a certain degree of redundancy.

[0078] Receive text files (CSV and JSON formats, 1MB in total): The system automatically recognizes them as text files by parsing the file formats (CSV library, JSON library). The data size is 1MB and the structure is relatively simple (stored in the form of key-value pairs).

[0079] Feature analysis:

[0080] Data size (D): D 图像 =4MB, D 音频 =10MB, D 文本 =1MB;

[0081] Data type: JPEG for images, MP3 for audio, CSV / JSON for text;

[0082] Complexity (C): Calculate the data complexity, the formula is:

[0083]

[0084] Among them, S is the structural information of the data, and D is the data size;

[0085] Image file (JPEG): Image files usually contain a lot of redundant data. Assuming that the amount of structural information S is 2MB, its complexity is:

[0086]

[0087] Audio file (MP3): Audio files have frequency band redundancy. Assuming that the amount of structural information S is 4MB, its complexity is:

[0088]

[0089] Text file (CSV / JSON): The text data structure is simple. Assuming that the amount of structural information S is 0.3MB, its complexity is:

[0090]

[0091] Based on the above analysis, the system selects the corresponding Huffman encoding method:

[0092] Image files (JPEG): Adaptive Huffman coding is selected, and the image data has high redundancy (for example, adjacent pixels are usually similar);

[0093] Audio file (MP3): Select adaptive Huffman coding. Similar frequency bands and samples in audio data will increase redundancy, so adaptive Huffman coding is suitable.

[0094] Text files (CSV / JSON): select standard Huffman encoding because the redundancy of text data is relatively low;

[0095] Huffman coding implementation process: For each data type, the Huffman tree is established by counting the frequency of symbol occurrence and generating the corresponding coding table;

[0096] Image (JPEG): Assume that the frequency of occurrence of each pixel in an image is:

[0097] f 像素1 =0.4,f 像素2 =0.3,f 像素3 =0.2,f 像素4 =0.1

[0098] Build a Huffman tree by frequency and generate a coding table;

[0099] Audio (MP3): Assume the frequency of the frequency band in the audio signal is:

[0100] f 频段1 =0.5,f 频段2 =0.2,f 频段3 =0.15,f 频段4 =0.15

[0101] Build a Huffman tree by frequency and generate a coding table;

[0102] Text (CSV / JSON): Assume the frequency of symbols in the text data is:

[0103] f 字符1 =0.3,f 字符2 =0.2,f 字符3 =0.25,f 字符4 =0.25

[0104] Use standard Huffman coding for processing;

[0105] For a symbol i, assume its frequency is f i , the Huffman code length L can be expressed as:

[0106]

[0107] For example, if a symbol frequency is 0.4, its code length is:

[0108]

[0109] Input the intermediate compressed data of each file into the context modeling function for optimization;

[0110] Audio file optimization: Each time an audio file is encoded, the frequency table is updated, plus 1 smoothing technology is applied, and the symbol probability distribution is adjusted using maximum likelihood estimation (MLE) to avoid zero frequency problems;

[0111] For example, in an audio signal, the probability of a certain frequency component is P f =0.3, and the probability distribution of this frequency may be affected by outdated data. After adjustment through maximum likelihood estimation, the frequency distribution is updated, and the updated probability distribution is:

[0112]

[0113] Image file optimization: optimize the image encoding process and further adjust the correlation of pixel blocks and encoding tables through context modeling;

[0114] Context modeling is used to model the correlation of each pixel block, adjust the frequency table, and reduce redundancy by dynamically adjusting the context model to further compress the data;

[0115] Optimization formula: Assume that the original probability of a symbol i is P i , the probability after context modeling optimization is P ′ i , then the optimized compression ratio O can be expressed as:

[0116]

[0117] The final compressed data is stored in the specified storage medium, and a block storage strategy is adopted;

[0118] Block storage: For example, if the final compressed data size is 15MB, it is divided into 4 data blocks, each of which is 4MB, and the last data block will be slightly smaller than 4MB (such as 3MB);

[0119] The metadata for each data block includes:

[0120] Block number: 1, 2, 3, 4;

[0121] Compression rate: For each data block, calculate its compression rate, for example, the compression rate is 40%;

[0122] Timestamp: records the timestamp of each data block, for example, the timestamp is "2024-11-04 14:30";

[0123] Integrity check code: Calculate the integrity check code (SHA256 hash value) of each data block and get the check value:

[0124] Checksum1=SHA256(data_block_1)=d1a1b1c1...

[0125] For example, the hash value of data block 1 is d1a1b1c1..., and the hash value of data block 2 is e2a2b2c2...;

[0126] Finally, the compressed data is stored in the specified storage medium, using a block storage strategy, and metadata is generated;

[0127] After each data block is written, the storage status (such as available space, storage speed) is updated and the write pointer position is recorded to ensure that the data is written in order;

[0128] Generate index: including block number, compression ratio, timestamp, and integrity check code information for fast retrieval and access;

[0129] Assuming that the data is ultimately stored in a cloud storage platform, each data block is 4MB in size, and the compression rate is 40%, the actual storage space occupied is:

[0130]

[0131] The data is stored using the lossless compression algorithm provided by the above embodiment, which saves about 60% of the storage space and ensures the integrity and accessibility of the data.

[0132] Preferably, in order to further verify the advantages of a data storage method based on a lossless compression algorithm proposed in an embodiment of the present invention over a traditional compression algorithm in terms of data compression rate, storage efficiency, data integrity and storage security, this embodiment conducts the following experiments:

[0133]

Experimental subjects

[0134] Traditional methods: Unable to dynamically select the best encoding method based on data type and complexity, resulting in low compression efficiency, especially in the processing of audio and video data. The storage process is not intelligent enough and cannot efficiently manage large-scale data. In addition, there is a risk of data loss or tampering during data transmission and storage.

[0135] The method of the present invention improves the compression rate by selecting a suitable encoding method (standard Huffman and adaptive Huffman) and combining context modeling optimization. It can ensure the integrity of data and prevent data from being tampered with during transmission and storage through SHA256 hash value generation and metadata management. It can select a suitable storage location according to the state of the storage medium (such as available space, storage speed, access requirements, etc.) to ensure the efficiency and scalability of the storage process. It can also dynamically adjust the compression method (such as standard Huffman and adaptive Huffman) according to the complexity of the data to optimize the compression effect.

[0136]

Experimental environment conditions

[0137] Hardware environment:

[0138] Processor: Intel i7-9700K 3.6GHz;

[0139] Memory: 32GB RAM;

[0140] Storage: 500GB SSD;

[0141] Network: Gigabit network for data transmission;

[0142] Software environment:

[0143] Operating system: Windows 10;

[0144] Programming language: Python 3.8;

[0145] Data analysis libraries: mimetypes, python-magic, pandas;

[0146] Compression algorithm library: zlib (for standard Huffman encoding), Python adaptive Huffman library (customized);

[0147] Data repository: custom storage module (used to simulate storage media);

[0148] Experimental data:

[0149] JSON file: size is 1024KB;

[0150] CSV file: size is 512KB;

[0151] Audio file: size is 2048KB;

[0152] Video file: size is 4096KB;

[0153] Image file: size is 2048KB;

[0154] Text file: size is 256KB;

[0155]

Experimental process

[0156] Traditional method (standard Huffman coding):

[0157] Receive various types of data through the data interface, first determine the data format (JSON, CSV, audio);

[0158] Use standard Huffman coding to compress text data, audio, video and image files;

[0159] After compressing the data, the size and compression rate of the compressed data are recorded, and the data is stored in the analog storage medium;

[0160] The method of the present invention:

[0161] Receive various types of data through data interfaces and use data analysis tools to automatically identify data formats (such as JSON, CSV, audio files, video files);

[0162] Choose the appropriate encoding method based on the data type (e.g., standard Huffman for text data, adaptive Huffman for audio and video) and the data complexity calculation;

[0163] Use standard Huffman encoding for JSON files;

[0164] Use adaptive Huffman coding for audio files;

[0165] Use adaptive Huffman coding for video files and image files;

[0166] Input the intermediate compressed data into the context modeling function for further optimization and generate the final compressed data;

[0167] According to the status of the storage medium (available space, storage speed, etc.), the final compressed data is stored in blocks, and metadata (block number, compression ratio, timestamp and integrity check code) is generated for each data block;

[0168]

Data Analysis

[0169] Reference Figure 2 It can be seen intuitively that for text data, the compression rate of the method of the present invention is 0.45, while that of the traditional method is 0.40, which reduces the storage space by about 12.5% ​​and occupies less space. For audio data, the compression rate of the method of the present invention is 0.60, while that of the traditional method is 0.50, which reduces the storage requirement by about 20%. For image data, the compression rate of the method of the present invention is 0.55, while that of the traditional method is 0.45, which reduces the storage requirement by about 22%. For video data, the compression rate of the method of the present invention is 0.50, while that of the traditional method is 0.42, which reduces the storage space by about 19%.

[0170] It should be noted that the method of the present invention has a higher efficiency in compressing text data and is suitable for scenarios that require a large amount of text data to be stored, such as large-scale document processing or database storage. Audio files are usually large, especially long-term audio data. The method of the present invention can effectively reduce storage space and effectively optimize image storage;

[0171] Reference Figure 3It can be seen intuitively that, for text data, the storage space occupied ratio of the method of the present invention is 0.50, and the traditional method is 0.60, saving about 16.67% of the storage space. For audio data, the storage space occupied ratio of the method of the present invention is 0.45, and the traditional method is 0.55, saving about 18.18% of the storage space. For image data, the storage space occupied ratio of the method of the present invention is 0.48, and the traditional method is 0.62, saving about 22.58% of the storage space. For video data, the storage space occupied ratio of the method of the present invention is 0.52, and the traditional method is 0.65, saving about 20% of the storage space.

[0172] It should be further explained that, by applying the method of the present invention, the storage efficiency of audio data is improved, and more audio files can be stored under the same storage capacity. While improving the storage efficiency, the storage cost is also effectively reduced. Especially for projects that require a large amount of image storage, the storage efficiency of video data can be effectively improved. It is suitable for video surveillance and high-definition video storage that require efficient storage and fast access to video data scenarios.

[0173] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data storage method based on a lossless compression algorithm, characterized in that: include: Receive data to be stored through a data interface, automatically identify the data format and perform feature analysis using a data analysis tool to obtain the data size, data type and complexity of the data; Selecting appropriate Huffman coding to process the received data according to the data type and the complexity to obtain intermediate compressed data; Inputting the intermediate compressed data into a context modeling function for encoding optimization to generate final compressed data; The final compressed data is stored in a designated storage medium, and the final compressed data is divided into data blocks of fixed size using a block storage strategy, metadata is generated for each data block, and then the status is updated to record the storage information of all data; The metadata at least includes a block number, a compression ratio, a timestamp and an integrity check code.

2. The data storage method based on lossless compression algorithm according to claim 1, characterized in that: The data size, data type and complexity of the data are obtained, including: Receiving multi-source data to be stored through an HTTP protocol data interface or an FTP protocol data interface, wherein the multi-source data at least includes JSON, CSV, XML, image files, and audio files; The data analysis tool is used to automatically identify the data format and output the data type. The data analysis tool is a file extension, MIME type, JSON library, and CSV library. The MIME type of the data is checked, and the data format is identified using the mimetypes library or the python-magic library. For text data, the content is read by parsing. If the parsing is successful, the type is recorded. The received multi-source data is stored as a byte array, and the number of bytes therein is calculated according to the byte length function to obtain the data size D, which is mathematically expressed as follows: D=len(data) Wherein, D represents the size of the data, that is, the number of bytes, data is the data, and len() is the byte length function; the structural characteristics of the statistical data are obtained by the ratio of the structural information amount of the data to the data size D, and the complexity of the data is calculated as follows: Among them, C represents complexity, S represents the amount of structural information of the data, and D is the data size.

3. The data storage method based on lossless compression algorithm according to claim 2, characterized in that: The amount of structural information of the data at least includes the number of key-value pairs or the number of features in the data.

4. The data storage method based on lossless compression algorithm according to claim 1 or 2, characterized in that: If the data type of the output is text, standard Huffman coding is selected; If the output data type is image or audio, adaptive Huffman coding is selected; Then the data to be processed is passed into the selected Huffman code for encoding processing, and the intermediate compressed data D is output. H .

5. The data storage method based on lossless compression algorithm according to claim 4, characterized in that: Output the intermediate compressed data D H ,include: Traverse the data, count the frequency of occurrence of each symbol, and establish a symbol frequency table; Constructing a Huffman tree according to the symbol frequency table and generating a corresponding coding table; The data to be processed is encoded using the encoding table to obtain intermediate compressed data D H , the mathematical expression formula is: D H =Huffman_encode(data) Among them, the efficiency of Huffman coding is directly related to the symbol frequency. The higher the frequency of the symbol, the shorter the code. The symbol includes characters and bytes. Huffman_encode is determined by selecting the corresponding Huffman coding processing method.

6. The data storage method based on lossless compression algorithm according to claim 5, characterized in that: The intermediate compressed data D H Input into the context modeling function, count the frequency of occurrence of symbols, maintain a dynamically updated frequency table, update the context model according to the frequency table, use the plus 1 smoothing technology to avoid the zero frequency problem, use the maximum likelihood estimation to adjust the frequency distribution, dynamically adjust each time encoding, and perform encoding optimization, including: Where O represents the optimized compression ratio, and p(i) is the probability of the occurrence of symbol i; By updating the probability distribution to improve coding efficiency and reduce redundancy, the final compressed data D is generated based on the results of context modeling. F : D F =Context_Modeling(D H ) Among them, D F To finally compress the data, Context_Modeling() is the context modeling function.

7. The data storage method based on lossless compression algorithm according to claim 6, characterized in that: The final compressed data D F The specified storage medium is selected based on the comparison result between the medium state and the threshold value obtained by calling the API, wherein the medium state at least includes available space, storage speed and access requirements, and when the threshold value is less than the available space in the medium state, the final compressed data D is selected. F writing to the selected available space; The final compressed data D to be stored F Divide into fixed-size data blocks; Generate metadata for each data block, the metadata including at least a block number, a compression ratio, a timestamp, and an integrity check code; Each time a data block is written, the write pointer is updated to ensure sequential writing; Perform status updates to record storage information of all data; Create indexes for quick retrieval and access to stored data.

8. The data storage method based on lossless compression algorithm according to claim 7, characterized in that: Each time a data block is divided, the corresponding metadata is generated, and the SHA256 hash value is calculated for each data block to generate an integrity check code, verify the integrity of the data block and record the check value. The mathematical expression formula is: Checksum = SHA256 (data_block) Among them, Checksum is the checksum, which is a fixed-length 256-bit output value calculated by the SHA256 hash function, that is, the check value, and data_block is the data block to be checked; The hash value of each data block is recorded as part of the metadata to ensure that the data has not been tampered with during transmission and storage.

Images