Data optimization method and system based on adaptive multi-stage compression

Abstract

The invention provides a data optimization method and system based on self-adaptive multi-stage compression, and relates to the technical field of data processing, and the method comprises the steps: obtaining to-be-compressed data; preprocessing the to-be-compressed data to obtain preprocessed data; performing primary compression on the preprocessed data to obtain primary compressed data; according to the access frequency and the storage requirement of the first-level compressed data, judging whether the first-level compressed data needs to be subjected to second-level compression or not; if yes, entering the next step; otherwise, storing the first-level compressed data into a rapid storage medium; and carrying out secondary compression on the primary compressed data by adopting a progressive storage partitioning technology to obtain secondary compressed data, and storing the secondary compressed data into a low-speed storage medium. The adaptive multi-stage compression technology is adopted to ensure that a lightweight compression strategy is used for frequently accessed data, so that additional processing overhead for high-frequency accessed data is avoided, and deeper two-stage compression is performed on low-frequency accessed data to save storage space to the greatest extent.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data optimization method and system based on adaptive multi-level compression. Background Technology

[0002] The significance of data compression lies in improving storage efficiency and processing speed by reducing data storage space and transmission bandwidth requirements. It effectively saves storage resources, reduces storage and transmission costs, and accelerates data transmission. Especially when network bandwidth is limited or data volumes are large, compression technology optimizes data storage and access efficiency, ensuring high system performance. By precisely controlling data compression and decompression, compression technology can also improve overall system performance without losing critical information.

[0003] There are many commonly used data compression technologies, such as Huffman compression, LZ78 compression, JPEG compression, and MP3 compression. However, in current data transmission processes, often only one compression technology is selected for data compression, employing a uniform compression strategy. This fails to automatically adapt to different storage needs or access patterns based on data changes, potentially leading to poor compression performance. For example, using a uniform compression strategy for frequently accessed and infrequently accessed data may slow down access speeds for frequently accessed data, as the compressed data may require frequent decompression operations; while for infrequently accessed data, insufficient compression may occur, resulting in wasted storage space. Summary of the Invention

[0004] To address the technical problem that existing technologies employ a uniform compression strategy, which cannot automatically adapt to different storage needs or access patterns based on data changes, potentially leading to poor compression performance, this invention provides a data optimization method and system based on adaptive multi-level compression.

[0005] The technical solutions provided by the embodiments of the present invention are as follows: First aspect This invention provides a data optimization method based on adaptive multi-level compression, comprising: S1: Obtain the data to be compressed; S2: Preprocess the data to be compressed to obtain preprocessed data; S3: Perform first-level compression on the preprocessed data to obtain first-level compressed data; S4: Based on the access frequency and storage requirements of the first-level compressed data, determine whether the first-level compressed data needs to be compressed in the second level; if yes, proceed to S5; otherwise, store the first-level compressed data in a fast storage medium. S5: Using progressive storage partitioning technology, the first-level compressed data is compressed a second time to obtain second-level compressed data, and the second-level compressed data is stored in a slow storage medium.

[0006] Second aspect

[0007] This invention provides a data optimization system based on adaptive multi-level compression, comprising: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the data optimization method based on adaptive multi-level compression as described in the first aspect.

[0008] Third aspect

[0009] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data optimization method based on adaptive multi-level compression as described in the first aspect.

[0010] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) In this embodiment of the invention, an adaptive multi-level compression technology is adopted. By dynamically determining whether to perform secondary compression based on the access frequency and storage requirements of the primary compressed data, a lightweight compression strategy is used for frequently accessed data, thereby avoiding the extra processing overhead of high-frequency accessed data. For low-frequency accessed data, a deeper secondary compression is performed to save storage space to the maximum extent.

[0011] (2) In this embodiment of the invention, different storage media are used to store the compressed data. According to the access frequency and importance of the data, data with different compression levels are stored in fast storage media and slow storage media respectively. High-frequency access data is stored in fast storage media. The fast storage media is used to store frequently accessed data, which ensures fast access and efficient read and write performance of the data. Low-frequency access data is stored in slow storage media, which improves the utilization efficiency of storage resources. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a data optimization method based on adaptive multi-level compression provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a data optimization system based on adaptive multi-level compression, provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0015] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0016] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0017] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0018] Reference manual attached Figure 1 The diagram shows a flowchart of a data optimization method based on adaptive multi-level compression provided by an embodiment of the present invention.

[0019] This invention provides a data optimization method based on adaptive multi-level compression. This method can be implemented by a data optimization device based on adaptive multi-level compression, which can be a terminal or a server. The processing flow of the data optimization method based on adaptive multi-level compression may include the following steps: S1: Obtain the data to be compressed.

[0020] Specifically, the data to be compressed can be text data, image data, audio data, video data, etc. This invention does not limit the specific type of data to be compressed.

[0021] S2: Preprocess the data to be compressed to obtain preprocessed data.

[0022] Specifically, preprocessing methods that can be used include noise reduction, missing value imputation, deduplication, and normalization.

[0023] S3: Perform first-level compression on the preprocessed data to obtain first-level compressed data.

[0024] Optionally, during primary compression, some existing lossless or lossy compression methods can be selected according to actual needs.

[0025] In one possible implementation, S3 specifically includes: using Huffman compression technology, LZ78 compression technology, DEFLATE compression technology, BWT compression technology and / or LZMA compression technology to perform primary compression on the preprocessed data to obtain primary compressed data.

[0026] Among them, Huffman compression technology, LZ78 compression technology, DEFLATE compression technology, BWT compression technology, and LZMA compression technology are already very mature existing technologies, and will not be described in detail in this invention.

[0027] S4: Based on the access frequency and storage requirements of the first-level compressed data, determine whether the first-level compressed data needs to be compressed a second time. If yes, proceed to S5. Otherwise, store the first-level compressed data in a high-speed storage medium.

[0028] Optionally, the high-speed storage media include: solid-state drives, non-volatile memory, and / or dynamic random access memory.

[0029] In one possible implementation, S4 specifically includes sub-steps S401 and S402: S401: Calculate the compression level decision index based on the access frequency and storage requirements of the Level 1 compressed data:

[0030] in, ρ This indicates the compression level decision index. f Indicates access frequency. η Weighting coefficients representing access frequency. C Indicates storage cost, H Represents information entropy. t Indicates time, This represents the partial derivative operation. γ This represents the sensitivity coefficient to entropy changes.

[0031] Those skilled in the art can set the weighting coefficient of access frequency and the magnitude of the entropy change sensitivity coefficient according to the actual situation; this invention does not impose any limitations.

[0032] In this embodiment of the invention, considering changes in data access frequency, storage cost, and information entropy, the data compression strategy can be adjusted more intelligently. If data access is frequent, the compression depth can be reduced to improve access speed; if data storage cost is high or information entropy changes significantly, deeper compression is performed to save storage space. This allows for flexible adjustment of the compression level based on real-time storage needs and data access patterns, thereby ensuring storage efficiency without affecting data access performance and optimizing the utilization of system storage and computing resources.

[0033] S402: Determine if the compression level decision index of the Level 1 compressed data is less than the preset index value. If yes, determine that the Level 1 compressed data needs compression and proceed to S5. Otherwise, determine that the Level 1 compressed data does not need compression and store the Level 1 compressed data in the fast storage medium.

[0034] Those skilled in the art can set the value of the preset index according to the actual situation, and the present invention does not limit it.

[0035] In this embodiment of the invention, by determining whether the compression level decision index of the first-level compressed data is less than a preset index value, the system can intelligently decide whether to perform further compression, thereby optimizing storage space utilization, improving data access performance, reducing computational overhead, and lowering storage costs. This method not only improves the utilization efficiency of storage resources but also ensures that performance will not degrade due to additional compression when processing frequently accessed data, thus achieving more flexible and efficient data storage and access management.

[0036] S5: Employs progressive storage partitioning technology to perform secondary compression on the primary compressed data, resulting in secondary compressed data, which is then stored in a slow storage medium.

[0037] In one possible implementation, S5 specifically includes sub-steps S501 to S505: S501: Regarding the current time t Perform an SVD decomposition dimensionality assessment on the first-level compressed data to determine the estimated rank of the first-level compressed data at the current moment.

[0038] Singular Value Decomposition (SVD) is a matrix factorization method that is widely used in fields such as data dimensionality reduction, signal processing, and image compression. By extracting principal components or features from data and removing redundant information, SVD achieves the goals of data compression, noise reduction, and feature extraction.

[0039] Furthermore, S501 specifically includes: S5011: Construct the original data matrix from the current level-one compressed data.

[0040] S5012: Perform SVD decomposition on the data matrix:

[0041] in, Y Represents the original data matrix. U Let Σ denote the orthogonal matrix composed of left singular vectors, and let Σ denote the diagonal matrix of singular values, formed by arranging the singular values ​​in descending order. V Describes an orthogonal matrix composed of right singular vectors. H Represents conjugate matrix operations. u r Indicates the first r The time mode vector of each mode, σ r Indicates the first r Singular values ​​of each mode, v r Indicates the first r Spatial mode vectors of a mode, m This indicates the dimension of the original data matrix.

[0042] S5013: Determine the contribution of each singular mode to the data:

[0043] in, Y r Indicates the first r The contribution of each modality to the data.

[0044] S5014: Calculate the RMS value of each singular mode based on its contribution to the data:

[0045] in, Indicates the first r RMS values ​​of each modality Indicates the first r In the mode, the th i Line number j Column elements, h This indicates the number of rows in the original data matrix. n This indicates the number of columns in the original data matrix.

[0046] S5015: Determine whether the RMS value of each singular mode is greater than the RMS value of the measurement error. If yes, retain the corresponding singular mode. Otherwise, delete the corresponding singular mode.

[0047] It's important to note that the RMS error value is a standard method for measuring data error, used to quantify the difference between predicted and actual values ​​in data. It calculates the average of the squared errors, then takes the square root. The RMS error value is used in data compression and reconstruction to measure the difference between the recovered data and the original data.

[0048] It should be noted that traditional SVD compression methods typically select a fixed number of modes to retain, perhaps based on experience or some simple threshold. The embodiments of this application can achieve adaptive SVD mode selection based on the relationship between the RMS value of each singular mode and the RMS value of the measurement error.

[0049] In this embodiment of the invention, effective modes are selected based on the contribution and RMS value of each singular mode. By comparing the RMS value of the mode with the RMS value of the measurement error, modes that make significant contributions to the data and are unaffected by noise can be effectively retained. This method ensures that only the parts that significantly contribute to the information content are retained during data compression, while removing redundant information and noise, thereby improving compression efficiency and data recovery accuracy. By calculating and retaining the number of effective modes as an estimated rank, the accuracy and storage requirements of the compressed data can be precisely controlled, ensuring that the compression ratio is maximized without losing key information, thus improving the overall performance of data storage and processing.

[0050] S5016: Count the number of retained modes and use the number of retained modes as the estimated rank.

[0051] S502: Compare the estimated rank of the first-level compressed data at the current time with the estimated rank of the first-level compressed data at the previous time to determine if a disturbance has occurred. If yes, proceed to S504. Otherwise, proceed to S503.

[0052] It's important to note that when a disturbance is detected, it signifies a change in the system state, potentially causing changes in data dimensions. At this point, the data changes are significant, and it's crucial to process this changed data as quickly as possible to avoid data loss or insufficient compression. Therefore, once a disturbance is detected, the system directly enters S504, performing secondary compression on the currently stored partition data to ensure data validity while completing the compression. On the other hand, after detecting a disturbance in S502, the system proceeds to S504, skipping S503. This is because, at the time of the event or disturbance, buffer capacity may not be the most critical factor. Compression of the disturbed data needs to be performed as quickly as possible to ensure that critical data is not lost during the disturbance. Simply waiting to see if the partition is full might delay the processing of the disturbed data.

[0053] S503: Assess whether the current storage partition of the slow storage medium is full. If yes, proceed to S504. Otherwise, proceed to S505.

[0054] S504: Employs the SVD dimensionality reduction algorithm to perform secondary compression on the primary compressed data in the current storage partition, obtaining secondary compressed data. The secondary compressed data is then stored in the current storage partition, and a new storage partition is opened in the slow storage medium.

[0055] In one possible implementation, S504 specifically involves: combining the retained singular modes to obtain secondary compressed data, storing the secondary compressed data in the current storage partition, and opening a new storage partition in the slow storage medium.

[0056] In this embodiment of the invention, by employing the SVD dimensionality reduction algorithm to perform secondary compression on the first-level compressed data within the current storage partition, and storing the second-level compressed data in the current partition, the system can further reduce storage space requirements. Combining the retained singular modes ensures that only the parts that significantly contribute to the data are retained, thereby improving the compression ratio and preserving the key information of the data.

[0057] S505: Return to S501 and continue waiting for the next moment's first-level compressed data.

[0058] In this embodiment of the invention, as data continues to increase and change, progressive storage partitioning technology is used to perform secondary compression on the data. This allows for flexible responses to dynamic data changes, ensuring timely deep compression when disturbances occur, while maintaining efficient storage management. If a storage partition is full or a disturbance occurs, the system will use the SVD dimensionality reduction algorithm to further compress the data and store it in a new storage partition. This not only maximizes storage space utilization and avoids over-compression affecting data accuracy, but also ensures data access efficiency and efficient use of storage resources, improving system performance and flexibility.

[0059] Optionally, the slow storage medium includes: a hard disk drive and / or a cloud storage medium.

[0060] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) In this embodiment of the invention, an adaptive multi-level compression technology is adopted. By dynamically determining whether to perform secondary compression based on the access frequency and storage requirements of the primary compressed data, a lightweight compression strategy is used for frequently accessed data, thereby avoiding the extra processing overhead of high-frequency accessed data. For low-frequency accessed data, a deeper secondary compression is performed to save storage space to the maximum extent.

[0061] (2) In this embodiment of the invention, different storage media are used to store the compressed data. According to the access frequency and importance of the data, data with different compression levels are stored in fast storage media and slow storage media respectively. High-frequency access data is stored in fast storage media. The fast storage media is used to store frequently accessed data, which ensures fast access and efficient read and write performance of the data. Low-frequency access data is stored in slow storage media, which improves the utilization efficiency of storage resources.

[0062] Reference manual attached Figure 2 The diagram shows a schematic of the structure of a data optimization system based on adaptive multi-level compression provided by the present invention.

[0063] The present invention also provides a data optimization system 20 based on adaptive multi-level compression, applied to the above-mentioned data optimization method based on adaptive multi-level compression, comprising: Processor 201.

[0064] The memory 202 stores computer-readable instructions, which, when executed by the processor 201, implement the data optimization method based on adaptive multi-level compression as described in the method embodiment.

[0065] The data optimization system 20 based on adaptive multi-level compression provided by the present invention can execute the above-mentioned data optimization method based on adaptive multi-level compression and achieve the same or similar technical effects. To avoid repetition, the present invention will not elaborate further.

[0066] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) In this embodiment of the invention, an adaptive multi-level compression technology is adopted. By dynamically determining whether to perform secondary compression based on the access frequency and storage requirements of the primary compressed data, a lightweight compression strategy is used for frequently accessed data, thereby avoiding the extra processing overhead of high-frequency accessed data. For low-frequency accessed data, a deeper secondary compression is performed to save storage space to the maximum extent.

[0067] (2) In this embodiment of the invention, different storage media are used to store the compressed data. According to the access frequency and importance of the data, data with different compression levels are stored in fast storage media and slow storage media respectively. High-frequency access data is stored in fast storage media. The fast storage media is used to store frequently accessed data, which ensures fast access and efficient read and write performance of the data. Low-frequency access data is stored in slow storage media, which improves the utilization efficiency of storage resources.

[0068] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0069] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0070] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0071] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0072] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0073] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0076] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0079] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] This invention provides a computer-readable storage medium storing a computer program thereon, characterized in that, when executed by a processor, the program implements the data optimization method based on adaptive multi-level compression as described in the method embodiment.

[0081] The present invention provides a computer-readable storage medium that can implement the steps and effects of the data optimization method based on adaptive multi-level compression in the above-described method embodiments. To avoid repetition, the present invention will not repeat the details.

[0082] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) In this embodiment of the invention, an adaptive multi-level compression technology is adopted. By dynamically determining whether to perform secondary compression based on the access frequency and storage requirements of the primary compressed data, a lightweight compression strategy is used for frequently accessed data, thereby avoiding the extra processing overhead of high-frequency accessed data. For low-frequency accessed data, a deeper secondary compression is performed to save storage space to the maximum extent.

[0083] (2) In this embodiment of the invention, different storage media are used to store the compressed data. According to the access frequency and importance of the data, data with different compression levels are stored in fast storage media and slow storage media respectively. High-frequency access data is stored in fast storage media. The fast storage media is used to store frequently accessed data, which ensures fast access and efficient read and write performance of the data. Low-frequency access data is stored in slow storage media, which improves the utilization efficiency of storage resources.

[0084] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0085] The above technical solution has at least the following advantages compared with the existing technology: In this embodiment of the invention, lighting and shadow optimization can be performed on static objects based on light baking technology, and light flow processing technology can be used to optimize game animations, thereby optimizing the graphics of retro online games, enhancing their appeal to new players, and improving their market competitiveness.

[0086] The following points need to be explained: (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.

[0087] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the invention, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.

[0088] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0089] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data optimization method based on adaptive multi-level compression, characterized in that, include: S1: Obtain the data to be compressed; S2: Preprocess the data to be compressed to obtain preprocessed data; S3: Perform first-level compression on the preprocessed data to obtain first-level compressed data; S4: Based on the access frequency and storage requirements of the first-level compressed data, determine whether the first-level compressed data needs to be compressed in the second level; if so, proceed to S5; Otherwise, the first-level compressed data is stored in a high-speed storage medium; S5: Using progressive storage partitioning technology, the first-level compressed data is compressed a second time to obtain second-level compressed data, and the second-level compressed data is stored in a slow storage medium.

2. The data optimization method based on adaptive multi-level compression according to claim 1, characterized in that, The high-speed storage media include: solid-state drives, non-volatile memory, and / or dynamic random access memory.

3. The data optimization method based on adaptive multi-level compression according to claim 1, characterized in that, The slow storage media include: hard disk drives and / or cloud storage media.

4. The data optimization method based on adaptive multi-level compression according to claim 1, characterized in that, S3 specifically includes: The preprocessed data is subjected to primary compression using Huffman compression, LZ78 compression, DEFLATE compression, BWT compression, and / or LZMA compression techniques to obtain primary compressed data.

5. The data optimization method based on adaptive multi-level compression according to claim 1, characterized in that, S4 specifically includes: S401: Calculate the compression level decision index based on the access frequency and storage requirements of the first-level compressed data; S402: Determine whether the compression level decision index of the first-level compressed data is less than the preset index value; if so, determine that the first-level compressed data needs to be compressed and proceed to S5; otherwise, determine that the first-level compressed data does not need to be compressed and store the first-level compressed data in the fast storage medium.

6. The data optimization method based on adaptive multi-level compression according to claim 1, characterized in that, S5 specifically includes: S501: Regarding the current time t The first-level compressed data is subjected to an SVD decomposition dimensionality evaluation to determine the estimated rank of the first-level compressed data at the current moment; S502: Compare the estimated rank of the first-level compressed data at the current moment with the estimated rank of the first-level compressed data at the previous moment to determine whether a disturbance has occurred; if yes, proceed to S504; otherwise, proceed to S503. S503: Assess whether the current storage partition of the slow storage medium is full; if yes, proceed to S504; otherwise, proceed to S505. S504: Using the SVD dimensionality reduction algorithm, the first-level compressed data in the current storage partition is compressed a second time to obtain second-level compressed data. The second-level compressed data is stored in the current storage partition, and a new storage partition is opened in the slow storage medium. S505: Return to S501 and continue waiting for the next moment's first-level compressed data.

7. The data optimization method based on adaptive multi-level compression according to claim 6, characterized in that, S501 specifically includes: S5011: Construct the original data matrix from the first-level compressed data at the current moment; S5012: Perform SVD decomposition on the data matrix; S5013: Determine the contribution of each singular mode to the data; S5014: Calculate the RMS value of each singular mode based on its contribution to the data; S5015: Determine whether the RMS value of each singular mode is greater than the RMS value of the measurement error; if so, retain the corresponding singular mode; otherwise, delete the corresponding singular mode. S5016: Count the number of retained modes and use the number of retained modes as the estimated rank.

8. The data optimization method based on adaptive multi-level compression according to claim 7, characterized in that, Specifically, S504 is: The retained singular modes are combined to obtain secondary compressed data, which is then stored in the current storage partition, and a new storage partition is opened in the slow storage medium.

9. A data optimization system based on adaptive multi-level compression, characterized in that, include: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement the data optimization method based on adaptive multi-level compression as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data optimization method based on adaptive multi-level compression as described in any one of claims 1 to 8.

Images