Data compression and block storage processing method based on deep learning

Abstract

The present invention relates to the field of data processing technology, and in particular to a data compression and block storage processing method based on deep learning. The method comprises the following steps: obtaining integrated data to be compressed; analyzing effective integrated analytical data to be compressed according to the integrated data to be compressed; constructing a heterogeneous data feature analysis tree model; extracting feature data to be compressed of effective integrated data to be compressed through the heterogeneous data feature analysis tree model; designing a recursive sparse coefficient evaluation objective function based on the feature data to be compressed; performing adaptive sparse compression processing on the effective integrated data to be compressed based on the recursive sparse coefficient evaluation objective function to obtain compressed data; performing block processing on the compressed data to generate block compressed data; performing compressed data block storage optimization processing on the block compressed data to generate optimized storage compressed data blocks. The present invention analyzes data features through a deep learning algorithm to achieve efficient processing of data compression and block storage.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data compression and block storage processing method based on deep learning. Background Art

[0002] With the rapid development of information technology, the amount of data generated and stored is growing exponentially, especially in the fields of big data, the Internet of Things, and cloud computing, where the types and complexity of data are increasing. How to effectively compress, store, and manage these massive amounts of data has become one of the hot spots and difficulties in current research. Data compression mainly relies on fixed algorithms, such as Huffman coding and LZW compression. The rapid development of deep learning technology has provided new ideas for data compression, block processing, and other processing. Deep learning models automatically extract features by learning a large amount of data, thereby achieving higher efficiency and accuracy in the process of data compression and storage. Using deep learning methods for data compression can dynamically adjust the compression strategy according to the characteristics of the data, improve the compression ratio and data recovery quality. However, the existing data compression and block storage processing methods based on deep learning cannot dynamically adjust the compression processing according to the characteristics of the data, cannot guarantee the quality of data compression, and have no targeted data block storage management, resulting in a waste of storage resources and low scheduling efficiency. Summary of the invention

[0003] Based on this, the present invention provides a data compression and block storage processing method based on deep learning to solve at least one of the above technical problems.

[0004] To achieve the above purpose, a data compression and block storage processing method based on deep learning includes the following steps:

[0005] Step S1: obtaining integrated data to be compressed; performing effective parsing data analysis and processing according to the integrated data to be compressed to generate effective integrated parsing data to be compressed;

[0006] Step S2: construct a heterogeneous data feature analysis tree model; perform feature subset extraction processing on the effective integrated data to be compressed according to the heterogeneous data feature analysis tree model to generate feature data to be compressed; design a recursive sparse coefficient evaluation objective function based on the feature data to be compressed;

[0007] Step S3: performing adaptive sparse compression processing on the effective integrated data to be compressed based on the recursive compression sparse coefficient evaluation objective function to obtain compressed data;

[0008] Step S4: block-processing the compressed data to generate block compressed data; extracting the data access pattern of the compressed data block according to the block compressed data to generate the data access pattern; performing compressed data block storage optimization processing on the block compressed data based on the data access pattern to generate optimized storage compressed data blocks.

[0009] Further, step S1 includes the following steps:

[0010] Step S11: Obtaining integrated data to be compressed;

[0011] Step S12: performing data analysis on the integrated data to be compressed to generate integrated analysis data to be compressed;

[0012] Step S13: performing heterogeneous data type analysis on the compressed integrated parsed data to generate parsed heterogeneous data types;

[0013] Step S14: Based on parsing heterogeneous data types, compression requirement anomaly detection feedback processing is performed on the integrated parsed data to be compressed to obtain valid integrated parsed data to be compressed.

[0014] Further, step S14 includes the following steps:

[0015] Step S141: performing analysis on compression requirements of heterogeneous data according to the parsed heterogeneous data types to obtain a decision on compression requirements of heterogeneous data;

[0016] Step S142: Designing a compression demand anomaly detection engine by analyzing the compression demand decisions of heterogeneous data;

[0017] Step S143: The compression demand anomaly detection engine performs compression demand anomaly detection feedback processing on the integrated parsed data to be compressed, so as to obtain valid integrated parsed data to be compressed.

[0018] Further, step S143 includes the following steps:

[0019] When the compression requirement anomaly detection engine identifies that the integrated parsing data to be compressed does not meet the compression requirement decision for parsing heterogeneous data, the corresponding integrated parsing data to be compressed is fed back to the terminal to execute the compression anomaly feedback job; or, when the integrated data anomaly detection engine identifies that the integrated parsing data to be compressed meets the compression requirement decision for parsing heterogeneous data, the integrated parsing data to be compressed is marked as valid integrated parsing data to be compressed.

[0020] Further, step S2 includes the following steps:

[0021] Step S21: performing heterogeneous data feature analysis rule analysis according to parsing heterogeneous data types to design heterogeneous data feature analysis rule decision;

[0022] Step S22: extracting heterogeneous data feature analysis factors from the effective integrated data to be compressed through heterogeneous data feature analysis rule decision, and generating heterogeneous data feature analysis factors;

[0023] Step S23: establishing a mapping relationship between heterogeneous data and feature subsets based on a preset decision tree model and heterogeneous data feature analysis factors to construct a heterogeneous data feature analysis tree model;

[0024] Step S24: performing feature subset extraction processing on the effective integrated data to be compressed according to the heterogeneous data feature analysis tree model to generate feature data to be compressed;

[0025] Step S25: performing feature subset information quantity measurement processing according to the feature data to be compressed to generate feature subset information quantity data;

[0026] Step S26: performing mutual information correlation analysis of feature subsets according to the feature data to be compressed, and generating feature subset mutual information correlation data;

[0027] Step S27: Design a recursive sparse coefficient evaluation objective function based on the feature subset information volume data and the feature subset mutual information correlation data.

[0028] Further, step S26 includes the following steps:

[0029] Step S261: performing joint probability distribution and marginal probability distribution analysis of feature subsets according to the feature data to be compressed, and generating feature subset joint probability distribution data and feature subset marginal probability distribution data respectively;

[0030] Step S262: Perform feature subset mutual information correlation analysis based on the feature subset joint probability distribution data and the feature subset edge probability distribution data to generate feature subset mutual information correlation data.

[0031] Further, step S27 includes the following steps:

[0032] Step S271: taking the feature subset information volume data and the feature subset mutual information correlation data as feature subset negative correlation sparse evaluation data, and transmitting the feature subset negative correlation sparse evaluation data to a preset deep learning algorithm for feature subset sparse coefficient evaluation learning processing, so as to construct a sparse coefficient evaluation objective function;

[0033] Step S272: Perform recursive optimization processing on the objective function according to the sparse coefficient evaluation objective function to generate a recursive sparse coefficient evaluation objective function.

[0034] Further, step S3 includes the following steps:

[0035] Step S31: performing feature subset sparse non-zero element analysis on the feature data to be compressed corresponding to the valid integrated data to be compressed based on the recursive compression sparse coefficient evaluation objective function to generate feature subset sparse non-zero element data;

[0036] Step S32: performing compression sparsity evaluation processing according to the sparse non-zero element data of the feature subset to generate compression sparsity evaluation data;

[0037] Step S33: designing a compressed sparse matrix according to the compressed sparsity evaluation data and the corresponding node positions;

[0038] Step S34: performing adaptive sparse compression processing on the effective integrated data to be compressed by compressing the sparse matrix to obtain compressed data.

[0039] Further, step S4 includes the following steps:

[0040] Step S41: performing storage demand distribution feature analysis according to the compressed sparse matrix of the compressed data to generate storage demand distribution feature data;

[0041] Step S42: performing cluster analysis based on the storage demand distribution characteristic data to generate cluster storage demand distribution characteristic data;

[0042] Step S43: performing data block configuration requirement analysis according to the cluster storage requirement distribution characteristic data to generate data block configuration requirement data;

[0043] Step S44: block-processing the compressed data according to the data block configuration requirement data to generate block compressed data;

[0044] Step S45: extracting the data access pattern of the compressed data block according to the block compressed data, and generating a data access pattern;

[0045] Step S46: performing compressed data block storage optimization processing on the block compressed data based on the data access mode to generate optimized storage compressed data blocks.

[0046] Further, step S46 includes the following steps:

[0047] Step S461: performing access pattern time-frequency feature analysis according to the data access pattern to generate access pattern time-frequency feature data;

[0048] Step S462: performing access pattern collaborative correlation feature analysis according to the access pattern time-frequency feature data to generate access pattern collaborative correlation feature data;

[0049] Step S463: performing collaborative association storage processing on the block compressed data according to the access mode collaborative association feature data to generate a storage compressed data block;

[0050] Step S464: performing storage queue scheduling resource analysis and processing according to the access pattern time-frequency characteristic data corresponding to the storage compressed data block, and generating storage queue scheduling resource data;

[0051] Step S465: Perform storage queue optimization processing on the storage compressed data block through the storage queue scheduling resource data to generate an optimized storage compressed data block.

[0052] The beneficial effect of the present invention is that, by systematically acquiring and parsing the data to be compressed, the integrity and accuracy of the data are ensured, a solid foundation is provided for subsequent processing, data preparation time is significantly reduced, and overall processing efficiency is improved. The heterogeneous data type analysis of the parsed data can identify and classify different types of data, which helps to better understand the data characteristics, so as to adopt a more appropriate strategy during compression and improve the compression effect. Through the designed anomaly detection engine, problems in the compression process can be identified in real time and feedback can be given in a timely manner, which not only improves the reliability of data processing, but also provides an improvement basis for subsequent processing, reduces potential risks, clarifies which data meets the compression requirements, thereby ensuring that only verified data is compressed, enhances the standardization and security of data processing, and effectively improves the overall data quality. By parsing heterogeneous data types for feature analysis and extraction, key information in the data can be effectively identified and utilized, thereby providing a more accurate basis for subsequent compression, improving compression efficiency, and constructing a heterogeneous data feature analysis tree model, which can dynamically map the relationship between data and feature subsets, so that data characteristics can be processed more accurately during the compression process. The application of this model enhances the understanding and analysis of data characteristics, promotes the effective measurement of information volume and mutual information correlation analysis, and further optimizes the compression effect. By designing a recursive sparse coefficient evaluation objective function, the compression strategy can be continuously optimized, making the compression process more adaptable and flexible. The application of deep learning technology not only improves the efficiency of data compression and storage, but also provides innovative solutions for processing complex and heterogeneous data, and enhances the intelligence and automation level of data management. The recursive compression sparse coefficient evaluation objective function is used to analyze the sparse non-zero elements of the feature subset of the effective integrated data to be compressed, which realizes the in-depth mining and evaluation of data sparsity. It can not only accurately identify the key information in the data to be compressed, but also provide a scientific basis for the compression strategy, thereby optimizing the compression efficiency. By designing a compressed sparse matrix, data can be stored in a more flexible and efficient way, significantly reducing storage requirements and improving data reading and processing speed. In addition, the application of adaptive sparse compression processing enables data to minimize redundant information while maintaining important features, further improving the compression ratio, showing high adaptability and intelligence when processing complex data, and improving the efficiency and security of data management. According to the compressed sparse matrix of compressed data, the storage demand distribution characteristics are analyzed to achieve an accurate evaluation of storage requirements. This characteristic data lays the foundation for subsequent cluster analysis, thereby generating more detailed cluster storage demand distribution characteristic data and improving the intelligent level of storage configuration. After further data block configuration demand analysis, the configuration of data blocks is optimized according to different storage requirements to ensure efficient use of storage resources.In the block processing process, the generated block compressed data can clarify the user's behavioral characteristics in data storage and access by extracting the data access pattern. This analysis of data access patterns not only provides an important basis for the storage optimization of compressed data blocks, but also helps to identify the best storage strategy and improve the efficiency of data access. Through the time-frequency feature analysis and collaborative correlation feature analysis of the access pattern, the data access behavior can be understood more deeply, and then the collaborative correlation storage processing is implemented to improve the performance of the storage compressed data blocks. Through the analysis and optimization processing of the storage queue scheduling resources, the efficient scheduling and management of the storage compressed data blocks in actual use are ensured, and excellent effects are shown in optimizing data storage, improving access efficiency and reducing storage costs. Therefore, the data compression and block storage processing method based on deep learning of the present invention can learn data characteristics through a deep learning algorithm to dynamically adjust the compression processing, etc., to ensure that data compression is targeted at redundant information and retains important feature information, and has targeted data block storage management, so that storage resources are efficiently utilized and the scheduling efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the steps of a data compression and block storage processing method based on deep learning of the present invention;

[0054] Figure 2 for Figure 1 Detailed implementation steps of step S2 in the flowchart;

[0055] Figure 3 for Figure 1 Detailed implementation steps of step S3 in FIG.

[0056] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0059] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0060] To achieve this, please refer to Figures 1 to 3 The present invention provides a data compression and block storage processing method based on deep learning. In the embodiment of the present invention, please refer to Figure 1 As shown, it is a schematic diagram of the step flow of the data compression and block storage processing method based on deep learning of the present invention, and the data compression and block storage processing method based on deep learning includes the following steps:

[0061] Step S1: obtaining integrated data to be compressed; performing effective parsing data analysis and processing according to the integrated data to be compressed to generate effective integrated parsing data to be compressed;

[0062] In an embodiment of the present invention, integrated data to be compressed is obtained from multiple data sources through a data acquisition system, for example, extracted from an e-commerce platform database or an external data platform through an API interface. Assume that the data source includes structured data (such as table data in an SQL database), semi-structured data (such as JSON log files) and unstructured data (such as multimedia files). The data acquisition system can use data connection tools (such as Apache NiFi or Talend) to achieve efficient data acquisition. For example, SQL query statements can be set from an SQL database to obtain data in a custom time period; for JSON log files, log information in a target time range can be extracted by parsing specific API interface logs. In addition, batch mode or stream processing mode can be set for data acquisition. In batch mode, the time interval for data acquisition can be set to 10 minutes or 1 hour; while in stream processing mode, data can be monitored and processed in seconds through real-time monitoring tools (such as Kafka Streams). For structured data (such as SQL data tables), it is directly obtained and converted into a unified table format through database queries; for semi-structured data (such as JSON), it is parsed into DataFrame data format using the json library or pandas library in Python. For unstructured data (such as video files), you can use multimedia processing tools such as FFmpeg to convert video files into key frames, and use image processing tools (such as OpenCV) to extract key frames as image data to facilitate feature extraction of deep learning models. During the data analysis process, use ETL tools (such as Apache Spark or Apache Flink) to clean and convert data, standardize the format, and generate integrated analysis data to be compressed. Perform heterogeneous data type analysis on the integrated analysis data to be compressed, identify the characteristics of different data types, use data type detection tools (such as the pandas library in Python or the type detection function provided by the SQL database) to classify the parsed heterogeneous data types, and generate parsed heterogeneous data types. Perform compression requirement anomaly detection on the parsed heterogeneous data types to identify whether the data meets the compression requirement standards, and set the compression requirement detection standards (such as the maximum difference limit of numerical data, the fluctuation range of time series data, and the repetition frequency of text data) to determine whether the data compression requirements meet the standards. If it is detected that the integrated and parsed data to be compressed meets the set compression requirement standards, it is marked as valid data; otherwise, the detected abnormal data is fed back for processing, and the data acquisition or preprocessing parameters are readjusted to ensure that the generated valid integrated and parsed data to be compressed is suitable for subsequent compression processing.

[0063] Step S2: construct a heterogeneous data feature analysis tree model; perform feature subset extraction processing on the effective integrated data to be compressed according to the heterogeneous data feature analysis tree model to generate feature data to be compressed; design a recursive sparse coefficient evaluation objective function based on the feature data to be compressed;

[0064] In an embodiment of the present invention, the design of heterogeneous data feature analysis rules is performed based on the parsed heterogeneous data types, and the integrated data to be analyzed is loaded. The data types include structured data (such as transaction records in SQL databases), semi-structured data (such as user behavior data in JSON format) and unstructured data (such as text logs). For different data types, corresponding feature analysis rules are formulated. By analyzing historical data, the boundary conditions of effective features are determined to ensure that the designed rules can effectively support subsequent compression decisions and data storage optimization. Using the formulated feature analysis rules, the effective integrated data to be compressed is subjected to heterogeneous data feature analysis factor extraction, and all extracted feature factors will form a new data set. Based on the extracted heterogeneous data feature analysis factors, a mapping relationship between heterogeneous data and feature subsets is constructed to form a heterogeneous data feature analysis tree model, and a decision tree algorithm tool, such as Scikit-learn, is used to load the previously extracted feature factor data set to perform model training. The parameters of the decision tree are set, such as a maximum depth of 5, to ensure the interpretability of the model and prevent overfitting. The decision tree model constructs a tree structure by analyzing the distribution and mutual relationship of each feature factor. The leaf nodes of the tree correspond to different user categories or behavior patterns. Through this mapping relationship, efficient feature analysis results are provided. According to the constructed heterogeneous data feature analysis tree model, feature subsets are extracted from the effective integrated data to be compressed. Data processing tools such as Pandas are used to load the effective integrated data to be compressed. According to the structure of the feature analysis tree model, key feature subsets are screened out to generate feature data to be compressed, which contains all valid features. The number of rows in the data set is the same as that of the original integrated data, but the number of columns is limited to the extracted features. Feature subset information quantity measurement processing is performed on the feature data to be compressed to generate feature subset information quantity data, such as using the entropy concept in information theory for quantification. Through data analysis tools such as Pandas, the information entropy of each feature is calculated, and the probability distribution of each value of the feature in the data set is calculated. All features to be compressed are calculated accordingly. The entropy value is calculated, and finally the feature subset information quantity data set is summarized to generate the entropy value corresponding to each feature. According to the feature data to be compressed, the mutual information correlation analysis of feature subsets is performed. The mutual information calculation method in Scikit-learn is used to analyze the mutual dependence and redundancy between each feature subset, calculate the mutual information between each pair of features, set parameters, and summarize the mutual information results of all features into the mutual information correlation data set to provide a basis for subsequent feature selection and compression decisions, ensuring that effective information is retained as much as possible during the compression process while eliminating redundancy. Based on the feature subset information data and the feature subset mutual information correlation data, a recursive sparse coefficient evaluation objective function is designed. The entropy value in the feature subset information data and the feature subset mutual information correlation data are used to calculate the redundancy between features. Through a mathematical model, the relationship between the feature information and its mutual information is incorporated into the objective function design to minimize redundancy and maximize information retention.Specifically, the objective function is set to be. , represents the mutual information between the feature and the target variable, Represents the amount of information in a feature. By adjusting these parameters, the importance of a feature subset in the data compression process can be evaluated. In the implementation process, a recursive algorithm is used to iteratively solve the objective function. By setting the initial feature set and the corresponding entropy and mutual information values, the algorithm iteratively updates the feature set until the objective function converges, identifying a feature subset with high information content and low redundancy, ensuring that only features that are effective for the deep learning model are retained in subsequent data compression operations.

[0065] Step S3: performing adaptive sparse compression processing on the effective integrated data to be compressed based on the recursive compression sparse coefficient evaluation objective function to obtain compressed data;

[0066] In an embodiment of the present invention, a feature subset is extracted from the effective integrated data to be compressed, and the importance of the feature is determined by a sparsity evaluation model. The model evaluates the contribution of each feature to the data compression effect based on the established objective function. During the analysis process, feature selection techniques, such as LASSO regression, are applied to calculate the sparse coefficients of each feature and identify features related to the compression requirements. At this stage, the feature matrix is ​​converted into a sparse format, and the memory space is effectively saved by the sparse matrix representation method. The sparse matrix is ​​constructed and operated using the SciPy library in Python to generate sparse non-zero element data of the feature subset. The sparse non-zero element data is analyzed to determine the information retention capacity and compression effect of the feature after compression. The compression potential of the data is evaluated by calculating the sparsity indicators of the sparse matrix, such as sparsity and the ratio of non-zero elements. Each feature in the feature subset is analyzed one by one, and its performance under sparse representation is calculated. Sparsity is defined as the ratio of the number of non-zero elements to the total number of elements, and compressed sparsity evaluation data is generated. A compressed sparse matrix is ​​designed according to the compressed sparsity evaluation data and the corresponding node positions. An efficient sparse matrix is ​​constructed by combining the non-zero elements in the feature subset with their corresponding sparsity evaluation data. Using the coordinate compression method, the non-zero elements in the matrix and their corresponding row and column indices are recorded as triplets (row index, column index, value) to reduce storage requirements. This design process ensures that the compressed sparse matrix can be quickly accessed and processed to achieve efficient data compression processing. The effective integrated data to be compressed is adaptively sparsely compressed by compressing the sparse matrix. Adaptive compression algorithms, such as dynamic programming or sparse coding algorithms based on deep learning, are used to ensure that the structure and characteristics of the data are effectively modeled during the compression process. The input is a compressed sparse matrix and its corresponding feature subset. The goal is to generate compressed data. The compression coefficient is designed by compressing the sparse matrix, that is, a lower compression coefficient is set for a higher amount of information and higher information relevance. By adjusting the sparsity parameters and compression strategy, the compression efficiency and decompression quality are optimized. The output compressed data will meet the storage requirements while maintaining the integrity and availability of the data.

[0067] Step S4: block-processing the compressed data to generate block compressed data; extracting the data access pattern of the compressed data block according to the block compressed data to generate the data access pattern; performing compressed data block storage optimization processing on the block compressed data based on the data access pattern to generate optimized storage compressed data blocks.

[0068] In an embodiment of the present invention, a compressed sparse matrix is ​​analyzed in detail to determine the space required for each feature during storage. The distribution characteristics of storage requirements are identified by calculating the number of non-zero elements of each feature in the sparse matrix and its proportion in the overall matrix. The sparse matrix is ​​operated using a data analysis tool (such as Pandas or NumPy) to calculate the storage space requirements of each feature. The generated storage requirement distribution feature data will include the storage requirements of each feature and its distribution map in the overall feature set. These data will provide a basis for subsequent steps. Cluster analysis is performed based on the generated storage requirement distribution feature data to generate cluster storage requirement distribution feature data. A suitable clustering algorithm, such as K-means, is selected to perform cluster analysis using the storage requirement data of the feature as input. The center point of each cluster represents the average storage requirement of a group of features. By analyzing the clustering results, cluster storage requirement distribution feature data is generated. These data contain information such as the characteristics of each cluster, the corresponding storage requirement characteristics, and the number of samples of each cluster, providing a basis for subsequent configuration requirement analysis. The characteristics of each cluster are analyzed in detail using the cluster storage requirement distribution feature data. Assume that the clustering results are divided into 5 groups, each representing a different storage demand pattern. For each cluster, calculate the mean, standard deviation, maximum and minimum values ​​of its storage demand, and set the size of the data block based on these statistical features. For example, for clusters with higher storage requirements, data blocks can be configured on fast-access SSDs, while for clusters with lower storage requirements, traditional HDDs can be selected. The generated data block configuration demand data contains information such as the block size, number of blocks, storage location and access mode of each cluster. The compressed data is divided according to the previously set data block size, features are extracted from the compressed data, and these features are packaged. Each block should contain the index information of the feature and its corresponding storage location to ensure the efficiency of subsequent data access and generate block compressed data. Analyze the access logs of the block compressed data, which record the access to the compressed data blocks in a specific time period. Access logs usually contain timestamps, data block identifiers, access frequency and access type (e.g., read, write). By collating and summarizing these logs, the access frequency of each data block is counted. Using statistical analysis methods, such as frequency analysis and time series analysis, the access patterns of each data block are identified. Data blocks with high access frequencies are marked as hot blocks, which may require faster storage media. Data blocks with low access frequencies are marked as cold blocks, which can be stored on slower media. Data access patterns are generated, including the access frequency of each data block, the distribution of access types (such as the ratio of sequential access to random access), and time change trends. In the data access pattern report, hot blocks and cold blocks are identified.For hot blocks, move them to higher-performance storage devices (such as SSDs) to increase data access speed; while cold blocks can be migrated to low-cost storage devices (such as HDDs) to reduce storage costs. Rearrange data blocks based on the characteristics of the access pattern. For example, if it is found that some data blocks are often accessed sequentially, store these data blocks in sequence to improve access efficiency. In order to optimize storage efficiency, consider merging or splitting data blocks. For example, for "cold blocks" with small storage requirements, merge multiple small blocks into one large block to reduce the overhead of storing metadata. At the same time, ensure that access efficiency is maintained after merging. Generate a report on optimized storage of compressed data blocks, including the new storage location of each data block, storage device type, block size, and its access pattern. Through such optimization processing, data access speed can be significantly improved, storage costs can be reduced, and the overall performance of the system can be enhanced.

[0069] Further, step S1 includes the following steps:

[0070] Step S11: Obtaining integrated data to be compressed;

[0071] In an embodiment of the present invention, integrated data to be compressed is obtained from multiple data sources through a data acquisition system, for example, extracted from an e-commerce platform database or an external data platform through an API interface. It is assumed that the data source includes structured data (such as table data in an SQL database), semi-structured data (such as JSON log files), and unstructured data (such as multimedia files). The data acquisition system can use data connection tools (such as Apache NiFi or Talend) to achieve efficient data acquisition. For example, from an SQL database, an SQL query statement can be set to obtain data in a custom time period; for JSON log files, log information in a target time range can be extracted by parsing specific API interface logs. In addition, batch mode or stream processing mode can be set for data acquisition. In batch mode, the time interval for data acquisition can be set to 10 minutes or 1 hour; in stream processing mode, data can be monitored and processed in seconds through real-time monitoring tools (such as Kafka Streams).

[0072] Step S12: performing data analysis on the integrated data to be compressed to generate integrated analysis data to be compressed;

[0073] In an embodiment of the present invention, for structured data (such as SQL data tables), it is directly obtained through database queries and converted into a unified table format; for semi-structured data (such as JSON), it is parsed into DataFrame data format using the json library or pandas library in Python to facilitate subsequent processing. Assuming that the acquired JSON data contains fields, the json.loads() method is used to parse it into a Python object, and the key fields are extracted for data standardization. For unstructured data (such as video files), the video files can be converted into key frames through multimedia processing tools such as FFmpeg, and the key frames can be extracted as image data using image processing tools (such as OpenCV) to facilitate feature extraction of deep learning models. During the data parsing process, ETL tools (such as Apache Spark or Apache Flink) are used to clean and convert the data, standardize the format, and generate integrated parsed data to be compressed.

[0074] Step S13: performing heterogeneous data type analysis on the compressed integrated parsed data to generate parsed heterogeneous data types;

[0075] In an embodiment of the present invention, heterogeneous data types are analyzed for the compressed integrated parsed data, the features of different data types are identified, and data type detection tools (such as the pandas library in Python or the type detection function provided by the SQL database) are used. For example, for fields of structured data, the type of each column (such as integer, floating point number, string, etc.) is viewed through the pandas.DataFrame.dtypes method; for unstructured data (such as image data), the image content is annotated (such as image size, color distribution, etc.) through image classification tools. Assuming that the data contains numerical data, time series data (such as the timestamp of log records), text data (such as user log information), etc., it is labeled through specific analysis tools. For example, the timestamp is marked as "time series type", the text information is marked as "text type", etc. Use these labels to classify the parsed heterogeneous data types and generate parsed heterogeneous data types.

[0076] Step S14: Based on parsing heterogeneous data types, compression requirement anomaly detection feedback processing is performed on the integrated parsed data to be compressed to obtain valid integrated parsed data to be compressed.

[0077] In an embodiment of the present invention, compression demand anomaly detection is performed on the parsed heterogeneous data types to identify whether the data meets the compression demand standard, and the compression demand detection standard (such as the maximum difference limit of numerical data, the fluctuation range of time series data, and the repetition frequency of text data) is set to determine whether the compression demand of the data meets the standard. For example, a periodicity threshold is set for time series data. If the fluctuation within a period of time is less than the set 0.2 threshold, the data is considered suitable for compression; for text data, if the repetition frequency within a period of time is greater than 80%, the text is considered suitable for compression. During the detection process, an anomaly detection algorithm (such as Isolation Forest, DBSCAN or Z-score method) is used to analyze the data for compression demand anomaly. For example, the periodicity of time series data is analyzed by autocorrelation. If it is detected that the integrated parsed data to be compressed meets the set compression demand standard, it is marked as valid data; otherwise, the detected abnormal data is fed back and the data acquisition or preprocessing parameters are readjusted to ensure that the generated valid integrated parsed data to be compressed is suitable for subsequent compression processing.

[0078] Further, step S14 includes the following steps:

[0079] Step S141: performing analysis on compression requirements of heterogeneous data according to the parsed heterogeneous data types to obtain a decision on compression requirements of heterogeneous data;

[0080] In an embodiment of the present invention, a comprehensive analysis is performed on the parsed heterogeneous data types to determine the compression requirements of each data type. Assume that the integrated parsed data to be compressed includes text data, image data, and sensor data. The importance of text data can be analyzed by the TF-IDF (Term Frequency-Inverse Document Frequency) method, the text paragraphs that need to be compressed are evaluated, and a threshold is set. For example, if the TF-IDF value of a certain paragraph of text is lower than 0.1, it can be considered that the compression requirement of the text is high. For image data, image content analysis technology is used, such as using a convolutional neural network to extract features from the image and calculate the complexity of each image. Image complexity can be evaluated by indicators such as color histograms and texture features. Assume that after processing the image, the complexity value obtained is complexity_score. When complexity_score is greater than 0.5, the image is considered suitable for compression. Based on the above analysis, a compression requirement decision report for parsing heterogeneous data is formed to clarify the compression strategy for each data type.

[0081] Step S142: Designing a compression demand anomaly detection engine by analyzing the compression demand decisions of heterogeneous data;

[0082] In an embodiment of the present invention, a compression demand anomaly detection engine is designed to effectively monitor the integrated parsed data to be compressed based on the compression demand decision. The engine is implemented based on a rule engine (such as Drools). In the specific implementation, a rule-based engine is constructed to perform detection by setting feature thresholds of different types of data (such as TF-IDF of text, complexity of image, and sparsity of sensor data). For example, assume that the following rule is set: when the TF-IDF of text data is less than 0.1 and the complexity of the image is greater than 0.5, it is considered that there is a compression demand. The design of the engine needs to implement a data acquisition module, a rule processing module, and a feedback module to ensure smooth data flow and timely handling of anomalies. The engine can be implemented using the Python programming language and its Flask framework to ensure easy deployment and integration.

[0083] Step S143: The compression demand anomaly detection engine performs compression demand anomaly detection feedback processing on the integrated parsed data to be compressed, so as to obtain valid integrated parsed data to be compressed.

[0084] In an embodiment of the present invention, the integrated parsed data to be compressed will be passed to the compression requirement anomaly detection engine through the previously designed API. The engine first extracts features from the received data and uses the previously trained model for prediction. If the model output result shows that the data meets the compression requirements (such as the compression rate is within the required range), the data is marked as valid integrated parsed data to be compressed and stored in the database for subsequent processing; if it is detected that the data does not meet the compression requirements, the data is recorded as abnormal, and the automatic feedback mechanism is triggered to notify the data processing terminal for manual intervention or reprocessing. Assume that the data set to be compressed contains 1000 text data and 500 images. After the detection engine runs, the TF-IDF value of the text data is first analyzed and the average value of each record is calculated. If the average value is less than 0.1, it is recorded as valid integrated parsed data to be compressed; then the complexity of the image data is calculated. For images with a complexity value greater than 0.5, they are also marked as valid integrated parsed data to be compressed. All data marked as abnormal will be fed back to the data processing system.

[0085] Further, step S143 includes the following steps:

[0086] When the compression requirement anomaly detection engine identifies that the integrated parsing data to be compressed does not meet the compression requirement decision for parsing heterogeneous data, the corresponding integrated parsing data to be compressed is fed back to the terminal to execute the compression anomaly feedback job; or, when the integrated data anomaly detection engine identifies that the integrated parsing data to be compressed meets the compression requirement decision for parsing heterogeneous data, the integrated parsing data to be compressed is marked as valid integrated parsing data to be compressed.

[0087] In an embodiment of the present invention, the compression demand anomaly detection engine performs real-time analysis on the integrated parsing data to be compressed. If it is detected that the data does not meet the criteria for compression demand decision (for example, the compression rate does not reach the preset threshold, the data type does not meet expectations, etc.), the engine will record the anomaly and generate a detailed feedback report. The engine uses a Python script to generate a report, including data features that do not meet the conditions, the timestamp of the anomaly detection, and the recommended solutions (such as adjusting compression parameters or changing compression algorithms). The feedback report and the corresponding integrated parsing data to be compressed are sent to the terminal through the API interface to trigger the compression anomaly feedback job. At the terminal, the operation and maintenance personnel present the received anomaly information through the interface, including visualization effects of charts and data, so as to quickly identify the root cause of the problem. The operation and maintenance personnel can make adjustments according to the suggestions in the feedback report, and record the processing process and results to ensure that the problem is properly resolved and avoid similar problems in the future. The compression demand anomaly detection engine confirms that the data meets the criteria for parsing heterogeneous data compression demand decision (such as the data type is consistent with expectations, the compression rate meets the requirements, etc.) by analyzing the integrated parsing data to be compressed. The engine marks this data as valid integrated parsing data to be compressed, and classifies and stores the data that meets the standards through a marking mechanism. The compression demand anomaly detection engine writes valid data into the database, along with information such as data features, timestamps, and processing status, using data management tools (such as MongoDB or PostgreSQL) to ensure data integrity and traceability. In addition, the system triggers an automated workflow to send the integrated parsed data marked as valid for compression to the subsequent compression processing module.

[0088] Further, as an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Detailed step flow diagram of step S2 in FIG. 1 , in this embodiment, step S2 includes the following steps:

[0089] Step S21: performing heterogeneous data feature analysis rule analysis according to parsing heterogeneous data types to design heterogeneous data feature analysis rule decision;

[0090] In an embodiment of the present invention, the design of heterogeneous data feature analysis rules is performed based on the parsed heterogeneous data types. First, the integrated data to be analyzed is loaded through a data preprocessing tool, such as Apache Spark or Apache Flink. The data types include structured data (such as transaction records in an SQL database), semi-structured data (such as user behavior data in JSON format), and unstructured data (such as text logs). For different data types, corresponding feature analysis rules are formulated. For example, for transaction record data, the rules may include "transaction amount>1,000 yuan" and "transaction frequency>5 times / day". At the same time, for user behavior data, feature analysis rules such as "access duration>3 minutes" and "page views>10" are set. When designing rules, the business significance and compression requirements of various data features need to be considered. By analyzing historical data, the boundary conditions of effective features are determined to ensure that the designed rules can effectively support subsequent compression decisions and data storage optimization.

[0091] Step S22: extracting heterogeneous data feature analysis factors from the effective integrated data to be compressed through heterogeneous data feature analysis rule decision, and generating heterogeneous data feature analysis factors;

[0092] In an embodiment of the present invention, the established feature analysis rules are used to extract heterogeneous data feature analysis factors for the effective integrated data to be compressed. First, a data analysis platform such as Pandas is used to load and clean the compressed data to ensure the uniformity and integrity of the data format. In this process, for each data type, a data extraction tool is used to extract key feature factors. For example, factors such as "user ID", "access timestamp", and "visit page URL" are extracted from user behavior data. Then, through aggregation operations, the statistical characteristics of each factor are calculated, such as calculating the average access time and total number of visits for each user. Assuming that 1,000 user records are processed, the results show that the average access time of "User A" is 4 minutes and the total number of visits is 15 times. All extracted feature factors will form a new data set, providing a basis for subsequent model construction.

[0093] Step S23: establishing a mapping relationship between heterogeneous data and feature subsets based on a preset decision tree model and heterogeneous data feature analysis factors to construct a heterogeneous data feature analysis tree model;

[0094] In an embodiment of the present invention, based on the extracted heterogeneous data feature analysis factors, a mapping relationship between heterogeneous data and feature subsets is constructed to form a heterogeneous data feature analysis tree model, and a decision tree algorithm tool, such as Scikit-learn, is used to load the previously extracted feature factor data set to perform model training. The parameters of the decision tree are set, such as a maximum depth of 5, to ensure the interpretability of the model and prevent overfitting. The decision tree model constructs a tree structure by analyzing the distribution and mutual relationship of each feature factor, in which each node represents a feature and its decision condition, such as "access time> 3 minutes" as a node splitting criterion. Ultimately, the leaf nodes of the tree correspond to different user categories or behavior patterns. Through this mapping relationship, efficient feature analysis results are provided, which are helpful for subsequent data compression decisions and storage optimization operations, and ensure data integrity and efficiency during the compression process.

[0095] Step S24: performing feature subset extraction processing on the effective integrated data to be compressed according to the heterogeneous data feature analysis tree model to generate feature data to be compressed;

[0096] In an embodiment of the present invention, feature subsets are extracted from valid integrated data to be compressed based on the constructed heterogeneous data feature analysis tree model, and data processing tools such as Pandas are used to load valid integrated data to be compressed, and key feature subsets are screened out based on the structure of the feature analysis tree model. For example, if the tree model displays "user ID", "access duration" and "number of visited pages" as key features, these three features are extracted as feature data to be compressed. To ensure data consistency, all extracted features will be formatted in a standardized manner, using a unified numerical range and data type (for example, the access duration is converted to seconds, and the number of visited pages is kept as an integer). After the extraction is completed, feature data to be compressed is generated, including all valid features. The number of rows in the data set is the same as the original integrated data, but the number of columns is limited to the extracted features.

[0097] Step S25: performing feature subset information quantity measurement processing according to the feature data to be compressed to generate feature subset information quantity data;

[0098] In an embodiment of the present invention, feature subset information quantity measurement processing is performed on the feature data to be compressed to generate feature subset information quantity data. First, the selected data set contains multiple features, such as "user access frequency", "average access time" and "number of visited pages". The entropy concept in information theory is used for quantification. Through data analysis tools such as Pandas, the information entropy of each feature is calculated, and the probability distribution of each value of the feature in the data set is calculated. Taking "user access frequency" as an example, the frequency distribution of the feature is counted, the probability of occurrence of each value is calculated, and then the information entropy value is obtained. Assume that the entropy value of the feature is 1.5 in the analysis, indicating that it contains a high amount of information and can reflect changes in user behavior. Entropy values ​​are calculated for all features to be compressed in this way, and finally a feature subset information quantity data set is generated, including the entropy value corresponding to each feature. By combining these information quantity data with the mutual information correlation data of other features, a quantitative basis is provided for subsequent feature selection and compression strategies, and feature subset information quantity data is generated.

[0099] Step S26: performing mutual information correlation analysis of feature subsets according to the feature data to be compressed, and generating feature subset mutual information correlation data;

[0100] In the embodiment of the present invention, mutual information correlation analysis of feature subsets is performed based on the feature data to be compressed, and the mutual information calculation method in Scikit-learn is used to analyze the mutual dependence and redundancy between each feature subset, calculate the mutual information between each pair of features, and set parameters, such as calculating the mutual information of the features "visit duration" and "number of visited pages". If the calculated result is a mutual information value of 0.5, it means that there is a certain correlation between the two features. The mutual information results of all features are summarized into a mutual information correlation data set to provide a basis for subsequent feature selection and compression decisions, ensuring that valid information is retained as much as possible during the compression process while eliminating redundancy.

[0101] Step S27: Design a recursive sparse coefficient evaluation objective function based on the feature subset information volume data and the feature subset mutual information correlation data.

[0102] In the embodiment of the present invention, a recursive sparse coefficient evaluation objective function is designed based on the feature subset information data and the feature subset mutual information correlation data. First, the redundancy between features is calculated using the entropy value in the feature subset information data and the feature subset mutual information correlation data. Through a mathematical model, the relationship between the feature information and its mutual information is incorporated into the objective function design to minimize redundancy and maximize information retention. Specifically, the objective function is set to , represents the mutual information between the feature and the target variable, Represents the amount of information in a feature. By adjusting these parameters, the importance of a feature subset in the data compression process can be evaluated. In the implementation process, a recursive algorithm is used to iteratively solve the objective function. By setting the initial feature set and the corresponding entropy and mutual information values, the algorithm iteratively updates the feature set until the objective function converges. Through this process, feature subsets with high information content and low redundancy can be identified, ensuring that only features that are effective for the deep learning model are retained in subsequent data compression operations.

[0103] Further, step S26 includes the following steps:

[0104] Step S261: performing joint probability distribution and marginal probability distribution analysis of feature subsets according to the feature data to be compressed, and generating feature subset joint probability distribution data and feature subset marginal probability distribution data respectively;

[0105] In an embodiment of the present invention, the joint probability distribution and marginal probability distribution analysis of feature subsets are performed based on the feature data to be compressed. First, features such as "user access frequency" and "number of visited pages" are extracted from the feature data set to be compressed. Using statistical analysis tools such as NumPy, the joint probability distribution of the features is calculated, which is defined as the probability of two features occurring at the same time. For example, the frequency of users simultaneously accessing multiple pages in a specific time period is counted, and it is divided by the total number of visits to obtain the joint probability. Subsequently, the marginal probability distribution is calculated. The marginal probability describes the distribution of a single feature, and the same statistical tools are used to calculate the probability of each feature occurring independently. For example, the frequency of occurrence of "user access frequency" in the entire data set is counted to obtain its marginal probability distribution. In this way, the joint probability distribution data and marginal probability distribution data of the feature subset are generated, providing the necessary basic data for subsequent mutual information analysis.

[0106] Step S262: Perform feature subset mutual information correlation analysis based on the feature subset joint probability distribution data and the feature subset edge probability distribution data to generate feature subset mutual information correlation data.

[0107] In an embodiment of the present invention, mutual information correlation analysis is performed based on the generated feature subset joint probability distribution data and marginal probability distribution data. Mutual information is used to measure the dependency between two features, and the calculation formula is that the mutual information between feature X and feature Y is the ratio of their joint probability to marginal probability. Expressed by the formula, the mutual information is equal to the sum of the ratio of the joint probability of features X and Y to the marginal probability of features X and feature Y. Using the previously obtained joint probability distribution and marginal probability distribution data, the mutual information of the feature subsets is calculated pair by pair through data analysis tools (such as SciPy). Taking "user access frequency" and "number of visited pages" as examples, their mutual information values ​​are analyzed. If the mutual information result is 0.8, it means that there is a strong correlation between the two features. The generated feature subset mutual information correlation data can provide a basis for subsequent feature selection, ensuring that the most representative features are retained during the compression process, thereby improving the performance and accuracy of the subsequent deep learning model.

[0108] Further, step S27 includes the following steps:

[0109] Step S271: taking the feature subset information volume data and the feature subset mutual information correlation data as feature subset negative correlation sparse evaluation data, and transmitting the feature subset negative correlation sparse evaluation data to a preset deep learning algorithm for feature subset sparse coefficient evaluation learning processing, so as to construct a sparse coefficient evaluation objective function;

[0110] In an embodiment of the present invention, feature subset negative correlation sparse evaluation data is constructed based on feature subset information data and feature subset mutual information correlation data. Through statistical analysis, feature subsets with low information content and negative correlation with other features are identified. These features may have a negative impact on the predictive ability of the model, so sparse evaluation is required. These evaluation data are organized into a data structure (such as a NumPy array) for subsequent processing. The feature subset negative correlation sparse evaluation data is transmitted to a preset deep learning algorithm, and a suitable model such as a convolutional neural network (CNN) or a recurrent neural network (RNN) is used to evaluate the feature subset sparse coefficients. The model input includes negative correlation sparse evaluation data and corresponding labels, and the loss function is set to mean square error (MSE) to optimize the network weights. During the learning process, the standard gradient descent method is used, the learning rate is adjusted to 0.001, and the batch size is set to 32 to ensure the convergence of the model. Through this series of operations, a sparse coefficient evaluation objective function is constructed, which can effectively measure the importance of each feature in the data compression process.

[0111] Step S272: Perform recursive optimization processing on the objective function according to the sparse coefficient evaluation objective function to generate a recursive sparse coefficient evaluation objective function.

[0112] In an embodiment of the present invention, a recursive optimization process of the objective function is performed according to the constructed sparse coefficient evaluation objective function. This process uses an adaptive optimization algorithm (such as the Adam optimizer) to update the parameters of the model through continuous iteration to seek to minimize the loss function. The number of iterations is set to 1000, allowing the model to adjust the learning rate according to the current parameters in each iteration to achieve a faster convergence effect. During the optimization process, the sparse coefficient evaluation objective function is compared with the previous round of optimization results, and the relative rate of change is calculated to determine whether to continue iterating. When the rate of change is less than the set threshold (for example, 0.01), the optimization process will stop in advance to ensure the effective use of computing resources. The recursive sparse coefficient evaluation objective function finally generated can provide a clear basis for subsequent feature selection and data compression, ensuring that during the compression process, the features that have the greatest impact on model performance are retained, while redundant features are eliminated, thereby improving the efficiency and accuracy of data processing.

[0113] Further, as an embodiment of the present invention, refer to Figure 3 As shown, Figure 1 Detailed step flow diagram of step S3 in the embodiment, step S3 includes the following steps:

[0114] Step S31: performing feature subset sparse non-zero element analysis on the feature data to be compressed corresponding to the valid integrated data to be compressed based on the recursive compression sparse coefficient evaluation objective function to generate feature subset sparse non-zero element data;

[0115] In an embodiment of the present invention, a feature subset in the effective integrated data to be compressed is extracted, and the importance of the feature is determined by a sparsity evaluation model. The model evaluates the contribution of each feature to the data compression effect based on the established objective function. During the analysis process, feature selection techniques, such as LASSO regression, are applied to calculate the sparse coefficients of each feature and identify the features related to the compression requirements. At this stage, the feature matrix is ​​converted into a sparse format, and the memory space is effectively saved by the sparse matrix representation method. The SciPy library in Python is used to construct and calculate the sparse matrix. The generated feature subset sparse non-zero element data contains the values ​​of all non-zero elements and their corresponding positions, and the final result is output in the form of a sparse matrix, which is convenient for the processing and analysis of subsequent steps. For example, the corresponding compression sparse coefficients are allocated according to the feature data to be compressed corresponding to the effective integrated data to be compressed. For example, the compression sparse coefficients are correspondingly lower for more important features, and the feature subset sparse non-zero element data are generated.

[0116] Step S32: performing compression sparsity evaluation processing according to the sparse non-zero element data of the feature subset to generate compression sparsity evaluation data;

[0117] In an embodiment of the present invention, the obtained sparse non-zero element data is analyzed to determine the information retention ability and compression effect of the feature after compression. The compression potential of the data is evaluated by calculating the sparsity index of the sparse matrix, such as sparsity and the proportion of non-zero elements. Each feature in the feature subset is analyzed one by one to calculate its performance under sparse representation. Sparsity is defined as the ratio of the number of non-zero elements to the total number of elements. For example, if the sparsity of the feature is included in the feature subset, the sparsity threshold is set to 0.3, and the features below the threshold will be marked as inefficient features. Subsequently, the sparsity indicators of all features are summarized to generate compressed sparsity evaluation data. This evaluation data includes the sparsity of each feature, the number of non-zero elements and its proportion in the overall feature set. In order to ensure the effectiveness of the evaluation results, a graphical visualization tool is used to display the sparsity distribution, which intuitively reflects the compression potential of each feature. For example, the operating parameter example can be set to a feature subset size of 500 elements, of which the features with a sparsity of more than 0.3 account for 60%. Through such an evaluation, it is clear which features can effectively support subsequent compression operations. The final compression sparsity evaluation data will be used to guide the decision-making of subsequent steps, thereby improving the effectiveness of the overall data compression process.

[0118] Step S33: designing a compressed sparse matrix according to the compressed sparsity evaluation data and the corresponding node positions;

[0119] In an embodiment of the present invention, a compressed sparse matrix is ​​designed based on compressed sparsity evaluation data and corresponding node positions. An efficient sparse matrix is ​​constructed by combining the non-zero elements in the feature subset with their corresponding sparsity evaluation data. Using a coordinate compression method, the non-zero elements in the matrix and their corresponding row and column indexes are recorded as a triple (row index, column index, value) to reduce storage requirements. This design process ensures that the compressed sparse matrix can be quickly accessed and processed to achieve efficient data compression processing.

[0120] Step S34: performing adaptive sparse compression processing on the effective integrated data to be compressed by compressing the sparse matrix to obtain compressed data.

[0121] In an embodiment of the present invention, adaptive sparse compression processing is performed on the effective integrated data to be compressed by compressing the sparse matrix. An adaptive compression algorithm, such as dynamic programming or a sparse coding algorithm based on deep learning, is used to ensure that the structure and characteristics of the data are effectively modeled during the compression process. The input is a compressed sparse matrix and its corresponding feature subset. The goal is to generate compressed data. The compression coefficient is designed by compressing the sparse matrix, that is, a lower compression coefficient is set for a data with a higher amount of information and a higher information correlation. By adjusting the sparsity parameters and the compression strategy, the compression efficiency and decompression quality are optimized. The compressed data finally output will maintain the integrity and availability of the data while meeting the storage requirements, ensuring the effectiveness of subsequent data analysis and application.

[0122] Further, step S4 includes the following steps:

[0123] Step S41: performing storage demand distribution feature analysis according to the compressed sparse matrix of the compressed data to generate storage demand distribution feature data;

[0124] In an embodiment of the present invention, a compressed sparse matrix is ​​analyzed in detail to determine the space required for each feature during storage. The distribution characteristics of the storage requirements are identified by calculating the number of non-zero elements of each feature in the sparse matrix and its proportion in the overall matrix. The sparse matrix is ​​operated using a data analysis tool (such as Pandas or NumPy) to calculate the storage space requirements for each feature. A threshold is set, for example, features with a storage space of less than 100KB for each feature will be regarded as the first demand feature, and features between 100KB-100MB will be regarded as the second demand feature, etc. to design hierarchical demand distribution characteristics. Finally, the generated storage demand distribution feature data will include the storage requirements of each feature and its distribution map in the overall feature set, which will provide a basis for subsequent steps.

[0125] Step S42: performing cluster analysis based on the storage demand distribution characteristic data to generate cluster storage demand distribution characteristic data;

[0126] In an embodiment of the present invention, cluster analysis is performed based on the generated storage demand distribution characteristic data to generate cluster storage demand distribution characteristic data, and a suitable clustering algorithm, such as K-means, is selected to process the storage demand data of the feature as input. The K value is set to 5, that is, the feature is divided into 5 different clusters. During the clustering implementation process, the storage demand data is first standardized to eliminate the magnitude differences of different feature requirements. Then, the standardized data is input into the K-means algorithm for cluster analysis. The center point of each cluster represents the average storage requirement of a group of features. By analyzing the clustering results, cluster storage demand distribution characteristic data is generated. These data contain information such as the characteristics of each cluster, the corresponding storage demand characteristics, and the number of samples in each cluster, providing a basis for subsequent configuration demand analysis.

[0127] Step S43: performing data block configuration requirement analysis according to the cluster storage requirement distribution characteristic data to generate data block configuration requirement data;

[0128] In an embodiment of the present invention, the characteristics of each cluster are analyzed in detail using the cluster storage demand distribution characteristic data. Assume that the clustering results are divided into 5 groups, each group represents a different storage demand mode. For each cluster, calculate the average value, standard deviation, maximum value and minimum value of its storage demand. These statistical data help to understand the characteristic distribution of each cluster. Then, based on these statistical characteristics, set the size of the data block. For example, if the average storage demand of each cluster is 100MB and the standard deviation is 20MB, the data block size can be set to 120MB to ensure sufficient space to meet most data storage requirements. At the same time, considering the storage efficiency and access efficiency, set a reasonable number of blocks. Assuming that the total data volume is 500MB, it can be determined that the number of blocks to be used is 5 (500MB ÷ 120MB). Specific configuration requirements are formulated for each cluster, including the size of each data block, the storage location (for example, distributed on different storage devices) and its access mode (such as sequential access or random access). For example, for clusters with higher storage requirements, data blocks can be configured on fast-access SSDs, while for clusters with lower storage requirements, traditional HDDs can be selected. The generated data block configuration requirement data contains information such as the block size, number of blocks, storage location, and access mode of each cluster.

[0129] Step S44: block-processing the compressed data according to the data block configuration requirement data to generate block compressed data;

[0130] In the embodiment of the present invention, the compressed data is divided according to the previously set data block size. For example, for each 256KB data block, features are extracted from the compressed data and the features are packaged. Each block should contain the index information of the feature and its corresponding storage location to ensure the efficiency of subsequent data access and generate block compressed data.

[0131] Step S45: extracting the data access pattern of the compressed data block according to the block compressed data, and generating a data access pattern;

[0132] In an embodiment of the present invention, access logs of block compressed data are analyzed, and these logs record the access to compressed data blocks within a specific time period. Access logs usually contain a timestamp, a data block identifier, an access frequency, and an access type (e.g., read, write). By collating and summarizing these logs, the access frequency of each data block is counted. Statistical analysis methods, such as frequency analysis and time series analysis, are used to identify the access pattern of each data block. For data blocks with a higher access frequency, they are marked as hot blocks, and these data blocks may require faster storage media. For data blocks with a lower access frequency, they are marked as cold blocks, and these blocks can be stored on slower media to generate data access patterns, including the access frequency of each data block, the distribution of access types (such as the ratio of sequential access to random access), and the trend of time changes.

[0133] Step S46: performing compressed data block storage optimization processing on the block compressed data based on the data access mode to generate optimized storage compressed data blocks.

[0134] In an embodiment of the present invention, in the data access pattern report, hot blocks and cold blocks are identified. For hot blocks, they are moved to higher-performance storage devices (such as SSDs) to improve data access speed; while cold blocks can be migrated to low-cost storage devices (such as HDDs) to reduce storage costs. Data blocks are rearranged according to the characteristics of the access pattern. For example, if it is found that certain data blocks are often accessed sequentially, these data blocks can be stored in sequence to improve access efficiency. In order to optimize storage efficiency, consider merging or splitting data blocks. For example, for "cold blocks" with smaller storage requirements, multiple small blocks are merged into one large block to reduce the overhead of storing metadata. At the same time, ensure that access efficiency is maintained after merging. Generate a report on optimized storage of compressed data blocks, including the new storage location of each data block, storage device type, block size and its access mode. Through such optimization processing, data access speed can be significantly improved, storage costs can be reduced, and the overall performance of the system can be enhanced.

[0135] Further, step S46 includes the following steps:

[0136] Step S461: performing access pattern time-frequency feature analysis according to the data access pattern to generate access pattern time-frequency feature data;

[0137] In an embodiment of the present invention, the access timestamp and access frequency of each data block are extracted from the data access pattern report. By arranging these timestamps in chronological order, the time distribution characteristics of the access behavior can be identified. The access data is subjected to frequency feature extraction using time-frequency analysis methods, such as short-time Fourier transform or wavelet transform. These methods can effectively capture the frequency characteristics of access patterns that change over time. By calculating the access frequency in each time period, the access peak and trough periods can be obtained. The extracted time-frequency features are organized into structured data to record the access frequency of each data block in a specific time period. The access pattern time-frequency feature data finally generated includes the access peak time, the access frequency fluctuation range and its corresponding data block identifier.

[0138] Step S462: performing access pattern collaborative correlation feature analysis according to the access pattern time-frequency feature data to generate access pattern collaborative correlation feature data;

[0139] In an embodiment of the present invention, the access pattern time-frequency feature data is analyzed, and the correlation between the data blocks is calculated to identify which data blocks are often accessed at the same time. This can be achieved by calculating the Pearson correlation coefficient or the cosine similarity. An access association matrix is ​​constructed, in which the rows and columns of the matrix are data blocks, and each element in the matrix represents the frequency of collaborative access between corresponding data blocks. By performing cluster analysis on this matrix, the groups of collaborative access are identified. The features of collaborative access are extracted and access pattern collaborative association feature data is generated, including the main data block groups for collaborative access, the number of accesses, and related time period information. These data provide a basis for subsequent storage optimization.

[0140] Step S463: performing collaborative association storage processing on the block compressed data according to the access mode collaborative association feature data to generate a storage compressed data block;

[0141] In an embodiment of the present invention, based on the access pattern collaborative association feature data, it is determined which data blocks have a strong association. These data blocks are often requested together when accessed, so it can be considered to store them in the same physical location to reduce access latency. For collaboratively accessed data blocks, physical layout optimization is performed to ensure that highly associated data blocks are as close as possible by rearranging the storage structure. For example, blocks with a high frequency of collaborative access are merged into a logical group and stored adjacently on the storage medium. The optimized data blocks are recompressed and compressed data blocks are generated for storage. The new storage layout will significantly improve access performance in high-concurrency access scenarios.

[0142] Step S464: performing storage queue scheduling resource analysis and processing according to the access pattern time-frequency characteristic data corresponding to the storage compressed data block, and generating storage queue scheduling resource data;

[0143] In an embodiment of the present invention, the access mode time-frequency feature data of each data block in the storage queue is collected, and the access timing and access frequency of each data block are analyzed. These data will be used to analyze the load of the current storage queue. Based on the analysis results, the scheduling strategy of the storage queue is evaluated. A scheduling algorithm (such as polling scheduling, shortest job first, etc.) can be used to determine the access order of data blocks. By comparing the performance of different scheduling strategies, corresponding storage queue scheduling resource data is generated. The storage queue scheduling resource data formed will contain information such as the scheduling priority, expected access delay and current queue length of each data block. These data provide a basis for subsequent storage queue optimization.

[0144] Step S465: Perform storage queue optimization processing on the storage compressed data block through the storage queue scheduling resource data to generate an optimized storage compressed data block.

[0145] In an embodiment of the present invention, the current storage queue is analyzed based on the storage queue scheduling resource data. The access order of each data block in the storage queue is adjusted according to the priority to improve the data access efficiency. The optimized scheduling strategy is applied to reschedule the data blocks to ensure that the data blocks with high frequency access are accessed first. By reducing the waiting time of high frequency data blocks, the overall access delay is reduced. After the optimization processing of the storage queue is completed, the optimized storage compressed data blocks are generated. These optimized data blocks will show higher performance when accessed, meeting the user's needs for efficient storage and fast access. The formed optimized storage compressed data blocks will provide support for subsequent data processing and storage management.

[0146] The beneficial effect of the present invention is that, by systematically acquiring and parsing the data to be compressed, the integrity and accuracy of the data are ensured, a solid foundation is provided for subsequent processing, data preparation time is significantly reduced, and overall processing efficiency is improved. The heterogeneous data type analysis of the parsed data can identify and classify different types of data, which helps to better understand the data characteristics, so as to adopt a more appropriate strategy during compression and improve the compression effect. Through the designed anomaly detection engine, problems in the compression process can be identified in real time and feedback can be given in a timely manner, which not only improves the reliability of data processing, but also provides an improvement basis for subsequent processing, reduces potential risks, clarifies which data meets the compression requirements, thereby ensuring that only verified data is compressed, enhances the standardization and security of data processing, and effectively improves the overall data quality. By parsing heterogeneous data types for feature analysis and extraction, key information in the data can be effectively identified and utilized, thereby providing a more accurate basis for subsequent compression, improving compression efficiency, and constructing a heterogeneous data feature analysis tree model, which can dynamically map the relationship between data and feature subsets, so that data characteristics can be processed more accurately during the compression process. The application of this model enhances the understanding and analysis of data characteristics, promotes the effective measurement of information volume and mutual information correlation analysis, and further optimizes the compression effect. By designing a recursive sparse coefficient evaluation objective function, the compression strategy can be continuously optimized, making the compression process more adaptable and flexible. The application of deep learning technology not only improves the efficiency of data compression and storage, but also provides innovative solutions for processing complex and heterogeneous data, and enhances the intelligence and automation level of data management. The recursive compression sparse coefficient evaluation objective function is used to analyze the sparse non-zero elements of the feature subset of the effective integrated data to be compressed, which realizes the in-depth mining and evaluation of data sparsity. It can not only accurately identify the key information in the data to be compressed, but also provide a scientific basis for the compression strategy, thereby optimizing the compression efficiency. By designing a compressed sparse matrix, data can be stored in a more flexible and efficient way, significantly reducing storage requirements and improving data reading and processing speed. In addition, the application of adaptive sparse compression processing enables data to minimize redundant information while maintaining important features, further improving the compression ratio, showing high adaptability and intelligence when processing complex data, and improving the efficiency and security of data management. According to the compressed sparse matrix of compressed data, the storage demand distribution characteristics are analyzed to achieve an accurate evaluation of storage requirements. This characteristic data lays the foundation for subsequent cluster analysis, thereby generating more detailed cluster storage demand distribution characteristic data and improving the intelligent level of storage configuration. After further data block configuration demand analysis, the configuration of data blocks is optimized according to different storage requirements to ensure efficient use of storage resources.In the block processing process, the generated block compressed data can clarify the user's behavioral characteristics in data storage and access by extracting the data access pattern. This analysis of data access patterns not only provides an important basis for the storage optimization of compressed data blocks, but also helps to identify the best storage strategy and improve the efficiency of data access. Through the time-frequency feature analysis and collaborative correlation feature analysis of the access pattern, the data access behavior can be understood more deeply, and then the collaborative correlation storage processing is implemented to improve the performance of the storage compressed data blocks. Through the analysis and optimization processing of the storage queue scheduling resources, the efficient scheduling and management of the storage compressed data blocks in actual use are ensured, and excellent effects are shown in optimizing data storage, improving access efficiency and reducing storage costs. Therefore, the data compression and block storage processing method based on deep learning of the present invention can learn data characteristics through a deep learning algorithm to dynamically adjust the compression processing, etc., to ensure that data compression is targeted at redundant information and retains important feature information, and has targeted data block storage management, so that storage resources are efficiently utilized and the scheduling efficiency is high.

[0147] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0148] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A data compression and block storage processing method based on deep learning, characterized in that: The following steps are involved: Step S1: obtaining integrated data to be compressed; performing effective parsing data analysis and processing according to the integrated data to be compressed to generate effective integrated parsing data to be compressed; Wherein, step S1 comprises the following steps: Step S11: obtaining integrated data to be compressed; Step S12: performing data analysis on the integrated data to be compressed to generate integrated analysis data to be compressed; Step S13: performing heterogeneous data type analysis on the compressed integrated parsed data to generate parsed heterogeneous data types; Step S14: performing compression demand anomaly detection feedback processing on the integrated parsed data to be compressed based on the parsed heterogeneous data types, so as to obtain effective integrated parsed data to be compressed; Step S2: construct a heterogeneous data feature analysis tree model; perform feature subset extraction processing on the effective integrated data to be compressed according to the heterogeneous data feature analysis tree model to generate feature data to be compressed; design a recursive sparse coefficient evaluation objective function based on the feature data to be compressed; Wherein, step S2 comprises the following steps: Step S21: performing heterogeneous data feature analysis rule analysis according to parsing heterogeneous data types to design heterogeneous data feature analysis rule decision; Step S22: extracting heterogeneous data feature analysis factors from the effective integrated data to be compressed through heterogeneous data feature analysis rule decision, and generating heterogeneous data feature analysis factors; Step S23: establishing a mapping relationship between heterogeneous data and feature subsets based on a preset decision tree model and heterogeneous data feature analysis factors to construct a heterogeneous data feature analysis tree model; Step S24: performing feature subset extraction processing on the effective integrated data to be compressed according to the heterogeneous data feature analysis tree model to generate feature data to be compressed; Step S25: performing feature subset information quantity measurement processing according to the feature data to be compressed to generate feature subset information quantity data; Step S26: performing mutual information correlation analysis of feature subsets according to the feature data to be compressed, and generating feature subset mutual information correlation data; Wherein, step S26 comprises the following steps: Step S261: performing joint probability distribution and marginal probability distribution analysis of feature subsets according to the feature data to be compressed, and generating feature subset joint probability distribution data and feature subset marginal probability distribution data respectively; Step S262: performing feature subset mutual information correlation analysis according to feature subset joint probability distribution data and feature subset edge probability distribution data to generate feature subset mutual information correlation data; Step S27: designing a recursive sparse coefficient evaluation objective function based on the feature subset information volume data and the feature subset mutual information correlation data; Wherein, step S27 includes the following steps: Step S271: taking the feature subset information volume data and the feature subset mutual information correlation data as feature subset negative correlation sparse evaluation data, and transmitting the feature subset negative correlation sparse evaluation data to a preset deep learning algorithm for feature subset sparse coefficient evaluation learning processing, so as to construct a sparse coefficient evaluation objective function; Step S272: performing recursive optimization processing on the objective function according to the sparse coefficient evaluation objective function to generate a recursive sparse coefficient evaluation objective function; Step S3: performing adaptive sparse compression processing on the effective integrated data to be compressed based on the recursive sparse coefficient evaluation objective function to obtain compressed data; Wherein, step S3 comprises the following steps: Step S31: performing feature subset sparse non-zero element analysis on the feature data to be compressed corresponding to the valid integrated data to be compressed based on the recursive sparse coefficient evaluation objective function to generate feature subset sparse non-zero element data; Step S32: performing compression sparsity evaluation processing according to the sparse non-zero element data of the feature subset to generate compression sparsity evaluation data; Step S33: designing a compressed sparse matrix according to the compressed sparsity evaluation data and the corresponding node positions; Step S34: performing adaptive sparse compression processing on the effective integrated data to be compressed by compressing the sparse matrix to obtain compressed data; Step S4: block-processing the compressed data to generate block compressed data; extracting the data access pattern of the compressed data block according to the block compressed data to generate the data access pattern; performing compressed data block storage optimization processing on the block compressed data based on the data access pattern to generate optimized storage compressed data blocks; Wherein, step S4 comprises the following steps: Step S41: performing storage demand distribution feature analysis according to the compressed sparse matrix of the compressed data to generate storage demand distribution feature data; Step S42: performing cluster analysis based on the storage demand distribution characteristic data to generate cluster storage demand distribution characteristic data; Step S43: performing data block configuration requirement analysis according to the cluster storage requirement distribution characteristic data to generate data block configuration requirement data; Step S44: block-processing the compressed data according to the data block configuration requirement data to generate block compressed data; Step S45: extracting the data access pattern of the compressed data block according to the block compressed data, and generating a data access pattern; Step S46: performing compressed data block storage optimization processing on the block compressed data based on the data access mode to generate optimized storage compressed data blocks; Wherein, step S46 includes the following steps: Step S461: performing access pattern time-frequency feature analysis according to the data access pattern to generate access pattern time-frequency feature data; Step S462: performing access pattern collaborative correlation feature analysis according to the access pattern time-frequency feature data to generate access pattern collaborative correlation feature data; Step S463: performing collaborative association storage processing on the block compressed data according to the access mode collaborative association feature data to generate a storage compressed data block; Step S464: performing storage queue scheduling resource analysis and processing according to the access pattern time-frequency characteristic data corresponding to the storage compressed data block, and generating storage queue scheduling resource data; Step S465: Perform storage queue optimization processing on the storage compressed data block through the storage queue scheduling resource data to generate an optimized storage compressed data block.

2. The data compression and block storage processing method based on deep learning according to claim 1 is characterized in that: Step S14 includes the following steps: Step S141: performing analysis on compression requirements of heterogeneous data according to the parsed heterogeneous data types to obtain a decision on compression requirements of heterogeneous data; Step S142: Designing a compression demand anomaly detection engine by analyzing the compression demand decisions of heterogeneous data; Step S143: The compression demand anomaly detection engine performs compression demand anomaly detection feedback processing on the integrated parsed data to be compressed, so as to obtain valid integrated parsed data to be compressed.

3. The data compression and block storage processing method based on deep learning according to claim 2 is characterized in that: Step S143 includes the following steps: When the compression requirement anomaly detection engine identifies that the integrated parsing data to be compressed does not meet the compression requirement decision for parsing heterogeneous data, the corresponding integrated parsing data to be compressed is fed back to the terminal to execute the compression anomaly feedback job; or, when the integrated data anomaly detection engine identifies that the integrated parsing data to be compressed meets the compression requirement decision for parsing heterogeneous data, the integrated parsing data to be compressed is marked as valid integrated parsing data to be compressed.

Images