Efficient intelligent data compression and feature extraction method and system

Abstract

The invention discloses an efficient intelligent data compression and feature extraction method and system. The method comprises the following steps: S1, data preprocessing: performing data cleaning, denoising and standardization processing on to-be-processed data; s2, data compression and storage: determining a compression algorithm according to the type of the data, and uploading a compressed data file to a database; s3, feature extraction: deploying a machine learning model, and extracting representative features from the compressed data; and S4, data post-processing: further processing and optimizing the extracted features. According to the method, the data compression technology and the feature extraction technology are combined, the most representative information is extracted from the original data, meanwhile, the data storage and transmission cost is reduced, key features in the data are automatically recognized and reserved through an intelligent algorithm, and therefore efficient data compression and feature extraction are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of data science and technology, and more specifically, particularly relates to an efficient and intelligent data compression and feature extraction method. At the same time, the present invention also relates to an efficient and intelligent data compression and feature extraction system. Background Art

[0002] With the rapid development of artificial intelligence technology, efficient data processing and feature extraction have become the core requirements in various application scenarios. They are mainly used in three important application fields: image recognition, speech recognition, and natural language processing. They all rely on efficient data processing and feature extraction methods. Image data is difficult to deal with by traditional methods due to its large amount of information and high dimensionality. However, deep learning technologies such as convolutional neural networks (CNN) can efficiently extract image features and achieve accurate recognition, such as face recognition and license plate recognition. Speech recognition technology requires converting speech signals into text. Deep learning models such as long short-term memory networks (LSTM) and transformers can automatically learn speech features and accurately convert text. They are widely used in scenarios such as smartphone assistants and smart homes. Natural language processing requires efficient processing of text data, and intelligent word segmentation is achieved through machine learning models. It provides structured data support for subsequent semantic analysis, sentiment analysis, and topic modeling, and plays an important role in applications such as natural language generation and machine translation.

[0003] However, with the development of information technology, the amount of data generated by all walks of life is increasing dramatically, and the data contains rich information and value, but the huge amount of data will lead to problems such as increased storage costs, longer transmission time and reduced processing speed. Therefore, we propose an efficient and intelligent data compression and feature extraction method and system. Summary of the invention

[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide an efficient and intelligent data compression and feature extraction method and system. The present invention combines data compression technology and feature extraction technology to extract the most representative information from the original data, while reducing the storage and transmission costs of the data, and automatically identifies and retains the key features in the data through intelligent algorithms, thereby achieving efficient data compression and feature extraction.

[0005] To achieve the above object, the present invention provides the following technical solution: an efficient and intelligent data compression and feature extraction method and system, comprising the following steps:

[0006] S1, data preprocessing: data cleaning, denoising and standardization of the data to be processed;

[0007] S2, data compression and storage: Determine the compression algorithm based on the data type and upload the compressed data file to the database;

[0008] S3, feature extraction: deploy machine learning models to extract representative features on the compressed data;

[0009] S4, data post-processing: further processing and optimization of the extracted features.

[0010] Optionally, the data to be processed is multi-channel data, and the data sources include databases, sensors, IoT devices, web crawlers, user input, etc.;

[0011] The data to be processed includes structured data and unstructured data;

[0012] The data cleaning includes: missing value processing, outlier value processing and duplicate value processing;

[0013] The denoising is used to filter and smooth the image, audio and other data;

[0014] The standardization process includes data normalization and data standardization, which is used to eliminate the dimensional differences between different features.

[0015] Optionally, in step S2, the compression algorithm is determined as follows:

[0016] The compression algorithm for the structured data is a lossless compression algorithm, and the compression algorithm for the unstructured data is a lossy compression algorithm;

[0017] The lossless compression algorithm includes Huffman coding, arithmetic coding, LZW algorithm, LZ77 algorithm, etc.

[0018] The lossy compression algorithm includes a JPEG compression algorithm, an MP3 compression algorithm, and a video encoding algorithm.

[0019] Optionally, the specific process of data compression is:

[0020] S21, deploying both the lossless compression algorithm and the lossy compression algorithm in the compression module;

[0021] S22, determining the type of data through the data type identification submodule in the compression module;

[0022] S23, based on the preset judgment rules and according to the type of data, calling the corresponding compression algorithm;

[0023] S24, compressing the data to be processed;

[0024] S25, saving the compressed data to a database and inputting it into a feature extraction module.

[0025] Optionally, the preset determination rule is specifically:

[0026] The data type is a text file. If the character probability distribution is uneven, Huffman coding is called. If the symbol probability distribution of the data is known or can be estimated, arithmetic coding is called.

[0027] The data types are text files and image files. If there are a large number of repeated characters or character strings in the data, the LZW algorithm is called. If there are continuous data streams such as network data in the data, the LZ77 algorithm is called.

[0028] When the data type is a still grayscale or color image, the JPEG compression algorithm is called;

[0029] When the data type is audio data, the MP3 compression algorithm is called;

[0030] When the data type is a video file, a video encoding algorithm is called.

[0031] Optionally, the process of step S3 is specifically as follows:

[0032] S31, load the machine learning model using the deep learning framework;

[0033] S32, using a compressed file processing library to parse the format of the compressed file, and obtain information of the file header and each data block therein;

[0034] S33, selecting a data block that may contain useful features according to the format and content of the compressed file;

[0035] S34, partially decompressing the selected data block to obtain sufficient data for feature extraction;

[0036] S35, extracting features from the decompressed data through a convolutional neural network;

[0037] S36, further extracting representative features from the extracted features based on a preset importance evaluation algorithm;

[0038] S37, uploading the extracted representative features to the database and performing visualization processing.

[0039] Optionally, the importance evaluation algorithm is specifically:

[0040] Average Impurity Reduction: By calculating the average value of each feature's reduction in impurity across all decision trees, the contribution of the feature to the model performance is quantified. The larger the impurity reduction value, the more important the feature.

[0041] Average precision decreases: By disrupting the order of features and retraining the model, observe the changes in prediction accuracy. The more the accuracy decreases, the greater the impact of the feature on the prediction result.

[0042] Gini index: Calculate the Gini index of each feature in the data set, and select the feature with the smallest Gini index as the best feature for splitting the decision tree node. The smaller the Gini index, the greater the purity improvement when using the feature for splitting, and therefore the higher the importance of the feature.

[0043] SHAP value evaluation: Calculate the SHAP value of each feature based on the principles of game theory to indicate the contribution of each feature to the model prediction results.

[0044] Optionally, the algorithm formula of the visualization processing is specifically:

[0045] Similar calculations in high-dimensional space:

[0046]

[0047] Similarity calculation in low-dimensional space:

[0048]

[0049] Where: x i and x j are two high-dimensional data points;

[0050] σ i For data point x i Gaussian kernel width;

[0051] ||x i -x j || 2 Represents data point x i and x j The square of the Euclidean distance between them;

[0052] pij is the conditional probability, indicating that at a given data point x i In the case of data point x i and x j Similar probability.

[0053] An efficient and intelligent data compression and feature extraction system, the system is used to implement the above method, including: a server, a database, a user terminal, a data processing module, a compression module, a feature extraction module, and a data post-processing module;

[0054] The server is responsible for processing requests from user terminals, managing data in the database, and coordinating interactions between various modules;

[0055] The database is used to store and manage data in the system, including original data, compressed data, extracted features, etc.

[0056] The user terminal is used for user interaction with the system, including computers, mobile phones, tablets and other devices;

[0057] The data processing module is responsible for preprocessing the data to be processed, including steps such as data cleaning, denoising and standardization.

[0058] The compression module selects a suitable compression algorithm according to the type of data and compresses the pre-processed data.

[0059] The feature extraction module uses a machine learning or deep learning algorithm to extract representative features from the compressed data and uploads them to a database for storage;

[0060] The data post-processing module is used to further process and optimize the extracted features.

[0061] Optionally, the feature extraction module deploys a machine learning model, and the machine learning model includes:

[0062] The input layer is responsible for receiving raw data and converting it into a format that the model can process;

[0063] Each neuron in the fully connected layer is connected to all neurons in the previous layer for linear combination of features;

[0064] The pooling layer is used to reduce the feature dimension and the number of parameters and improve computational efficiency;

[0065] Activation function: introduces nonlinear factors to help the model learn complex feature representations.

[0066] The output layer is responsible for producing the final output of the model, which corresponds to the category label in the classification task or the continuous value in the regression task.

[0067] The loss function is used to measure the difference between the model output and the actual label.

[0068] Technical effects and advantages of the present invention: The present invention integrates two technologies: data compression and feature extraction. Data compression technology reduces data redundancy, realizes efficient use of storage space and improves transmission speed, which is particularly important for resource-limited environments. Feature extraction technology focuses on mining key features in data, which not only reduces data dimensions, but also improves the accuracy and efficiency of subsequent data analysis and model training. Combining these two technologies not only reduces the burden of storage and transmission, but also accelerates the data processing process, providing strong support for real-time analysis and rapid decision-making. In addition, the extracted key features are used as input for model training to further optimize the performance of the machine learning model, enabling it to more accurately capture data patterns and improve the accuracy of prediction and classification.

[0069] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic flow chart of the method provided by the present invention;

[0071] Figure 2 It is a schematic diagram of a specific process of data compression provided by the present invention;

[0072] Figure 3 It is a schematic diagram of the system execution flow provided by the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0074] like Figures 1 to 3 As shown, an embodiment of the present invention provides an efficient and intelligent data compression and feature extraction method and system, comprising the following steps:

[0075] S1, data preprocessing: data cleaning, denoising and standardization of the data to be processed;

[0076] S2, data compression and storage: Determine the compression algorithm based on the data type and upload the compressed data file to the database;

[0077] S3, feature extraction: deploy machine learning models to extract representative features on the compressed data;

[0078] S4, data post-processing: further processing and optimization of the extracted features.

[0079] Furthermore, the present invention realizes efficient management and utilization of data through a series of steps. The data preprocessing stage ensures the accuracy and consistency of the data, laying a solid foundation for subsequent steps. Data compression and storage effectively reduce storage requirements and speed up data transmission. The feature extraction stage uses machine learning models to accurately extract key information from compressed data, thereby improving data utilization. Finally, data post-processing further optimizes feature quality, making it more suitable for subsequent analysis. Overall, this process improves data processing efficiency, reduces storage and transmission costs, and provides efficient and accurate data support for data analysis and decision-making.

[0080] Specifically, the data to be processed is multi-channel data, and the data sources include databases, sensors, IoT devices, web crawlers, user input, etc.;

[0081] The data to be processed includes structured data and unstructured data;

[0082] The data cleaning includes: missing value processing, outlier value processing and duplicate value processing;

[0083] The denoising is used to filter and smooth the image, audio and other data;

[0084] The standardization process includes data normalization and data standardization, which is used to eliminate the dimensional differences between different features.

[0085] Furthermore, embodiments of the present invention support multi-channel and multi-type data input, including structured and unstructured data from databases, sensors, IoT devices, etc. Through data cleaning, the problems of missing values, outliers, and duplicate values ​​are effectively solved, ensuring the accuracy and reliability of the data. Denoising improves the clarity of image, audio and other data. Standardization eliminates dimensional differences between features and provides a comparable basis for subsequent analysis. Overall, the process improves data quality, provides high-quality input for subsequent data compression, feature extraction, and post-processing, and ensures the accuracy and efficiency of data processing.

[0086] Specifically, in step S2, the compression algorithm is determined as follows:

[0087] The compression algorithm for the structured data is a lossless compression algorithm, and the compression algorithm for the unstructured data is a lossy compression algorithm;

[0088] The lossless compression algorithm includes Huffman coding, arithmetic coding, LZW algorithm, LZ77 algorithm, etc.

[0089] The lossy compression algorithm includes a JPEG compression algorithm, an MP3 compression algorithm, and a video encoding algorithm.

[0090] Further, in step S2, a suitable compression algorithm is used for different types of data. For structured data, a lossless compression algorithm, such as Huffman coding, arithmetic coding, etc., is used to ensure that no information is lost during the compression process of the data, and the integrity and accuracy of the data are maintained. For unstructured data, such as images, audio, etc., a lossy compression algorithm, such as JPEG, MP3, etc., is used to effectively reduce the data volume while maintaining a high data quality to meet the needs of practical applications. This approach of selecting a compression algorithm based on the data type not only improves the data compression efficiency, but also ensures the data quality, providing strong support for subsequent feature extraction and data analysis.

[0091] Specifically, the specific process of data compression is as follows:

[0092] S21, deploying both the lossless compression algorithm and the lossy compression algorithm in the compression module;

[0093] S22, determining the type of data through the data type identification submodule in the compression module;

[0094] S23, based on the preset judgment rules and according to the type of data, calling the corresponding compression algorithm;

[0095] S24, compressing the data to be processed;

[0096] S25, saving the compressed data to a database and inputting it into a feature extraction module.

[0097] Furthermore, the specific process of data compression achieves efficient data processing through intelligent deployment and automatic identification. First, the lossless and lossy compression algorithms are integrated into the compression module for unified management. Then, the data type is automatically determined through the data type identification submodule, avoiding manual intervention and improving processing efficiency. Based on preset rules, the appropriate compression algorithm is automatically called to ensure the accuracy and efficiency of data compression. The compressed data not only reduces the storage space occupied, but also speeds up the subsequent processing speed. Finally, the compressed data is saved to the database and input into the feature extraction module, providing high-quality input for subsequent analysis. Overall, this process improves the intelligence and automation level of data compression.

[0098] Specifically, the preset judgment rule is:

[0099] The data type is a text file. If the character probability distribution is uneven, Huffman coding is called. If the symbol probability distribution of the data is known or can be estimated, arithmetic coding is called.

[0100] The data types are text files and image files. If there are a large number of repeated characters or character strings in the data, the LZW algorithm is called. If there are continuous data streams such as network data in the data, the LZ77 algorithm is called.

[0101] When the data type is a still grayscale or color image, the JPEG compression algorithm is called;

[0102] When the data type is audio data, the MP3 compression algorithm is called;

[0103] When the data type is a video file, a video encoding algorithm is called.

[0104] Furthermore, through carefully designed preset judgment rules, the data compression process can intelligently select the optimal compression algorithm according to the data type and characteristics. For text files, Huffman coding or arithmetic coding is selected according to the character probability distribution to improve the compression efficiency. For data containing repeated characters or strings, the LZW algorithm is more efficient; and for continuous data streams, the LZ77 algorithm is more advantageous. For image, audio and video files, JPEG, MP3 and video encoding algorithms are selected respectively to ensure high-quality compression effects. This intelligent selection mechanism not only improves compression efficiency, but also ensures data quality, laying a solid foundation for subsequent feature extraction and data analysis.

[0105] Specifically, the process of step S3 is as follows:

[0106] S31, load the machine learning model using the deep learning framework;

[0107] S32, using a compressed file processing library to parse the format of the compressed file, and obtain information of the file header and each data block therein;

[0108] S33, selecting a data block that may contain useful features according to the format and content of the compressed file;

[0109] S34, partially decompressing the selected data block to obtain sufficient data for feature extraction;

[0110] S35, extracting features from the decompressed data through a convolutional neural network;

[0111] S36, further extracting representative features from the extracted features based on a preset importance evaluation algorithm;

[0112] S37, uploading the extracted representative features to the database and performing visualization processing.

[0113] Specifically, the importance evaluation algorithm is as follows:

[0114] Average Impurity Reduction: By calculating the average value of each feature's reduction in impurity across all decision trees, the contribution of the feature to the model performance is quantified. The larger the impurity reduction value, the more important the feature.

[0115] Average precision decreases: By disrupting the order of features and retraining the model, observe the changes in prediction accuracy. The more the accuracy decreases, the greater the impact of the feature on the prediction result.

[0116] Gini index: Calculate the Gini index of each feature in the data set, and select the feature with the smallest Gini index as the best feature for splitting the decision tree node. The smaller the Gini index, the greater the purity improvement when using the feature for splitting, and therefore the higher the importance of the feature.

[0117] SHAP value evaluation: Calculate the SHAP value of each feature based on the principles of game theory to indicate the contribution of each feature to the model prediction results.

[0118] Furthermore, these importance assessment algorithms can accurately assess the importance of features from different perspectives, providing strong support for data preprocessing, feature selection, and model optimization. By properly applying these algorithms, the performance and accuracy of the model can be significantly improved, thereby promoting the development of data science and machine learning.

[0119] Specifically, the algorithm formula of the visualization processing is:

[0120] Similar calculations in high-dimensional space:

[0121]

[0122] Similarity calculation in low-dimensional space:

[0123]

[0124] Where: x i and x j are two high-dimensional data points;

[0125] σ i For data point x i Gaussian kernel width;

[0126] ||x i -x j || 2 Represents data point x i and x j The square of the Euclidean distance between them;

[0127] pij is the conditional probability, indicating that at a given data point x i In the case of data point x i and x j Similar probabilities;

[0128] An efficient and intelligent data compression and feature extraction system, the system is used to implement the above method, including: a server, a database, a user terminal, a data processing module, a compression module, a feature extraction module, and a data post-processing module;

[0129] The server is responsible for processing requests from user terminals, managing data in the database, and coordinating interactions between various modules;

[0130] The database is used to store and manage data in the system, including original data, compressed data, extracted features, etc.

[0131] The user terminal is used for user interaction with the system, including computers, mobile phones, tablets and other devices;

[0132] The data processing module is responsible for preprocessing the data to be processed, including steps such as data cleaning, denoising and standardization.

[0133] The compression module selects a suitable compression algorithm according to the type of data and compresses the pre-processed data.

[0134] The feature extraction module uses a machine learning or deep learning algorithm to extract representative features from the compressed data and uploads them to a database for storage;

[0135] The data post-processing module is used to further process and optimize the extracted features.

[0136] Furthermore, the system realizes efficient data processing and management through the collaborative work of servers, databases, user terminals and multiple data processing modules. The server serves as the hub, processing requests, managing data, coordinating module interactions, and ensuring the stable operation of the system. The database safely stores various types of data, providing solid data support for the system. The user terminal facilitates user interaction with the system and improves user experience. The data processing module, compression module and feature extraction module realize data preprocessing, compression and feature extraction, improving the efficiency and accuracy of data processing. The data post-processing module further optimizes the feature quality and provides strong support for subsequent analysis.

[0137] Specifically, the feature extraction module deploys a machine learning model, and the machine learning model includes:

[0138] The input layer is responsible for receiving raw data and converting it into a format that the model can process;

[0139] Each neuron in the fully connected layer is connected to all neurons in the previous layer for linear combination of features;

[0140] The pooling layer is used to reduce the feature dimension and the number of parameters and improve computational efficiency;

[0141] Activation function: introduces nonlinear factors to help the model learn complex feature representations.

[0142] The output layer is responsible for producing the final output of the model, which corresponds to the category label in the classification task or the continuous value in the regression task.

[0143] The loss function is used to measure the difference between the model output and the actual label.

[0144] In summary, the present invention combines the two technologies of data compression and feature extraction, aiming to extract the most representative information from massive raw data while significantly reducing the cost of data storage and transmission. Data compression technology achieves efficient use of storage space and significantly improves transmission speed by reducing data redundancy, which is particularly important for environments with limited resources. Feature extraction technology focuses on mining key features in data, which not only reduces data dimensionality, but also improves the accuracy and efficiency of subsequent data analysis and model training.

[0145] Combining these two technologies, the present invention can achieve efficient data compression while retaining the core value of the data. This not only reduces the burden of storage and transmission, but also accelerates the data processing process, providing strong support for real-time analysis and rapid decision-making. In addition, the extracted key features are used as input for model training, further optimizing the performance of the machine learning model, enabling it to more accurately capture data patterns and improve the accuracy of prediction and classification.

[0146] In general, the innovation of this invention lies in the realization of efficient data management and intelligent analysis through the dual effects of data compression and feature extraction. This not only optimizes the data processing process and reduces costs, but also provides strong technical support for data-driven decision-making and promotes the continuous advancement of data processing and machine learning technologies.

[0147] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient and intelligent data compression and feature extraction method, characterized in that: The steps include: S1, data preprocessing: data cleaning, denoising and standardization of the data to be processed; S2, data compression and storage: Determine the compression algorithm based on the data type and upload the compressed data file to the database; S3, feature extraction: deploy machine learning models to extract representative features on the compressed data; S4, data post-processing: further processing and optimization of the extracted features.

2. The method of efficient and intelligent data compression and feature extraction according to claim 1, characterized in that: The data to be processed is multi-channel data, and the data sources include databases, sensors, IoT devices, web crawlers, user input, etc.; The data to be processed includes structured data and unstructured data; The data cleaning includes: missing value processing, outlier value processing and duplicate value processing; The denoising is used to filter and smooth the image, audio and other data; The standardization process includes data normalization and data standardization, which is used to eliminate the dimensional differences between different features.

3. The method of efficient and intelligent data compression and feature extraction according to claim 1, characterized in that: In step S2, the compression algorithm is determined as follows: The compression algorithm for the structured data is a lossless compression algorithm, and the compression algorithm for the unstructured data is a lossy compression algorithm; The lossless compression algorithm includes Huffman coding, arithmetic coding, LZW algorithm, LZ77 algorithm, etc. The lossy compression algorithm includes a JPEG compression algorithm, an MP3 compression algorithm, and a video encoding algorithm.

4. The method of efficient and intelligent data compression and feature extraction according to claim 3, characterized in that: The specific process of data compression is as follows: S21, deploying both the lossless compression algorithm and the lossy compression algorithm in the compression module; S22, determining the type of data through the data type identification submodule in the compression module; S23, based on the preset judgment rules and according to the type of data, calling the corresponding compression algorithm; S24, compressing the data to be processed; S25, saving the compressed data to a database and inputting it into a feature extraction module.

5. The method of efficient and intelligent data compression and feature extraction according to claim 4, characterized in that: The preset judgment rules are specifically: The data type is a text file. If the character probability distribution is uneven, Huffman coding is called. If the symbol probability distribution of the data is known or can be estimated, arithmetic coding is called. The data types are text files and image files. If there are a large number of repeated characters or character strings in the data, the LZW algorithm is called. If there are continuous data streams such as network data in the data, the LZ77 algorithm is called. When the data type is a still grayscale or color image, the JPEG compression algorithm is called; When the data type is audio data, the MP3 compression algorithm is called; When the data type is a video file, a video encoding algorithm is called.

6. The efficient and intelligent data compression and feature extraction method according to claim 1, characterized in that: The specific process of step S3 is as follows: S31, load the machine learning model using the deep learning framework; S32, using a compressed file processing library to parse the format of the compressed file, and obtain information of the file header and each data block therein; S33, selecting a data block that may contain useful features according to the format and content of the compressed file; S34, partially decompressing the selected data block to obtain sufficient data for feature extraction; S35, extracting features from the decompressed data through a convolutional neural network; S36, further extracting representative features from the extracted features based on a preset importance evaluation algorithm; S37, uploading the extracted representative features to the database and performing visualization processing.

7. The efficient and intelligent data compression and feature extraction method according to claim 6, characterized in that: The importance evaluation algorithm is specifically: Average Impurity Reduction: By calculating the average value of each feature's reduction in impurity across all decision trees, the contribution of the feature to the model performance is quantified. The larger the impurity reduction value, the more important the feature. Average precision decreases: By disrupting the order of features and retraining the model, observe the changes in prediction accuracy. The more the accuracy decreases, the greater the impact of the feature on the prediction result. Gini index: Calculate the Gini index of each feature in the data set, and select the feature with the smallest Gini index as the best feature for splitting the decision tree node. The smaller the Gini index, the greater the purity improvement when using the feature for splitting, and therefore the higher the importance of the feature. SHAP value evaluation: Calculate the SHAP value of each feature based on the principles of game theory to indicate the contribution of each feature to the model prediction results.

8. The efficient and intelligent data compression and feature extraction method according to claim 6, characterized in that: The algorithm formula of the visualization processing is specifically: Similar calculations in high-dimensional space: Similarity calculation in low-dimensional space: Where: x i and x j are two high-dimensional data points; σ i For data point x i Gaussian kernel width; ||x i -x j || 2 Represents data point x i and x j The square of the Euclidean distance between them; pij is the conditional probability, indicating that at a given data point x i In the case of data point x i and x j Similar probability.

9. An efficient and intelligent data compression and feature extraction system, characterized in that: The system is used to implement the method described in any one of claims 1 to 8, comprising: a server, a database, a user terminal, a data processing module, a compression module, a feature extraction module, and a data post-processing module; The server is responsible for processing requests from user terminals, managing data in the database, and coordinating interactions between various modules; The database is used to store and manage data in the system, including original data, compressed data, extracted features, etc. The user terminal is used for user interaction with the system, including computers, mobile phones, tablets and other devices; The data processing module is responsible for preprocessing the data to be processed, including steps such as data cleaning, denoising and standardization. The compression module selects a suitable compression algorithm according to the type of data and compresses the pre-processed data. The feature extraction module uses a machine learning or deep learning algorithm to extract representative features from the compressed data and uploads them to a database for storage; The data post-processing module is used to further process and optimize the extracted features.

10. The efficient and intelligent data compression and feature extraction system according to claim 9, characterized in that: The feature extraction module deploys a machine learning model, and the machine learning model includes: The input layer is responsible for receiving raw data and converting it into a format that the model can process; Each neuron in the fully connected layer is connected to all neurons in the previous layer for linear combination of features; The pooling layer is used to reduce the feature dimension and the number of parameters and improve computational efficiency; Activation function: introduces nonlinear factors to help the model learn complex feature representations. The output layer is responsible for producing the final output of the model, which corresponds to the category label in the classification task or the continuous value in the regression task. The loss function is used to measure the difference between the model output and the actual label.

Images