Chip data compression method and device, equipment and storage medium

Abstract

The invention relates to a chip data compression method and device, equipment and a storage medium, and the method comprises the following steps: obtaining to-be-compressed data of a target chip, and carrying out the type analysis of the to-be-compressed data, and obtaining the type of the to-be-compressed data; based on the to-be-compressed data type, performing data compression algorithm selection on the to-be-compressed data to obtain a selected data compression algorithm; performing content perception identification on the to-be-compressed data through a preset content perception algorithm to obtain perception content identification data; performing Huffman coding on the perceived content identification data and non-perceived content identification data in the to-be-compressed data through a selected data compression algorithm to obtain compressed coding data; and performing format conversion on the compressed coded data to obtain standardized coded data, and storing the standardized coded data in the target chip, thereby solving the technical problem that the traditional compression efficiency is not high or the quality of the decompressed data is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and in particular to a chip data compression method, device, equipment and storage medium. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded, which has put forward higher requirements for data storage and transmission efficiency. Especially in embedded systems and mobile devices, due to more obvious resource constraints, efficient data compression technology has become one of the key factors to improve system performance. However, traditional data compression methods often ignore the characteristics and application scenarios of the data itself, resulting in low compression efficiency or degraded data quality after decompression, which cannot meet the high requirements for data quality and real-time performance in specific fields such as image processing and video streaming.

[0003] In addition, in the process of chip design and application, different types of data (such as text, images, and audio) have different structural characteristics, which requires data compression methods to be able to flexibly select the most appropriate compression algorithm according to the data type. Although some existing data compression solutions provide a certain degree of flexibility, in practical applications, there are still problems such as insufficient intelligence in algorithm selection, low compression ratio, and slow decoding speed. Especially for data with high content complexity, how to achieve efficient compression while ensuring data quality is a difficulty in current research.

[0004] In response to these problems, this study proposed a new chip data compression method, which aims to improve the efficiency and quality of data compression by combining data type analysis, content-aware technology and efficient encoding strategies. This method first conducts in-depth analysis of the data to be compressed on the target chip to understand the specific type and characteristics of the data; then uses the content-aware algorithm to further identify the key information in the data to ensure that this information is retained during the compression process; finally, an optimized post-selection data compression algorithm is used for Huffman coding, which not only improves the compression rate but also ensures the quality of the decompressed data. This method provides a new idea for solving the problems existing in existing data compression technology and helps promote the development and application of related technologies. Summary of the invention

[0005] The main purpose of the present invention is to provide a chip data compression method, device, equipment and storage medium, which solves the technical problems of low traditional compression efficiency or reduced data quality after decompression.

[0006] To achieve the above object, the present invention provides a chip data compression method, comprising the following steps: Acquire the data to be compressed of the target chip, and perform type analysis on the data to be compressed to obtain the type of data to be compressed; Based on the type of data to be compressed, selecting a data compression algorithm for the data to be compressed to obtain a selected data compression algorithm; Performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain perceived content recognition data; Performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data; The compressed coded data is format-converted to obtain standardized coded data, and the standardized coded data is stored in a target chip.

[0007] Further, the step of obtaining the data to be compressed of the target chip and performing type analysis on the data to be compressed to obtain the type of data to be compressed includes: Scanning the storage area of ​​the target chip using a preset scanning algorithm to obtain data to be compressed; Performing feature extraction on the data to be compressed to obtain extracted scanning features; Using the Shannon entropy formula to calculate the information entropy of the extracted scanning features respectively, and arranging all the information entropies into an information entropy vector; Inputting the information entropy vector into a preset neural network model for classification to obtain a classification label; The type of data to be compressed is determined based on the classification label; wherein the type of data to be compressed includes message data, image data and audio data.

[0008] Furthermore, the Shannon entropy formula is used to respectively calculate the information entropy of the extracted scanning features, and all the information entropies are arranged into an information entropy vector, including: Discretization processing is performed on the extracted scanning features to convert the continuous extracted scanning features into discrete value features to obtain discrete scanning features; Using a parallel computing architecture, the probability of occurrence of each discrete value in the discretized scanning feature is calculated to obtain a feature probability distribution; Using the Shannon entropy formula, the information entropy of each discrete value is calculated based on the characteristic probability distribution; The information entropy is obtained using the following formula; ; is the information entropy, is the total number of discrete values, is the discretized scan feature No. The probability of a discrete value, is the logarithm to base 2; An index marking algorithm is used to assign a unique index to each of the information entropy to obtain an indexed information entropy value; All the information entropy values ​​with indices are arranged according to the index order in the index mapping table to obtain an information entropy vector.

[0009] Furthermore, based on the type of data to be compressed, selecting a data compression algorithm for the data to be compressed to obtain a selected data compression algorithm includes: Calling a preset algorithm library, and determining a list of candidate compression algorithms corresponding to the data type to be compressed in the algorithm library based on the data type to be compressed; Performing dimensional decomposition on the information entropy vector to obtain vector dimensional information; wherein the vector dimensional information includes vector dimensional correlation, vector dimensional distribution, vector dimensional weight, vector dimensional sparsity and vector dimensional size; Using a wavelet transform algorithm, frequency domain analysis is performed on each of the vector dimension information to obtain frequency domain feature information; Performing element analysis on the information entropy vector based on the frequency domain feature information to obtain element features of the analysis vector; Based on the analytical vector element characteristics, the candidate compression algorithms in the candidate compression algorithm list are predicted to obtain a performance prediction probability distribution; The candidate compression algorithms are prioritized and selected based on the performance prediction probability distribution to obtain a selected data compression algorithm.

[0010] Furthermore, performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain perceived content recognition data includes: Performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain recognized perceived content; Marking the identified perceived content to obtain the marked identified perceived content; Dividing the marked recognized perceived content into data blocks to obtain content data blocks; Performing hash conversion on the content data block to obtain a converted hash value; Detect whether there is a duplicate conversion hash value, and if so, delete the recognition and perception content corresponding to the duplicate conversion hash value, and then recognize the perception content after obtaining a deletion mark; The deleted marked recognized perceptual content is encrypted to obtain encrypted perceptual content data, and the encrypted perceptual content data is used as perceptual content recognition data.

[0011] Furthermore, the step of performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data includes: Using a sliding window technique, the frequency of occurrence of each symbol in the unperceived content recognition data is analyzed and recorded to obtain a frequency statistics table; wherein the frequency statistics table is used to reflect the distribution and frequency of each symbol; Obtain symbol frequency statistics results based on the frequency statistics table; Constructing a Huffman tree based on the frequency statistics result and the frequency statistics table; Assigning a unique binary codeword to each node in the constructed Huffman tree by a preset recursive coding algorithm to obtain a coding mapping table; Performing Huffman coding conversion on the symbols in the non-perceived content identification data through a coding mapping table to generate a preliminary compressed data stream; The perceived content recognition data is merged with the preliminary compressed data stream to form compressed encoded data.

[0012] Furthermore, the format conversion of the compressed coded data to obtain standardized coded data, and storing the standardized coded data in the target chip includes: The compressed coded data is parsed by a preset data parsing algorithm to obtain a compressed coded data structure; wherein the compressed coded data structure includes a data header, a data body and a check bit; Performing format mapping on the compressed coded data based on the compressed coded data structure to obtain mapping format data, and using the mapping format data as standardized coded data; Encapsulating the mapping format data according to a predefined data encapsulation rule to obtain an encapsulated data packet; Encrypting the encapsulated data packet using a data encryption algorithm to obtain an encrypted data packet; Performing storage path planning on the encrypted data packet based on a preset data storage protocol to obtain storage path information; The encrypted data packet is written to a designated location of the target chip through a specific data writing interface.

[0013] The present invention also provides a chip data compression device, comprising: An acquisition module is used to acquire the data to be compressed of the target chip, and perform type analysis on the data to be compressed to obtain the type of data to be compressed; A selection module, configured to select a data compression algorithm for the data to be compressed based on the type of data to be compressed, and obtain a selected data compression algorithm; An identification module, used for performing content-aware identification on the data to be compressed by using a preset content-aware algorithm to obtain perceived content identification data; An encoding module, used for performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data; The conversion module is used to convert the format of the compressed coded data to obtain standardized coded data, and store the standardized coded data in the target chip.

[0014] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0016] The data compression method of the chip provided by the present invention comprises the following steps: obtaining the data to be compressed of the target chip, and performing type analysis on the data to be compressed to obtain the data type to be compressed; based on the data type to be compressed, selecting the data compression algorithm for the data to be compressed to obtain the selected data compression algorithm; performing content-aware identification on the data to be compressed by a preset content-aware algorithm to obtain perceived content identification data; performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by the selected data compression algorithm to obtain compressed coded data; performing format conversion on the compressed coded data to obtain standardized coded data, and storing the standardized coded data in the target chip. Through the above technical means, the technical problems of low efficiency of traditional compression or decreased quality of data after decompression are solved, and the format conversion of the compressed coded data is realized, and standardized coded data is finally generated and stored in the target chip. This process is not only conducive to unified management and rapid retrieval of data, but also convenient for data exchange and sharing between different systems, and improves the technical effect of data compatibility and portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of the steps of a data compression method for a chip in one embodiment of the present invention; Figure 2 is a structural block diagram of a data compression device of a chip in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0018] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] like Figure 1 As shown, Figure 1 This is a schematic diagram of the steps of a data compression method for a chip in one embodiment of the present invention; In one embodiment of the present invention, a chip data compression method is provided, comprising the following steps: Step S1, obtaining the data to be compressed of the target chip, and performing type analysis on the data to be compressed to obtain the type of data to be compressed.

[0021] Specifically, the step of obtaining the data to be compressed of the target chip and performing type analysis on the data to be compressed to obtain the type of data to be compressed means extracting the data to be compressed from the target chip before starting the data compression process. Then, a series of preset analysis tools or algorithms are used to perform in-depth type analysis on the data to be compressed to determine their specific properties and categories. For example, in the application scenario of image processing, if the data to be compressed is a set of high-definition picture files, the type analysis process includes identifying the format (such as JPEG, PNG), resolution size, color depth and other characteristics of these picture files. Through such analysis, the system can understand whether these picture files contain a large amount of color information, whether there are repeated texture patterns or simple backgrounds, and provide a basis for the subsequent selection of data compression algorithms. Similarly, in the application of video streaming, the data to be compressed may be a continuous sequence of video frames, and the type analysis will focus on factors such as the video encoding format (such as H.264, HEVC), frame rate, resolution, and the dynamic range and motion complexity of the video content, so that the compression method suitable for this type of video data can be selected more accurately in the future. This preliminary analysis based on data types not only helps to improve the efficiency of subsequent compression processing, but also can guarantee the quality of compressed data to a certain extent and meet the specific needs of different application scenarios.

[0022] Step S2: Based on the type of data to be compressed, select a data compression algorithm for the data to be compressed to obtain a selected data compression algorithm.

[0023] Specifically, based on the data type to be compressed, a data compression algorithm is selected for the data to be compressed to obtain a selected data compression algorithm, which means that after completing the data type analysis, the most appropriate compression algorithm is selected according to the analysis result. This process not only relies on the basic understanding of the data type, but also needs to consider the specific characteristics of the data and the needs of the application scenario. For example, in the application scenario of image processing, if the data to be compressed is mainly composed of static images, and these images have high resolution and rich color levels, then compression algorithms such as JPEG or WebP that focus on maintaining image quality while reducing file size can be selected. Both algorithms can effectively remove redundant information in the image, such as spatial redundancy and visual redundancy, thereby achieving efficient compression without significantly affecting the image quality. On the contrary, if the data to be compressed is some icons or graphic files with a large number of pure color areas, it may be more appropriate to select the lossless compression algorithm PNG at this time, because PNG can well preserve the details and transparency information of these images. Similarly, in the application scenario of video streaming, the type analysis of the data to be compressed may reveal the characteristics of the video content, such as whether it contains a large number of fast-action scenes, high-contrast picture changes, etc. Based on these analysis results, advanced video compression standards such as H.264 or HEVC can be selected. H.264 is widely used in network video streaming transmission. It can provide good compression efficiency and compatibility. HEVC is the next generation standard of H.264, which provides higher compression rate, especially suitable for 4K and higher resolution video content. Both algorithms can effectively reduce the size of video files while minimizing the impact on video quality. They are very suitable for applications that require efficient transmission of large amounts of video data. In short, by deeply analyzing the type of compressed data and selecting the most appropriate data compression algorithm accordingly, it can ensure that the compression process is both efficient and can meet the quality requirements of specific application scenarios. This intelligent selection mechanism not only improves the overall performance of data compression, but also lays a good foundation for subsequent data processing and transmission.

[0024] Step S3, performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain perceived content recognition data.

[0025] Specifically, the data to be compressed is subjected to content-aware recognition through a preset content-aware algorithm to obtain the perceived content recognition data. This step is to further refine and optimize the data compression process. After completing the data type analysis and selecting the appropriate compression algorithm, the introduction of the content-aware algorithm can help the system understand the information distribution and importance within the data more accurately, so as to take more targeted measures in the compression process. For example, in the application scenario of image processing, the content-aware algorithm can identify which parts are more easily noticed by the human eye, such as faces, text or specific objects, by analyzing the features such as edges, textures, and color distribution in the image. In this way, in the subsequent compression process, the information in these key areas can be protected first to avoid distortion or blurring caused by excessive compression, ensuring that the decompressed image still maintains a high visual quality. Similarly, in the application of video streaming, the content-aware algorithm can identify elements such as the main objects, motion trajectories, and background changes in the video. Through these analyses, the system can distinguish which parts of the video are dynamic and frequently changing, and which parts are relatively static. For the dynamic part, the algorithm can allocate more bit rate to ensure the smoothness and clarity of the motion; for the static or less changing part, the bit rate can be appropriately reduced to reduce unnecessary data redundancy. In this way, not only can the efficiency of video compression be improved, but also the smoothness and visual effect of video playback can be guaranteed. In addition, the application of content-aware algorithms can also help discover repeated patterns or redundant information in the data. In image processing, some textures or patterns may appear repeatedly in different positions of the image. By identifying these repeated parts, more effective encoding strategies can be adopted in the compression process to reduce the number of bits required to store the same information. In video processing, there may be a large amount of similar content between adjacent frames. The content-aware algorithm can identify these similarities and use predictive coding technology to reduce inter-frame redundancy and further improve compression efficiency. In summary, the content-aware identification of the data to be compressed by the preset content-aware algorithm can not only deepen the system's understanding of the internal structure of the data, but also guide the subsequent data compression processing to achieve a more refined and efficient compression effect. This content-aware data processing method provides strong technical support for the efficient storage and transmission of multimedia data such as images and videos.

[0026] Step S4, performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data.

[0027] Specifically, the perceived content recognition data and the unperceived content recognition data in the data to be compressed are Huffman encoded by selecting a data compression algorithm to obtain compressed encoded data. This process is a key step after content-perceived recognition, and the purpose is to further improve the efficiency and quality of data compression. After completing the content-perceived analysis, the system can already identify the key information and non-key information in the data, and then it is necessary to adopt an efficient encoding method to achieve data compression. Huffman encoding is selected here, which is a lossless compression algorithm based on probability statistics. It can assign different encoding lengths to symbols according to the frequency of occurrence of symbols in the data, so that symbols with high frequency of occurrence are represented by shorter codes, while symbols with low frequency of occurrence are represented by longer codes, thereby achieving the purpose of compression. For example, in the application scenario of image processing, assuming that the content-perceived algorithm has identified the face area in the image as key information, this part of the data will be marked as content that needs special protection. For this part of the perceived content recognition data, the system will use the selected data compression algorithm combined with Huffman encoding for processing. Since the face area usually contains more details and changes, Huffman coding can assign the optimal coding length to it according to the frequency of occurrence of each pixel value in these details, so as to achieve efficient compression while maintaining high quality. At the same time, for other areas in the image that are not perceived as key information, such as background or unimportant texture, the system will also use Huffman coding, but may allow a higher compression rate because these areas have less impact on the overall visual effect. In the application scenario of video streaming, the content-aware algorithm may have identified the main moving objects and background areas in the video. For these perceived content recognition data, Huffman coding can assign the optimal coding to them according to the frequency of pixel value changes of the moving objects, ensuring that these objects remain clear and smooth in the compressed video. For background areas or other parts that do not change much and have little impact on the viewing experience, Huffman coding can use a higher compression rate to reduce the storage space occupied by these parts. This differentiated coding strategy can not only improve the overall compression efficiency of the video, but also ensure the integrity of key information and provide users with a better viewing experience. In this way, the post-selection data compression algorithm is combined with Huffman coding to process the perceived content recognition data and the non-perceived content recognition data in the data to be compressed, which not only achieves efficient data compression but also ensures the quality of the compressed data. The application of this technology provides strong support for the efficient storage and transmission of multimedia data such as images and videos, especially in resource-constrained embedded systems and mobile devices, which can significantly improve system performance and user experience.

[0028] Step S5, converting the format of the compressed coded data to obtain standardized coded data, and storing the standardized coded data in a target chip.

[0029] Specifically, the compressed coded data is format-converted to obtain standardized coded data, and the standardized coded data is stored in the target chip. This is the last important step in the data compression process, which is intended to ensure that the compressed data can be stored in a standardized form for subsequent management and use. After completing Huffman coding, although the compressed coded data has achieved a high compression efficiency, these data usually exist in a specific coding format and may not be directly applicable to all systems or applications. Therefore, it is necessary to convert these compressed coded data into a standardized coding format through format conversion so that they can be widely accepted and supported. For example, in the application scenario of image processing, the image data after Huffman coding may exist in an internal proprietary binary format. In order to enable these data to be used smoothly on various image processing software and hardware platforms, they need to be converted into common standardized formats, such as JPEG, PNG or WebP. In this format conversion process, the system will make necessary adjustments and reorganizations to the compressed coded data according to the requirements of the target format. For example, for the JPEG format, the system needs to add corresponding file header information, define image parameters such as width, height, color space, etc., and organize data blocks according to the standard format of JPEG. In this way, the converted standardized coded data can be directly read and displayed by various devices and software that support the JPEG format. Similarly, in the application scenario of video streaming, the Huffman-encoded video data also needs to be converted to adapt to different players and transmission protocols. Assuming that the target format is H.264, the system needs to reorganize the compressed coded data, add the necessary encoding parameters and timestamp information, and ensure that each video frame meets the requirements of the H.264 standard. In addition, it is necessary to generate file headers and index information that conform to the ISO Base Media File Format (such as the MP4 file format) to facilitate the playback and random access of video files. Through this standardized format conversion, it can be ensured that the compressed video data can be seamlessly played on a variety of devices and platforms, whether it is a smartphone, tablet or smart TV. After the format conversion, the standardized coded data is obtained, and the next step is to store the data in the target chip. The target chip can be the internal memory of the embedded system, an SD card or other types of external storage devices. The storage process needs to ensure the integrity and security of the data to prevent errors or damage during transmission. For example, in an embedded system, the system can write standardized coded data to the internal flash memory or external SD card through interfaces such as SPI, I2C or SDIO. During the writing process, the system performs verification and error detection to ensure that each piece of data is stored correctly.In general, by converting the format of compressed coded data to obtain standardized coded data and storing this data in the target chip, not only can the compatibility and portability of data be ensured, but also the convenience and reliability of data management can be improved. This standardized data processing method provides a solid foundation for the efficient storage and transmission of multimedia data such as images and videos, especially in resource-constrained embedded systems and mobile devices, which can significantly improve the overall performance of the system and user experience.

[0030] In a specific embodiment, the step of obtaining the data to be compressed of the target chip and performing type analysis on the data to be compressed to obtain the type of data to be compressed includes: Scanning the storage area of ​​the target chip using a preset scanning algorithm to obtain data to be compressed; Performing feature extraction on the data to be compressed to obtain extracted scanning features; Using the Shannon entropy formula to calculate the information entropy of the extracted scanning features respectively, and arranging all the information entropies into an information entropy vector; Inputting the information entropy vector into a preset neural network model for classification to obtain a classification label; The type of data to be compressed is determined based on the classification label; wherein the type of data to be compressed includes message data, image data and audio data.

[0031] Specifically, first, the storage area of ​​the target chip is scanned by a preset scanning algorithm to obtain the data to be compressed. This step is to extract the data to be compressed from the target chip. The scanning algorithm will traverse the storage area of ​​the chip and identify which data needs to be compressed. For example, in an embedded system, these data may be stored in an internal flash memory or an external SD card. The scanning algorithm will read the data blocks of these storage areas one by one to ensure that all the data to be compressed are completely extracted. Next, the data to be compressed is feature extracted to obtain extracted scanning features. Feature extraction is a key step in data type analysis. By performing a detailed analysis of the data, key information that can reflect the characteristics of the data is extracted. These features may include the format, structure, frequency distribution, etc. of the data. For example, in image data, feature extraction may involve color histograms, edge detection, texture analysis, etc.; in audio data, it may involve spectrum analysis, sound intensity distribution, etc. Through these feature extractions, the system can obtain detailed information about the data, laying the foundation for further analysis and classification. Subsequently, the Shannon entropy formula is used to calculate the information entropy of the extracted scanning features, and all the information entropies are arranged into information entropy vectors. Shannon entropy is an indicator to measure the uncertainty of data. By calculating the information entropy of each feature, the randomness and complexity of these features can be quantified. By calculating the information entropy of each extracted feature, the system can obtain a set of values ​​that reflect the complexity and uncertainty of the data. All information entropy values ​​are arranged into a vector to form an information entropy vector, which provides input data for subsequent classification. Then, the information entropy vector is input into a preset neural network model for classification to obtain a classification label. The neural network model is a powerful machine learning tool that can learn the inherent laws and patterns of data through training. In this step, the information entropy vector is sent as input data to the pre-trained neural network model. The neural network model gradually extracts high-level features of the data through the calculation of multiple layers of neurons, and finally outputs a classification label. This classification label represents the type of data to be compressed, such as message data, image data or audio data. Finally, the type of data to be compressed is determined based on the classification label. The classification label is the result of the output of the neural network model. By parsing this label, the system can clearly know the specific type of data to be compressed. For example, if the classification label indicates that the data type is image data, the system will select a compression algorithm suitable for image data in the subsequent compression step, such as JPEG or WebP; if the classification label indicates that the data type is audio data, the system will select a compression algorithm suitable for audio data, such as MP3 or AAC. Through this series of steps, the system can accurately identify and classify the type of data to be compressed, providing an important foundation for subsequent data compression. This data type analysis method based on feature extraction and machine learning not only improves the efficiency and quality of data compression, but also can adapt to the characteristics and needs of different types of data.For example, in the application scenario of image processing, after the system recognizes the image data through the above steps, it can select efficient image compression algorithms such as JPEG or WebP to ensure the quality of the compressed image; in the application scenario of video streaming, after the system recognizes the video data, it can select advanced video compression standards such as H.264 or HEVC to ensure efficient transmission and playback quality of the video. This intelligent data processing method provides strong technical support for the efficient storage and transmission of multimedia data.

[0032] In a specific embodiment, the Shannon entropy formula is used to respectively calculate the information entropy of the extracted scanning features, and all the information entropies are arranged into an information entropy vector, including: Discretization processing is performed on the extracted scanning features to convert the continuous extracted scanning features into discrete value features to obtain discrete scanning features; Using a parallel computing architecture, the probability of occurrence of each discrete value in the discretized scanning feature is calculated to obtain a feature probability distribution; Using the Shannon entropy formula, the information entropy of each discrete value is calculated based on the characteristic probability distribution; The information entropy is obtained using the following formula; ; is the information entropy, is the total number of discrete values, is the discretized scan feature No. The probability of a discrete value, is the logarithm to base 2; An index marking algorithm is used to assign a unique index to each of the information entropy to obtain an indexed information entropy value; All the information entropy values ​​with indices are arranged according to the index order in the index mapping table to obtain an information entropy vector.

[0033] Specifically, first, the extracted scanning features are discretized to convert the continuous extracted scanning features into discrete value features to obtain discretized scanning features. Discretization is to divide the continuous data values ​​into several intervals, each interval corresponding to a discrete value. For example, in the application scenario of image processing, the extracted features may be the grayscale values ​​of pixels, and these grayscale values ​​are usually continuous integers between 0 and 255. In order to perform discretization, these grayscale values ​​can be divided into several intervals, such as 0-63, 64-127, 128-191, 192-255, each interval corresponding to a discrete value. Through this processing, the continuous grayscale values ​​are converted into discrete interval values, which is convenient for subsequent calculations and analysis.

[0034] Next, the probability of occurrence of each discrete value in the discretized scan feature is calculated using a parallel computing architecture to obtain a feature probability distribution. The parallel computing architecture can significantly improve computing efficiency, especially when processing large-scale data. For example, a GPU or a multi-core CPU can be used for parallel computing. Each computing unit is responsible for calculating the probability of occurrence of a portion of discrete values. By counting the number of occurrences of each discrete value in all data and dividing it by the total number of samples, the probability of each discrete value can be obtained. For example, in image data, if a certain grayscale value interval appears 1000 times in all pixels, and the total number of pixels is 10,000, then the probability of this interval is 0.1.

[0035] Then, using the Shannon entropy formula , the information entropy of each discrete value is calculated based on the feature probability distribution. The Shannon entropy formula is used to measure the uncertainty or randomness of data. For each discrete value , calculate its information entropy Specifically, the probability of each high scatter value Substitute into the formula and calculate the corresponding information entropy. For example, if the probability of a discrete value is 0.1, then its information entropy is 0.332.

[0036] Next, an index marking algorithm is used to assign a unique index to each of the information entropy values ​​to obtain an indexed information entropy value. The index marking algorithm is used to assign a unique identifier to each information entropy value to facilitate subsequent sorting and processing. For example, an integer index can be assigned to each information entropy value in sequence, such as 1, 2, 3, etc. In this way, each information entropy value has a corresponding index, forming an indexed information entropy value.

[0037] Finally, all the information entropy values ​​with indexes are arranged according to the index order in the index mapping table to obtain the information entropy vector. The index mapping table is a table that records the correspondence between indexes and information entropy values. Through this table, all the information entropy values ​​with indexes can be arranged in the order of indexes to form an ordered information entropy vector. For example, if the information entropy values ​​and their indexes are (0.332, 1), (0.5, 2), (0.1, 3), respectively, the information entropy vector arranged in the order of indexes is [0.332, 0.5, 0.1].

[0038] Through this series of steps, the system can convert the extracted scanning features into information entropy vectors. These information entropy vectors contain the uncertainty information of the data and can be sent as input data to the neural network model for classification. For example, in the application scenario of image processing, the information entropy vector can reflect the complexity and randomness of the image, helping the system to identify the image data; in the application scenario of video streaming, the information entropy vector can reflect the changes in the video frame and help the system to identify the video data. This feature extraction and classification method based on information entropy not only improves the accuracy of data type identification, but also provides an important reference for subsequent data compression. In this way, the system can more intelligently select the appropriate compression algorithm and improve the efficiency and quality of data compression.

[0039] In a specific embodiment, the selecting of a data compression algorithm for the data to be compressed based on the type of data to be compressed to obtain a selected data compression algorithm includes: Calling a preset algorithm library, and determining a list of candidate compression algorithms corresponding to the data type to be compressed in the algorithm library based on the data type to be compressed; Performing dimensional decomposition on the information entropy vector to obtain vector dimensional information; wherein the vector dimensional information includes vector dimensional correlation, vector dimensional distribution, vector dimensional weight, vector dimensional sparsity and vector dimensional size; Using a wavelet transform algorithm, frequency domain analysis is performed on each of the vector dimension information to obtain frequency domain feature information; Performing element analysis on the information entropy vector based on the frequency domain feature information to obtain element features of the analysis vector; Based on the analytical vector element characteristics, the candidate compression algorithms in the candidate compression algorithm list are predicted to obtain a performance prediction probability distribution; The candidate compression algorithms are prioritized and selected based on the performance prediction probability distribution to obtain a selected data compression algorithm.

[0040] Specifically, first, a preset algorithm library is called, and a list of candidate compression algorithms corresponding to the data type to be compressed is determined in the algorithm library based on the data type to be compressed. The algorithm library is a collection of multiple data compression algorithms, which are applicable to different types of data. For example, for image data, the algorithm library may include compression algorithms such as JPEG, PNG, and WebP; for audio data, it may include compression algorithms such as MP3 and AAC; for video data, it may include compression algorithms such as H.264 and HEVC. The system selects applicable candidate compression algorithms from the algorithm library according to the data type to be compressed determined in the previous step to form a list of candidate compression algorithms. For example, if the data type to be compressed is image data, the system selects algorithms such as JPEG, PNG, and WebP as candidates from the algorithm library. Next, the information entropy vector is dimensionally decomposed to obtain vector dimension information. The information entropy vector contains information of multiple dimensions, which reflect the complexity and uncertainty of the data. The purpose of dimensional decomposition is to extract the features of these dimensions for subsequent analysis and processing. The vector dimension information includes vector dimension correlation, vector dimension distribution, vector dimension weight, vector dimension sparsity, and vector dimension size. Vector dimension correlation reflects the degree of association between different dimensions; vector dimension distribution describes the distribution of values ​​of each dimension; vector dimension weight indicates the importance of each dimension; vector dimension sparsity reflects the proportion of zero values ​​in the vector; vector dimension size indicates the length of the vector. For example, in image data, the dimensions of the information entropy vector may include color distribution, texture complexity, edge strength, etc., and these features can be extracted through dimensional decomposition. Then, using the wavelet transform algorithm, each of the vector dimension information is subjected to frequency domain analysis to obtain frequency domain feature information. Wavelet transform is a multi-resolution analysis method that can decompose a signal into components of different frequencies. By performing wavelet transform on the vector dimension information, the features of each dimension at different frequencies can be extracted. Frequency domain feature information reflects the distribution of data at different frequencies, which helps to identify periodic features and local changes in the data. For example, in image data, high-frequency details (such as edges and textures) and low-frequency backgrounds (such as smooth areas) in the image can be identified through wavelet transform. Next, the information entropy vector is element-parsed based on the frequency domain feature information to obtain the element features of the parsed vector. The analytical vector element features are the detailed analysis results of each element in the information entropy vector. These features describe the specific performance of the data in different dimensions. For example, the analytical vector element features may include statistical information such as the maximum value, minimum value, average value, variance, and the correlation coefficient between the dimensions. These features provide basic data for subsequent performance prediction. Then, based on the analytical vector element features, the performance of the candidate compression algorithms in the candidate compression algorithm list is predicted to obtain the performance prediction probability distribution.Performance prediction is to predict the performance of each candidate compression algorithm on the current data by analyzing the characteristics of the analytical vector elements. Performance prediction can be performed based on historical data and known algorithm performance using machine learning models (such as decision trees, support vector machines, neural networks, etc.). The performance prediction probability distribution represents the probability that each candidate compression algorithm performs well on the current data. For example, if the analytical vector element characteristics of a certain image data indicate that the image has complex textures and edges, then the JPEG algorithm may find a good balance between compression rate and image quality, so its performance prediction probability is higher. Finally, the candidate compression algorithms are prioritized and selected based on the performance prediction probability distribution to obtain the selected data compression algorithm. Priority sorting is to arrange the candidate compression algorithms in order from high to low according to the performance prediction probability distribution. The selected data compression algorithm is to select one or more algorithms with the best performance from the sorted candidate compression algorithms. For example, if the performance prediction probability of the JPEG algorithm is the highest, then the system will select JPEG as the final data compression algorithm. If further optimization is required, several algorithms with the highest performance prediction probability can be selected for actual testing, and the best performing algorithm can be finally selected. Through this series of steps, the system can intelligently select the most suitable compression algorithm based on the type and characteristics of the data to be compressed. This algorithm selection method based on data characteristics and performance prediction not only improves the efficiency of data compression, but also ensures the quality of compressed data. For example, in the application scenario of image processing, the system selects the JPEG algorithm through the above steps, which can achieve efficient compression while maintaining image quality; in the application scenario of video streaming, the system selects the H.264 algorithm, which can ensure efficient transmission and smooth playback of the video. This intelligent data processing method provides strong technical support for the efficient storage and transmission of multimedia data.

[0041] In a specific embodiment, performing content-aware recognition on the data to be compressed by a preset content-aware algorithm to obtain the perceived content recognition data includes: Performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain recognized perceived content; Marking the identified perceived content to obtain the marked identified perceived content; Dividing the marked recognized perceived content into data blocks to obtain content data blocks; Performing hash conversion on the content data block to obtain a converted hash value; Detect whether there is a duplicate conversion hash value, and if so, delete the recognition and perception content corresponding to the duplicate conversion hash value, and then recognize the perception content after obtaining a deletion mark; The deleted marked recognized perceptual content is encrypted to obtain encrypted perceptual content data, and the encrypted perceptual content data is used as perceptual content recognition data.

[0042] Specifically, first, the data to be compressed is subjected to content-aware recognition through a preset content-aware algorithm to obtain recognition-aware content. The content-aware algorithm is a technology that can understand and recognize data content and can be used to extract key information and features from the data. For example, in the application scenario of image processing, the content-aware algorithm can recognize key information such as faces, text, and objects in the image by analyzing features such as edges, textures, and color distribution in the image. In the application scenario of video streaming, the content-aware algorithm can recognize the main objects, motion trajectories, and background changes in the video. Through these recognitions, the system can understand the important parts of the data and provide a basis for subsequent processing. Next, the recognition-aware content is marked to obtain the marked recognition-aware content. The marking process is to mark the key information identified for subsequent processing and management. For example, in image data, the system can add bounding boxes or other marks around key information such as face areas and text areas so that the subsequent compression algorithm can give priority to protecting these areas. In video data, the system can add labels to each main object to record its position and motion trajectory. These labels not only help the compression algorithm to better process the data, but can also be used for subsequent data retrieval and analysis. Then, the identified and perceived content after the marking is subjected to data segmentation to obtain content data blocks. Data segmentation is to divide the data into multiple small blocks, each of which contains a part of the identified and perceived content after the marking. This segmentation helps to improve the parallelism and efficiency of processing. For example, in image data, the image can be divided into multiple small blocks, each of which contains a part of the face area or the text area; in video data, the video frame can be divided into multiple areas, each of which contains a main object or background part. Through data segmentation, multiple small blocks can be processed in parallel to improve the processing speed. Next, the content data block is hashed to obtain a converted hash value. Hash conversion is to convert each data block into a hash value of a fixed length, which can uniquely identify the content of the data block. Hash conversion helps to detect the repeatability and uniqueness of data blocks. For example, hash algorithms such as MD5 and SHA-1 can be used to perform hash conversion on each data block. Through hash conversion, the system can quickly compare the hash values ​​of different data blocks to detect whether there are repeated data blocks. Then, it is detected whether there is a duplicate conversion hash value. If so, the recognition perception content corresponding to the duplicate conversion hash value is deleted, and the recognition perception content is identified after the deletion mark is obtained. The deletion of duplicate data blocks is to reduce data redundancy and improve compression efficiency. For example, if the hash values ​​of multiple data blocks are the same, it means that the contents of these data blocks are exactly the same. The system can only retain one data block and delete the remaining duplicate data blocks. In this way, the storage space of the data can be significantly reduced and the compression efficiency can be improved. At the same time, the hash value and location information of a data block are retained to facilitate subsequent decompression and recovery.Finally, the deleted marked perceived content is encrypted to obtain encrypted perceived content data, and the encrypted perceived content data is used as perceived content recognition data. The encryption process is to protect the security and privacy of the data. For example, the data blocks after deduplication can be encrypted using encryption algorithms such as AES and RSA. The encrypted data blocks can only be decrypted by the correct key, thereby preventing unauthorized access and tampering. Through encryption, the system can ensure that the compressed data will not be illegally accessed during storage and transmission, thereby improving the security of the data. Through this series of steps, the system can identify and process key information in the data through content-aware algorithms, reduce data redundancy, improve compression efficiency, and ensure data security and privacy. For example, in the application scenario of image processing, the system identifies the face and text areas in the image through content-aware algorithms, marks these areas, performs data segmentation and hash conversion, deletes duplicate data blocks, and finally encrypts the remaining data blocks to ensure that the compressed image data is both efficient and secure. In the application scenario of video streaming, the system identifies the main objects in the video through a similar method, marks and processes these objects, reduces redundant data, improves the compression efficiency and transmission speed of the video, and protects the security of the video content. This content-aware data processing method provides strong technical support for the efficient storage and transmission of multimedia data.

[0043] In a specific embodiment, the step of performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data includes: Using a sliding window technique, the frequency of occurrence of each symbol in the unperceived content recognition data is analyzed and recorded to obtain a frequency statistics table; wherein the frequency statistics table is used to reflect the distribution and frequency of each symbol; Obtain symbol frequency statistics results based on the frequency statistics table; Constructing a Huffman tree based on the frequency statistics result and the frequency statistics table; Assigning a unique binary codeword to each node in the constructed Huffman tree by a preset recursive coding algorithm to obtain a coding mapping table; Performing Huffman coding conversion on the symbols in the non-perceived content identification data through a coding mapping table to generate a preliminary compressed data stream; The perceived content recognition data is merged with the preliminary compressed data stream to form compressed encoded data.

[0044] Specifically, first, the sliding window technology is used to analyze and record the frequency of occurrence of each symbol in the unperceived content recognition data to obtain a frequency statistics table. The sliding window technology is a commonly used data analysis method. By sliding in a fixed-size window, the symbol frequency in the data is gradually analyzed. For example, in the application scenario of image processing, the unperceived content recognition data may include background areas or unimportant textures in the image. The system can set a sliding window, and the window size can be adjusted according to actual needs. When the window slides in the data, the number of times each symbol (such as pixel value) appears is recorded. In this way, the system can count the frequency of occurrence of each symbol in the unperceived content recognition data and generate a frequency statistics table. The frequency statistics table records the distribution and frequency of each symbol in detail, providing basic data for subsequent encoding. Next, the symbol frequency statistics result is obtained based on the frequency statistics table. The symbol frequency statistics result is to summarize and organize the data in the frequency statistics table to form a clear symbol frequency distribution. For example, the frequency statistics table may record the number of times each pixel value appears in the image, and the symbol frequency statistics result will summarize which pixel values ​​appear most frequently and which appear least frequently. These statistical results are helpful for the subsequent Huffman tree construction and coding optimization. Then, a Huffman tree is constructed based on the frequency statistics and the frequency statistics table. The Huffman tree is a binary tree structure used to generate the optimal prefix code. The process of constructing the Huffman tree is as follows: First, each symbol and its frequency in the frequency statistics table is taken as a node to create an initial node set. Then, the two nodes with the smallest frequency are selected and merged into a new node, and the frequency of the new node is the sum of the frequencies of the two nodes. The new node is added to the set and the original two nodes are removed from the set. Repeat this process until only one node is left in the set, that is, the root node of the Huffman tree. In this way, the system can construct a Huffman tree, in which each leaf node in the tree represents a symbol, and the path length and direction determine the encoding of the symbol. Next, a unique binary codeword is assigned to each node in the constructed Huffman tree by a preset recursive encoding algorithm to obtain a coding mapping table. The recursive encoding algorithm starts from the root node of the Huffman tree, traverses downward along the path of the tree, and assigns a binary codeword to each node. Usually, the left branch is assigned 0 and the right branch is assigned 1. When a leaf node is reached, the symbol corresponding to the node is assigned a unique binary codeword. The binary codewords of all leaf nodes are combined together to form a coding mapping table. The coding mapping table records the mapping relationship between each symbol and its corresponding binary codeword, providing a basis for subsequent coding conversion. Then, the symbols in the non-perceived content identification data are Huffman coded through the coding mapping table to generate a preliminary compressed data stream. The coding conversion process is to replace each symbol in the non-perceived content identification data with its corresponding binary codeword.For example, in the image data, if the binary codeword of a pixel value is 010, then during the encoding conversion process, the pixel value will be replaced with 010. In this way, the system can convert each symbol in the non-perceived content recognition data into a shorter binary codeword to generate a preliminary compressed data stream. The preliminary compressed data stream is the data after Huffman encoding, which greatly reduces the space occupied by the original data. Finally, the perceived content recognition data and the preliminary compressed data stream are merged to form compressed encoded data. The perceived content recognition data is the data that has been identified and processed by the content-aware algorithm. This data usually contains key information and needs to be protected first during the compression process. The preliminary compressed data stream is the result of Huffman encoding of the non-perceived content recognition data. The two parts of data are merged to form the final compressed encoded data. The data merging process can be a simple splicing or a more complex interleaving or embedding method, depending on the needs of the application scenario. For example, in image processing, the perceived content recognition data (such as the face area) can be placed at the beginning of the compressed encoded data to ensure that these key information is restored first during the decompression process. Through this series of steps, the system can efficiently compress data through Huffman coding technology to ensure that the compressed data is both efficient and reliable. For example, in the application scenario of image processing, the system analyzes the symbol frequency of the background area through the sliding window technology, constructs the Huffman tree and generates a coding mapping table, performs Huffman coding on the background area, and generates a preliminary compressed data stream. Then, the face area processed by the content-aware algorithm is merged with the preliminary compressed data stream to form the final compressed coded data. In this way, the compressed image data not only takes up less space, but also effectively protects the key information. In the application scenario of video streaming transmission, the system can also use a similar method to efficiently compress the background area in the video, ensure the clarity and smoothness of the main object, and improve the efficiency and quality of video transmission. This data compression method based on Huffman coding provides strong technical support for the efficient storage and transmission of multimedia data.

[0045] In a specific embodiment, the format conversion of the compressed coded data to obtain standardized coded data, and storing the standardized coded data in the target chip includes: The compressed coded data is parsed by a preset data parsing algorithm to obtain a compressed coded data structure; wherein the compressed coded data structure includes a data header, a data body and a check bit; Performing format mapping on the compressed coded data based on the compressed coded data structure to obtain mapping format data, and using the mapping format data as standardized coded data; Encapsulating the mapping format data according to a predefined data encapsulation rule to obtain an encapsulated data packet; Encrypting the encapsulated data packet using a data encryption algorithm to obtain an encrypted data packet; Performing storage path planning on the encrypted data packet based on a preset data storage protocol to obtain storage path information; The encrypted data packet is written to a designated location of the target chip through a specific data writing interface.

[0046] Specifically, first, the compressed coded data is parsed by a preset data parsing algorithm to obtain a compressed coded data structure. The data parsing algorithm is used to analyze the internal structure of the compressed coded data and decompose it into different components. The compressed coded data structure usually includes a data header, a data body and a check bit. The data header contains basic information about the data, such as data type, length, version number, etc.; the data body is the actual data content; and the check bit is used to verify the integrity and correctness of the data. For example, in the application scenario of image processing, the compressed coded data may be a JPEG file. The data parsing algorithm will parse out the identifier, data length and other information in the file header, and the data body is the image data after Huffman encoding. The check bit is used to ensure that the data is not damaged during transmission. Next, the compressed coded data is format mapped based on the compressed coded data structure to obtain mapping format data, and the mapping format data is used as standardized coded data. Format mapping is to convert the parsed data structure into a data format that meets a specific standard. For example, if the target format is JPEG, the system will reorganize the information in the data header according to the requirements of the JPEG standard format to ensure that each field meets the standard. The data body part will also be rearranged and encoded according to the standard format. In this way, the system can convert the compressed encoded data into a standardized format so that it can be recognized and processed by various devices and software that support the format. Then, the mapping format data is encapsulated by predefined data encapsulation rules to obtain an encapsulated data packet. The data encapsulation rules define how to package the standardized encoded data into a complete data packet for easy transmission and storage. Encapsulation processing usually includes adding file headers, tail information, checksums, etc. For example, in image processing, the encapsulation rules may require adding file identifiers and length information at the beginning of the data packet and adding checksums at the end. Through encapsulation processing, the system can ensure the integrity and consistency of the data packet, which is convenient for subsequent processing and storage. Next, the encapsulated data packet is encrypted using a data encryption algorithm to obtain an encrypted data packet. Data encryption is to protect the security and privacy of data and prevent unauthorized access and tampering. Common encryption algorithms include AES (Advanced Encryption Standard), RSA (Rivest-Shamir-Adleman), etc. Encryption processing converts the encapsulated data packet into ciphertext form, and only the recipient with the correct key can decrypt and restore the original data. For example, in image processing, the system can use the AES algorithm to encrypt the encapsulated JPEG file to ensure that the image data will not be illegally accessed during storage and transmission. Then, the storage path of the encrypted data packet is planned based on the preset data storage protocol to obtain storage path information. The data storage protocol defines the storage rules and paths of data in the storage device. Storage path planning is to ensure that the data can be correctly stored in the specified location of the target chip.For example, in an embedded system, the data storage protocol may specify the storage path of data in the internal flash memory or external SD card. According to these rules, the system will assign a unique storage path to each encrypted data packet and generate storage path information. In this way, the system can ensure the orderly storage and fast retrieval of data. Finally, the encrypted data packet is written to the specified location of the target chip through a specific data write interface. The data write interface is a hardware interface for transferring data to a storage device. Common interfaces include SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit Bus), SDIO (Secure Digital Input Output), etc. The system writes the encrypted data packet to the specified location of the target chip through these interfaces. For example, in an embedded system, the system can write the encrypted image data to the specified address of the internal flash memory through the SPI interface to ensure the safe storage of the data. In this way, the system can efficiently store the compressed data in the target chip, improving the storage efficiency and security of the data. Through this series of steps, the system can convert the compressed encoded data into a standardized format and store it securely in the target chip. This standardized and encrypted data processing method not only improves the compatibility and security of the data, but also provides convenience for subsequent data management and use. For example, in the application scenario of image processing, the system stores the compressed image data in JPEG format in the internal flash memory of the embedded system through the above steps, ensuring that the image data is both efficient and secure. In the application scenario of video streaming, the system can also use a similar method to store the compressed video data in H.264 format in the target chip to improve the storage efficiency and transmission speed of video data. This data processing method based on standardization and encryption provides strong technical support for the efficient storage and transmission of multimedia data.

[0047] The data compression method of the chip in the embodiment of the present invention is described above. The data compression system of the chip in the embodiment of the present invention is described below. Figure 2 , an embodiment of the data compression system of the chip in the embodiment of the present invention includes: The acquisition module 21 is used to acquire the data to be compressed of the target chip, and perform type analysis on the data to be compressed to obtain the type of data to be compressed; A selection module 22, configured to select a data compression algorithm for the data to be compressed based on the type of data to be compressed, and obtain a selected data compression algorithm; The recognition module 23 is used to perform content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain perceived content recognition data; The encoding module 24 is used to perform Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data; The conversion module 25 is used to perform format conversion on the compressed coded data to obtain standardized coded data, and store the standardized coded data in the target chip.

[0048] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the above method embodiment, which will not be repeated here.

[0049] Reference Figure 3 The present invention also provides a computer device in an embodiment, wherein the internal structure of the computer device can be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0050] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0051] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0052] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0053] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0054] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A chip data compression method, characterized in that: The following steps are involved: Acquire the data to be compressed of the target chip, and perform type analysis on the data to be compressed to obtain the type of data to be compressed; Based on the type of data to be compressed, selecting a data compression algorithm for the data to be compressed to obtain a selected data compression algorithm; Performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain perceived content recognition data; Performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data; The compressed coded data is format-converted to obtain standardized coded data, and the standardized coded data is stored in a target chip.

2. The chip data compression method according to claim 1, characterized in that: The step of obtaining the data to be compressed of the target chip and performing type analysis on the data to be compressed to obtain the type of data to be compressed includes: Scanning the storage area of ​​the target chip using a preset scanning algorithm to obtain data to be compressed; Performing feature extraction on the data to be compressed to obtain extracted scanning features; Using the Shannon entropy formula to calculate the information entropy of the extracted scanning features respectively, and arranging all the information entropies into an information entropy vector; Inputting the information entropy vector into a preset neural network model for classification to obtain a classification label; The type of data to be compressed is determined based on the classification label; wherein the type of data to be compressed includes message data, image data and audio data.

3. The chip data compression method according to claim 2, characterized in that: The method of using the Shannon entropy formula to respectively calculate the information entropy of the extracted scanning features and arranging all the information entropies into an information entropy vector includes: Discretization processing is performed on the extracted scanning features to convert the continuous extracted scanning features into discrete value features to obtain discrete scanning features; Using a parallel computing architecture, the probability of occurrence of each discrete value in the discretized scanning feature is calculated to obtain a feature probability distribution; Using the Shannon entropy formula, the information entropy of each discrete value is calculated based on the characteristic probability distribution; The information entropy is obtained using the following formula; ； is the information entropy, is the total number of discrete values, is the discretized scan feature No. The probability of a discrete value, is the logarithm to base 2; An index marking algorithm is used to assign a unique index to each of the information entropy to obtain an indexed information entropy value; All the information entropy values ​​with indices are arranged according to the index order in the index mapping table to obtain an information entropy vector.

4. The chip data compression method according to claim 2, characterized in that: The step of selecting a data compression algorithm for the data to be compressed based on the type of data to be compressed to obtain a selected data compression algorithm includes: Calling a preset algorithm library, and determining a list of candidate compression algorithms corresponding to the data type to be compressed in the algorithm library based on the data type to be compressed; Performing dimensional decomposition on the information entropy vector to obtain vector dimensional information; wherein the vector dimensional information includes vector dimensional correlation, vector dimensional distribution, vector dimensional weight, vector dimensional sparsity and vector dimensional size; Using a wavelet transform algorithm, frequency domain analysis is performed on each of the vector dimension information to obtain frequency domain feature information; Performing element analysis on the information entropy vector based on the frequency domain feature information to obtain element features of the analysis vector; Based on the analytical vector element characteristics, the candidate compression algorithms in the candidate compression algorithm list are predicted to obtain a performance prediction probability distribution; The candidate compression algorithms are prioritized and selected based on the performance prediction probability distribution to obtain a selected data compression algorithm.

5. The chip data compression method according to claim 1, characterized in that: The performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain the perceived content recognition data includes: Performing content-aware recognition on the data to be compressed by using a preset content-aware algorithm to obtain recognized perceived content; Marking the identified perceived content to obtain the marked identified perceived content; Dividing the marked recognized perceived content into data blocks to obtain content data blocks; Performing hash conversion on the content data block to obtain a converted hash value; Detect whether there is a duplicate conversion hash value, and if so, delete the recognition and perception content corresponding to the duplicate conversion hash value, and then recognize the perception content after obtaining a deletion mark; The deleted marked recognized perceptual content is encrypted to obtain encrypted perceptual content data, and the encrypted perceptual content data is used as perceptual content recognition data.

6. The chip data compression method according to claim 1, characterized in that: The step of performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data includes: Using a sliding window technique, the frequency of occurrence of each symbol in the unperceived content recognition data is analyzed and recorded to obtain a frequency statistics table; wherein the frequency statistics table is used to reflect the distribution and frequency of each symbol; Obtain symbol frequency statistics results based on the frequency statistics table; Constructing a Huffman tree based on the frequency statistics result and the frequency statistics table; Assigning a unique binary codeword to each node in the constructed Huffman tree by a preset recursive coding algorithm to obtain a coding mapping table; Performing Huffman coding conversion on the symbols in the non-perceived content identification data through a coding mapping table to generate a preliminary compressed data stream; The perceived content recognition data is merged with the preliminary compressed data stream to form compressed encoded data.

7. The chip data compression method according to claim 1, characterized in that: The method of converting the format of the compressed coded data to obtain standardized coded data, and storing the standardized coded data in the target chip includes: The compressed coded data is parsed by a preset data parsing algorithm to obtain a compressed coded data structure; wherein the compressed coded data structure includes a data header, a data body and a check bit; Performing format mapping on the compressed coded data based on the compressed coded data structure to obtain mapping format data, and using the mapping format data as standardized coded data; Encapsulating the mapping format data according to a predefined data encapsulation rule to obtain an encapsulated data packet; Encrypting the encapsulated data packet using a data encryption algorithm to obtain an encrypted data packet; Performing storage path planning on the encrypted data packet based on a preset data storage protocol to obtain storage path information; The encrypted data packet is written to a designated location of the target chip through a specific data writing interface.

8. A data compression device for a chip, characterized in that: include: An acquisition module is used to acquire the data to be compressed of the target chip, and perform type analysis on the data to be compressed to obtain the type of data to be compressed; A selection module, configured to select a data compression algorithm for the data to be compressed based on the type of data to be compressed, and obtain a selected data compression algorithm; An identification module, used for performing content-aware identification on the data to be compressed by using a preset content-aware algorithm to obtain perceived content identification data; An encoding module, used for performing Huffman encoding on the perceived content identification data and the non-perceived content identification data in the data to be compressed by selecting a post-data compression algorithm to obtain compressed encoded data; The conversion module is used to convert the format of the compressed coded data to obtain standardized coded data, and store the standardized coded data in the target chip.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images