Big data compression method, system and storage medium based on neural network
By pruning the self-encoding network network, and optimizing the network structure using the degree of influence of neuron weight entropy, the problem of long training time of self-encoding network is solved and efficient data compression is achieved.
Patent Information
- Application Number
- CN202210351881.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-02
- Publication Date
- 2025-09-05
- Estimated Expiration
- 2042-04-02
AI Technical Summary
In the prior art, the self-encoding network is used for big data compression for a long training time and the network structure is complex, which leads to the compression and decompression process being too long.
Through arithmetic encoding, building an auto-coding network and performing network pruning, the degree of influence of neurons on the entropy of weight matrix is used as the importance value, redundant neurons and connections are deleted, and the structure of the auto-coding network is optimized.
While compressing data, it reduces compression time and improves data compression efficiency and accuracy.
Smart Images
Figure CN114640356B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data compression, and specifically to a big data compression method, system and storage medium based on neural network. Background Art
[0002] Big data, or massive data, refers to information so large in scale that it cannot be captured, managed, processed, and organized into information that helps businesses make more proactive business decisions within a reasonable timeframe using current mainstream software tools.
[0003] Generally speaking, data contains a large amount of redundant information, a characteristic that is particularly pronounced in large data. The purpose of data compression is to minimize data redundancy, and autoencoding networks are currently commonly used for data compression. However, the inventors of the present invention discovered that when using autoencoding networks to compress large data, the required training time is long, and the autoencoding networks required to be constructed are relatively complex, further increasing the training time and making the compression and decompression processes excessively time-consuming. Summary of the Invention
[0004] In response to the above technical problems, the present invention provides a big data compression method, system and storage medium based on neural networks, which can compress the data while also compressing the neural network model in combination with the distribution characteristics of the data, so that the data compression ratio meets the requirements while reducing the compression time.
[0005] In a first aspect, the embodiments of the present invention provide a big data compression method based on a neural network, comprising:
[0006] Each data to be compressed is arithmetically coded, and an initial weight of each data is obtained according to the similarity between the coded data and other coded data and the types of symbols contained in the data.
[0007] An autoencoding network is constructed, wherein the autoencoding network includes an input layer, an output layer, and at least one hidden layer, wherein the number of neurons in the input layer and the output layer is the same and greater than the number of neurons in the hidden layer.
[0008] The encoded data is used as the input and output of the autoencoding network at the same time, the initial weight of each data is used as the initial weight of the neuron corresponding to each data, and the autoencoding network is trained to obtain the weight matrix of the encoding network.
[0009] The degree of influence of neurons in the autoencoder network on the entropy of the weight matrix is used as the importance value of the neuron, and the autoencoder network is pruned. During the network pruning process, redundant neurons and connections are deleted according to the importance value of the neuron and the weight value of the connection in the weight matrix until at least one of the compression rate and accuracy of the autoencoder network is outside the corresponding preset threshold range.
[0010] The data corresponding to the hidden layer in the autoencoder network after network pruning is used as compressed data.
[0011] In a feasible embodiment, the autoencoder network is pruned, and during the network pruning process, redundant neurons and connections are deleted according to the importance values of neurons and the weight values of connections in the weight matrix, including:
[0012] Delete the connection corresponding to the minimum weight value in the weight matrix in the autoencoder network, and retrain the autoencoder network after deleting the connection.
[0013] When the set of input connections of multiple neurons is included in the set of input connections of any neuron, the neuron with the smallest importance value among the multiple neurons is deleted, and the autoencoder network after the deleted neuron is retrained.
[0014] The neurons without output connections or with output connections in the trained autoencoder network are deleted, and the autoencoder network is retrained.
[0015] In a feasible embodiment, the influence of the neuron in the autoencoder network on the entropy of the weight matrix is used as the importance value of the neuron, including:
[0016] The entropy of the weight matrix of the autoencoder network when the neuron exists in the autoencoder network is used as the first entropy value, the entropy of the weight matrix obtained by setting the value corresponding to the neuron in the weight matrix to 0 is used as the second entropy value, and the absolute value of the difference between the first entropy value and the second entropy value is used as the degree of influence of the neuron on the entropy of the weight matrix.
[0017] In a feasible embodiment, the initial weight of each data is obtained based on the similarity between the encoded data and other encoded data and the types of symbols contained in the data, including:
[0018] According to the mean of the similarity between the encoded data and other encoded data, the similarity of each encoded data is obtained respectively.
[0019] The types of all the data included in the statistics are used to obtain the types of all the data, and the ratio of the symbol types included in each of the data to the symbol types of all the data is used as the symbol proportion of each data.
[0020] The ratio of the symbol proportion of the data to the similarity of the data is used as the initial weight of the data.
[0021] In a feasible embodiment, before performing arithmetic coding on each data to be compressed, the method further includes processing abnormal symbols in each data separately.
[0022] In a feasible embodiment, processing abnormal symbols in each data is achieved through box plots.
[0023] In a feasible embodiment, a method for obtaining similarity between the encoded data and other encoded data includes:
[0024]
[0025] Among them, s ij is the similarity between the encoded i-th data and the encoded j-th data, a i is the number of decimal places of the ith data after encoding, a j is the number of decimal places of the jth data after encoding, b i is the value of the i-th data after encoding, b j The value of the j-th data after encoding, i, j are positive integers not greater than n, and i ≠ j, n is the number of encoded data.
[0026] In a second aspect, an embodiment of the present invention proposes a big data compression system based on a neural network, comprising:
[0027] The arithmetic coding module is used to perform arithmetic coding on each data to be compressed.
[0028] The initial weight acquisition module is used to obtain the initial weight of each data according to the similarity between the encoded data and other encoded data and the types of symbols contained in the data.
[0029] The autoencoding network construction module is used to construct an autoencoding network, wherein the autoencoding network includes an input layer, an output layer and at least one hidden layer, and the number of neurons in the input layer and the output layer is the same and greater than the number of neurons in the hidden layer.
[0030] The weight matrix acquisition module is used to use the encoded data as the input and output of the autoencoding network at the same time, use the initial weight of each data as the initial weight of the neuron corresponding to each data, and train the autoencoding network to obtain the weight matrix of the encoding network.
[0031] The network pruning module is used to prune the autoencoder network by taking the degree of influence of neurons in the autoencoder network on the entropy of the weight matrix as the importance value of the neurons. During the network pruning process, redundant neurons and connections are deleted according to the importance values of the neurons and the weight values of the connections in the weight matrix until at least one of the compression rate and accuracy of the autoencoder network is outside the corresponding preset threshold range.
[0032] The compressed data acquisition module is used to use the data corresponding to the hidden layer in the autoencoder network after network pruning as compressed data.
[0033] In a third aspect, an embodiment of the present invention provides a storage medium for big data compression based on a neural network, which stores a program that can be loaded and executed by a processor to implement a big data compression method based on a neural network as in an embodiment of the present invention.
[0034] Compared with the prior art, the beneficial effects of the embodiments of the present invention include at least: while compressing the data, the neural network model is also compressed in combination with the distribution characteristics of the data, so that the data compression ratio meets the requirements while reducing the compression time. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0036] Figure 1 This is a flowchart of a neural network-based big data compression method provided by an embodiment of the present invention.
[0037] Figure 2 Schematic diagram of the structure of the autoencoding network in an embodiment of the present invention.
[0038] Figure 3 This is a structural diagram of a neural network-based big data compression system provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0040] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.
[0041] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features being referred to. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of such features; and in the description of this embodiment, unless otherwise specified, "plurality" means two or more.
[0042] Generally speaking, data contains a large amount of redundant information, and big data contains even more redundant information. Data compression aims to minimize this redundancy. Traditional data compression methods often focus on reducing redundancy in data, focusing on coding design. However, the purpose of big data compression is to compress the data to a smaller size, transmit it, and then decompress it on the other side. If the neural network structure is too complex, that is, if there are many parameters, it will take a long time to compress and decompress it.
[0043] The embodiment of the present invention provides a big data compression method based on neural network, such as Figure 1 Shown, including:
[0044] Step S101: Perform arithmetic coding on each data to be compressed, and obtain the initial weight of each data according to the similarity between the coded data and other coded data and the types of symbols contained in the data.
[0045] Step S102: construct an autoencoding network, which includes an input layer, an output layer, and at least one hidden layer. The number of neurons in the input layer and the output layer is the same and greater than the number of neurons in the hidden layer.
[0046] Step S103: Use the encoded data as both the input and output of the autoencoder network, use the initial weight of each data as the initial weight of the neuron corresponding to each data, and train the autoencoder network to obtain the weight matrix of the encoding network.
[0047] Step S104: Using the degree of influence of neurons in the autoencoder network on the entropy of the weight matrix as the importance value of the neurons, the autoencoder network is pruned, and during the network pruning process, redundant neurons and connections are deleted according to the importance values of the neurons and the weight values of the connections until at least one of the compression rate and the accuracy rate of the autoencoder network is outside the corresponding preset threshold range.
[0048] Step S105: Using the data corresponding to the hidden layer in the autoencoder network after network pruning as compressed data.
[0049] The main purpose of the present invention is to construct an autoencoding network and compress the autoencoding network according to the characteristics of the data, thereby obtaining a targeted autoencoding network and a compressed combing, and facilitating the subsequent use of the autoencoding network for a more efficient big data compression process.
[0050] Furthermore, step S101 performs arithmetic coding on each data to be compressed, and obtains the initial weight of each data according to the similarity between the coded data and other coded data and the types of symbols contained in the data. Specifically, it includes:
[0051] Arithmetic coding is a type of entropy coding, a lossless compression method with a compression ratio of approximately 2:1 to 5:1. This method clearly cannot meet the needs of large data encoding. The result of arithmetic coding is an interval, and any number in the interval is selected as the encoding result of the symbol sequence.
[0052] Each data item contains at least one symbol. For example, if there are five different symbols, each symbol is contained in different data items. The width of each symbol after encoding represents the frequency of each symbol; that is, the greater the frequency, the greater the width after encoding. Using the arithmetic coding process, each data item is encoded separately to obtain the encoded data item.
[0053] Big data, or massive data, refers to information so large in scale that it cannot be captured, managed, processed, and organized into information that helps businesses make more proactive business decisions within a reasonable timeframe using current mainstream software tools.
[0054] Due to the huge data scale of big data, a large number of neurons are required in the input and output layers when constructing an autoencoding network. However, when the number of input neurons is large, the training time of the autoencoding network will be longer, and more layers are required to achieve good encoding results, which further increases the training time of the autoencoding network and reduces the efficiency of the data compression process.
[0055] Optionally, before performing arithmetic coding on each data point to be compressed, outliers can be removed from each data point. It should be noted that outliers are unreasonable values in the data, and unreasonable values are values that deviate from the normal range, not values that are erroneous. Outliers in a dataset may be caused by sensor failure, manual input errors, or abnormal events. This can produce more accurate data, effectively reducing the workload of subsequent processing, and thus shortening the training time of the autoencoder network.
[0056] It should be noted that the method for handling outliers includes: judging whether a specific value is within a reasonable range based on the maximum and minimum values of the value, and using box plots to screen and handle outliers.
[0057] Specifically, a box plot can be used to observe the overall distribution of data, using statistics such as the median, 25% quantile, 75% quantile, upper bound, and lower bound to describe the overall distribution of the data. By calculating these statistics, a box plot is generated. The box contains most of the normal data, and the values or symbols outside the upper and lower bounds of the box are outliers in the data.
[0058] First, the similarity of each encoded data is obtained based on the mean of the similarity between the encoded data and other encoded data. The similarity calculation process of any two different encoded data includes:
[0059] The similarity of the encoding results is calculated based on the number of decimal places of the encoding results and the numerical difference of the encoding results. Among them, s ij is the similarity between the encoded i-th data and the encoded j-th data, a i is the number of decimal places of the ith data after encoding, a j is the number of decimal places of the jth data after encoding, b i is the value of the i-th data after encoding, b j The value of the j-th data after encoding, i, j are positive integers not greater than n, and i ≠ j, n is the number of encoded data.
[0060] It should be noted that the greater the similarity of the encoded data, the greater the possibility that the data can be replaced. The greater the similarity, the greater the redundancy. The amount of information of the data can be represented by other data or a combination of other data, so its importance is lower. Conversely, its importance is greater.
[0061] Secondly, all the types of matches contained in all the data are counted to obtain the types of matches of all the data, and the ratio of the symbol types contained in each match to the symbol types of all the data is taken as the symbol proportion of each data.
[0062] It should be noted that the proportion of the number of symbols contained in each piece of data to the total number of symbols is calculated. The larger the proportion, the more indispensable the data is, that is, the more important the data is.
[0063] Finally, the ratio of the symbol proportion of the data to the similarity of the data is used as the initial weight of the data.
[0064] Furthermore, in step S102, an autoencoding network is constructed, which includes an input layer, an output layer, and at least one hidden layer. The number of neurons in the input layer and the output layer is the same and greater than the number of neurons in the hidden layer. Specifically, the following steps are performed:
[0065] It should be noted that the autoencoder is a special neural network. The number of neurons in the hidden layer of the network is less than the number of input neurons. The output of the network is the same as the input. In the embodiment of the present invention, the hidden layer in the autoencoder network is used as the compressed result.
[0066] Autoencoding networks are often used for data compression. In order to achieve better compression effects, autoencoding networks generally require a deeper network structure. However, a deeper network structure will result in longer training time when compressing big data. Therefore, the embodiment of the present invention calculates the weights of neurons in the autoencoding network and prunes the autoencoding network according to the weights of the input neurons, thereby removing redundant neurons or nodes in the autoencoding network, obtaining a more targeted autoencoding network, and thus improving the efficiency of data compression.
[0067] Figure 2 FIG. 4 shows a schematic diagram of the structure of the autoencoding network in an embodiment of the present invention. Figure 2 As shown, an autoencoder network consists of an input layer, an output layer, and at least one hidden layer. The number of neurons in the input and output layers is the same, but greater than the number of neurons in the hidden layer, also known as the intermediate layer. This creates an initial autoencoder network, facilitating subsequent training and network pruning.
[0068] Furthermore, in step S103, the encoded data is used as both input and output of the autoencoder network, the initial weight of each data is used as the initial weight of the neuron corresponding to each data, and the autoencoder network is trained to obtain the weight matrix of the encoding network.
[0069] It should be noted that the encoded data is used as both the input and output of the autoencoding network in order to test the accuracy of the data decompressed after being processed by the autoencoding network.
[0070] Secondly, the initial weights of each data are used as the initial weights of the neurons corresponding to each data to train the autoencoder network. This makes it easier to prune the autoencoder network in the subsequent process to simplify the network structure of the autoencoder network while improving processing efficiency.
[0071] Furthermore, in step S104, the degree of influence of neurons in the autoencoding network on the entropy of the weight matrix is used as the importance value of the neurons, and the autoencoding network is pruned. During the network pruning process, redundant neurons and connections are deleted according to the importance values of the neurons and the weight values of the connections in the weight matrix until at least one of the compression rate and accuracy of the autoencoding network is outside the corresponding preset threshold range.
[0072] First, the influence of neurons in the autoencoder network on the entropy of the weight matrix is taken as the importance value of the neuron.
[0073] Specifically, obtaining the degree of influence of the neuron on the entropy of the weight matrix includes taking the entropy of the weight matrix of the autoencoder network when the neuron exists in the autoencoder network as the first entropy value, setting the value corresponding to the neuron in the weight matrix to 0 to obtain the entropy of the weight matrix as the second entropy value, and taking the absolute value of the difference between the first entropy value and the second entropy value as the degree of influence of the neuron on the entropy of the weight matrix.
[0074] Secondly, the autoencoder network is pruned, and during the network pruning process, redundant neurons and connections are deleted according to the importance values of neurons and the weight values of connections in the weight matrix until at least one of the compression rate and accuracy of the autoencoder network is outside the corresponding preset threshold range.
[0075] Specifically, during the network pruning process, redundant neurons and connections are deleted according to the importance values of neurons and the weight values of connections in the weight matrix, including:
[0076] The connection corresponding to the minimum weight value in the weight matrix in the autoencoder network is deleted, and the autoencoder network after the deletion of the connection is retrained; when the set of input connections of multiple neurons is included in the set of input connections of any neuron, the neuron with the smallest importance value among the multiple neurons is deleted, and the autoencoder network after the deletion of the neuron is retrained; neurons without output connections or output connections in the trained autoencoder network are deleted, and the autoencoder network is retrained. Connections or neurons are deleted and the autoencoder network is retrained until at least one of the compression rate and accuracy of the autoencoder network is outside the corresponding preset threshold range.
[0077] It should be noted that unimportant connections or neurons are pruned in order of weight value from small to large. After pruning a connection, if there is a neuron without input connection or output connection, it means that this neuron is no longer effective for the entire model and can be deleted directly.
[0078] At the same time, the compression accuracy can be obtained by comparing the output of the autoencoder network with the input results. When the compression accuracy is outside the preset accuracy threshold, it indicates that the previously deleted input neurons need to be retained to ensure accuracy. Similarly, when the compression rate of the autoencoder network is outside the preset compression rate threshold, it indicates that the previously deleted input neurons need to be retained to ensure accuracy and then retrain the neural network to improve the accuracy of the pruned model. The network pruning and retraining process is repeated until at least one of the network's compression rate or accuracy falls below the corresponding threshold. The neural network before the neuron deletion is selected as the trained autoencoder network.
[0079] Furthermore, in step S105, the data corresponding to the hidden layer in the autoencoder network after network pruning is used as compressed data.
[0080] After obtaining the pruned autoencoder network, the data corresponding to the hidden layer in the obtained autoencoder network is transmitted as compressed data.
[0081] Based on the same inventive concept as the above method, this embodiment also provides a big data compression system based on neural network, such as Figure 3 Shown, including:
[0082] The arithmetic coding module 201 is used to perform arithmetic coding on each data to be compressed.
[0083] The initial weight acquisition module 202 is used to obtain the initial weight of each data according to the similarity between the encoded data and other encoded data and the types of symbols contained in the data.
[0084] The autoencoding network construction module 203 is used to construct an autoencoding network, which includes an input layer, an output layer and at least one hidden layer. The number of neurons in the input layer and the output layer is the same and greater than the number of neurons in the hidden layer.
[0085] The weight matrix acquisition module 204 is used to use the encoded data as the input and output of the autoencoding network, use the initial weight of each data as the initial weight of the neuron corresponding to each data, and train the autoencoding network to obtain the weight matrix of the encoding network.
[0086] The network pruning module 205 is used to prune the autoencoder network based on the influence of the neurons in the autoencoder network on the entropy of the weight matrix as the importance value of the neurons. During the network pruning process, redundant neurons and connections are deleted according to the importance values of the neurons and the weight values of the connections in the weight matrix until at least one of the compression rate and the accuracy rate of the autoencoder network is outside the corresponding preset threshold range.
[0087] The compressed data acquisition module 206 is configured to use the data corresponding to the hidden layer in the autoencoder network after network pruning as compressed data.
[0088] An embodiment of the present invention also provides a storage medium for big data compression based on a neural network, wherein the storage medium stores computer instructions, and the instructions are executed to satisfy the big data compression method based on a neural network described in any of the above specific embodiments.
[0089] In summary, the embodiments of the present invention propose a big data compression method, system and storage medium based on neural networks, which compress the neural network model while combining the distribution characteristics of the data to ensure that the data compression ratio meets the requirements while reducing the compression time.
[0090] The words "including," "comprising," "having," and the like in this disclosure are open-ended words, meaning "including but not limited to," and are used interchangeably therewith. The words "or" and "and" used herein mean the words "and / or" and are used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to," and is used interchangeably therewith.
[0091] It should also be noted that in the method and system of the present invention, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0092] The above embodiments are merely examples for clarity of description and do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description, and it is not necessary or possible to enumerate all embodiments here. Any designs that are identical or similar to the present invention fall within the scope of protection of the present invention.
Claims
1. A big data compression method based on neural network, characterized in that: include: Perform arithmetic coding on each data to be compressed, and obtain the similarity of each encoded data based on the mean of the similarity between the encoded data and other encoded data; perform statistics on all the types of coincidences contained in all the data to obtain the types of coincidences of all the data, and use the ratio of the types of symbols contained in each coincidence to the types of symbols of all the data as the symbol proportion of each data; and use the ratio of the symbol proportion of the data to the similarity of the data as the initial weight of the data; Constructing an autoencoding network, the autoencoding network comprising an input layer, an output layer, and at least one hidden layer, wherein the number of neurons in the input layer and the output layer is the same and greater than the number of neurons in the hidden layer; The encoded data is used as the input and output of the autoencoder network at the same time, the initial weight of each data is used as the initial weight of the neuron corresponding to each data, and the autoencoder network is trained to obtain a weight matrix of the encoding network; The influence of neurons in the autoencoder network on the entropy of the weight matrix is used as the importance value of the neurons, and the autoencoder network is pruned. During the network pruning process, redundant neurons and connections are deleted according to the importance values of the neurons and the weight values of the connections in the weight matrix until at least one of the compression rate and the accuracy rate of the autoencoder network is outside the corresponding preset threshold range; The data corresponding to the hidden layer in the autoencoder network after network pruning is used as compressed data.
2. The big data compression method based on neural network according to claim 1, characterized in that: Prune the autoencoder network and remove redundant neurons and connections according to the importance of neurons and the weights of connections in the weight matrix. This includes: Delete the connection corresponding to the minimum weight value in the weight matrix in the autoencoder network, and retrain the autoencoder network after deleting the connection; When the set of input connections of multiple neurons is included in the set of input connections of any neuron, the neuron with the smallest importance value among the multiple neurons is deleted, and the autoencoder network after the deleted neuron is retrained; The neurons without output connections or with output connections in the trained autoencoder network are deleted, and the autoencoder network is retrained.
3. The big data compression method based on neural network according to claim 1, characterized in that: The influence of neurons in the autoencoder network on the entropy of the weight matrix is taken as the importance value of the neuron, including: The entropy of the weight matrix of the autoencoder network when the neuron exists in the autoencoder network is used as the first entropy value, the entropy of the weight matrix obtained by setting the value corresponding to the neuron in the weight matrix to 0 is used as the second entropy value, and the absolute value of the difference between the first entropy value and the second entropy value is used as the degree of influence of the neuron on the entropy of the weight matrix.
4. The big data compression method based on neural network according to claim 1, characterized in that: Before performing arithmetic coding on each data to be compressed, the method further comprises processing abnormal symbols in each data respectively.
5. The big data compression method based on neural network according to claim 1, characterized in that: The abnormal symbols in each data are processed separately through box plots.
6. The big data compression method based on neural network according to claim 3, characterized in that: Methods for obtaining similarity between encoded data and other encoded data include: Among them, S i,j is the similarity between the encoded i-th data and the encoded j-th data, a i is the number of decimal places of the i-th data after encoding, a j is the number of decimal places of the jth data after encoding, b i is the value of the i-th data after encoding, b j The value of the j-th data after encoding, i, j are positive integers not greater than n, and i≠j, n is the number of encoded data.
7. A neural network-based big data compression system, implementing the method as claimed in claim 1, comprising: an arithmetic coding module, used for performing arithmetic coding on each data to be compressed; An initial weight acquisition module is used to obtain the initial weight of each data according to the similarity between the encoded data and other encoded data and the types of symbols contained in the data; An autoencoding network construction module is used to construct an autoencoding network, wherein the autoencoding network includes an input layer, an output layer, and at least one hidden layer, wherein the number of neurons in the input layer and the output layer is the same and greater than the number of neurons in the hidden layer; A weight matrix acquisition module is used to use the encoded data as both the input and output of the autoencoding network, use the initial weight of each data as the initial weight of the neuron corresponding to each data, and train the autoencoding network to obtain the weight matrix of the encoding network; a network pruning module for pruning the autoencoder network using the influence of neurons on the entropy of the weight matrix as the importance value of the neurons, and deleting redundant neurons and connections in the network pruning process according to the importance values of the neurons and the weight values of the connections in the weight matrix until at least one of the compression rate and accuracy of the autoencoder network is outside a corresponding preset threshold range; The compressed data acquisition module is used to use the data corresponding to the hidden layer in the autoencoder network after network pruning as compressed data.
8. A computer-readable storage medium, characterized in that A program is stored which can be loaded and executed by a processor to implement the neural network-based big data compression method as described in any one of claims 1 to 6.
Citation Information
Patent Citations
KR20200115239A