Data compression and storage system based on Internet mass data management

Abstract

The invention belongs to the field of data compression and storage, relates to a data processing technology, and aims to solve the problem that a proper data compression mode cannot be selected according to the data type of a data packet to be compressed in the prior art, in particular to a data compression and storage system based on Internet mass data management. The server is in communication connection with a type management module, an encryption management module, a critical analysis module, a compression processing module and a database; the type management module is used for managing and analyzing different types of data storage states in the data marking a data compression packet corresponding to the management object as a subordinate object of the management object, generating a management period, and obtaining a management coefficient of the management object at the end moment of the management period; different types of to-be-compressed data packets are subjected to encryption compression mode matching in combination with the marking condition of the lossless object and the decision critical value of the deep analysis object, and the overall processing efficiency is improved while the data compression integrity is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of data compression storage and relates to data processing technology, and in particular to a data compression storage system based on Internet mass data management. Background Art

[0002] Data compression plays a vital role in database management systems. Through data compression, the storage space occupied by the database can be reduced, query efficiency can be improved, and the cost of data transmission and backup can be reduced. In data compression, data is encoded using fewer bits than the original data. Data compression methods generally include lossy data compression and lossless data compression.

[0003] The invention patent with announcement number CN108304472A discloses a data compression storage method and a data compression storage device. The data compression method can adopt different compression methods in consideration of different data contents, and can effectively improve the data compression efficiency. Compared with general data compression tools such as GZIP and SNAPPY, the data compression rate is significantly improved; however, the data compression method cannot select a suitable data compression method according to the data type of the data packet to be compressed, and cannot ensure data security and compression processing efficiency at the same time.

[0004] In view of the above technical problems, this application proposes a solution. Summary of the invention

[0005] The purpose of the present invention is to provide a data compression storage system based on Internet mass data management, which is used to solve the problem that the existing technical method cannot select a suitable data compression method according to the data type of the data packet to be compressed;

[0006] The technical problem to be solved by the present invention is: how to provide a data compression storage system based on Internet mass data management that can select a suitable data compression method according to the data type of the data packet to be compressed.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A data compression storage system based on Internet mass data management includes a server, wherein the server is communicatively connected with a type management module, an encryption management module, a critical analysis module, a compression processing module and a database;

[0009] The type management module is used to manage and analyze the storage status of different types of data in the database: mark the data type of the data compression package stored in the database before encryption as a management object, mark the data compression package corresponding to the management object as a subordinate object of the management object, generate a management cycle, obtain the management coefficient GL of the management object at the end of the management cycle, and send the management coefficients GL of all management objects to the encryption management module through the server;

[0010] The encryption management module is used to manage and analyze the encryption compression method of the compressed data packet and mark the management object as a lossless object or a deep analysis object;

[0011] The critical analysis module is used to analyze the decision critical value of the deep analysis object; the decision critical value of the deep analysis object is sent to the database through the server for storage;

[0012] The compression processing module is used to perform encryption compression processing on different types of data packets to be compressed.

[0013] Furthermore, the process of obtaining the management coefficient GL of the management object includes: obtaining the frequency modulation value DP, loss value DS and recovery value ZH of the management object; obtaining the management coefficient GL of the management object during the management period through the formula GL=k1*DS+k2*DS-k3*ZH, wherein k1, k2 and k3 are all proportional coefficients, and k1>k2>k3>1.

[0014] Furthermore, the process of obtaining the frequency modulation value DP includes: obtaining the number of calls of the subordinate objects of the management object within the management cycle, summing up the number of calls of all subordinate objects corresponding to the management object and taking the average value to obtain the frequency modulation value DP of the management object; the process of obtaining the loss value DS includes: obtaining the ratio of the number of lost bytes of the subordinate objects of the management object during encryption processing to the number of bytes of the data packet before encryption and marking it as the lost data of the subordinate objects, summing up the lost data of all subordinate objects corresponding to the management object and taking the average value to obtain the lost value DS of the management object; the process of obtaining the retrieval value ZH includes: obtaining the number of data retrieval requests of the subordinate objects of the management object within the management cycle and marking it as the retrieval data of the subordinate objects, summing up the retrieval data of all subordinate objects corresponding to the management object and taking the average value to obtain the retrieval value ZH of the management object.

[0015] Furthermore, the specific process of marking a management object as a lossless object or a deep analysis object includes: obtaining the management threshold GLmax through the database, and comparing the management coefficient GL of the management object within the management period with the management threshold GLmax: if the management coefficient GL is less than the management threshold GLmax, the management object is marked as a lossless object; if the management coefficient GL is greater than or equal to the management threshold GLmax, the corresponding management object is marked as a deep analysis object, and the deep analysis object is sent to the critical analysis module through the server.

[0016] Furthermore, the process of obtaining the decision critical value includes: marking all subordinate objects of the deep analysis objects that complete the data encryption, compression and storage process within the management cycle as monitoring objects, marking the data encryption, compression and storage process performed by the monitoring objects within the management cycle as monitoring processes, marking the number of bytes of data packets of the monitoring objects before executing the monitoring process as the volume value of the monitoring objects, marking the execution time of the monitoring process corresponding to the monitoring objects as the execution value, marking the ratio of the volume value to the execution value as the processing coefficient of the monitoring object and the monitoring process, forming a processing set composed of the processing coefficients corresponding to the monitoring processes of all the encryption and compression methods for lossless analysis objects, and performing comparison and analysis on the processing sets to obtain the decision critical value.

[0017] Furthermore, the specific process of comparative analysis of the processing set includes: performing variance calculation on the processing set to obtain a processing deviation value, obtaining a processing deviation threshold through a database, and comparing the processing deviation value with the processing deviation threshold: if the processing deviation value is less than the processing deviation threshold, then the minimum value of the monitoring process volume values ​​corresponding to all elements in the processing set is marked as the decision critical value; if the processing deviation value is greater than or equal to the processing deviation threshold, then the minimum element and the maximum element in the processing set are eliminated, and then the processing deviation value of the processing set is recalculated, and so on, until the processing deviation value is less than the processing deviation threshold.

[0018] Furthermore, the specific process of performing encryption compression processing on different types of to-be-compressed data packets includes: marking the to-be-compressed data packet as a processing object, and determining whether the data type of the processing object is marked as a lossless object in the previous management cycle:

[0019] If yes, the object is directly encrypted and compressed using a lossless compression method;

[0020] If not, obtain the number of bytes in the data packet of the processing object and mark it as the processing value of the processing object, call the decision critical value of the previous management cycle, and compare the processing value with the decision critical value: if the processing value is less than or equal to the decision critical value, directly use lossless compression to encrypt and compress the processing object; if the processing value is greater than the decision critical value, directly use lossy compression to encrypt and compress the processing object.

[0021] Furthermore, the working method of the data compression storage system based on Internet mass data management includes the following steps:

[0022] Step 1: Perform management analysis on the storage status of different types of data in the database: mark the data type of the data compression package stored in the database before encryption as a management object, generate a management cycle, and obtain the management coefficient GL of the management object at the end of the management cycle;

[0023] Step 2: manage and analyze the encryption compression method of the compressed data packet, and mark the managed object as a lossless object or a deep analysis object through the management coefficient GL;

[0024] Step 3: Analyze the decision critical value of the lossy compression method: mark all subordinate objects of the deep analysis object that complete the data encryption compression storage process within the management cycle as monitoring objects, and calculate the execution value and volume value of all monitoring objects to obtain the decision critical value;

[0025] Step 4: Encrypt and compress different types of data packets to be compressed.

[0026] The present invention has the following beneficial effects:

[0027] 1. The type management module can manage and analyze the storage status of different types of data in the database, and comprehensively analyze and calculate multiple management parameters of the data type within the management cycle to obtain the management coefficient. The management coefficient can be used to provide feedback on the suitability of the data type for encryption and compression processing using lossless data compression;

[0028] 2. The encryption management module can manage and analyze the encryption compression method of the compressed data packet, mark the management object differently according to the management coefficient, and directly match the lossless data compression method for the management object when it is marked as a lossless object, so as to ensure the data security of the data packet with high data integrity requirements after encryption and compression;

[0029] 3. The critical analysis module can be used to analyze the decision critical value of the deep analysis object, and the volume value, execution value and processing coefficient of all monitored objects can be counted. Then, the decision critical value can be obtained based on the comparison analysis. When the deep analysis object is encrypted and compressed by lossy data compression, the encryption efficiency is significantly improved. Then, the encryption compression method of the deep analysis object can be rationally screened according to the decision critical value.

[0030] 4. Combine the marking of lossless objects with the decision threshold of deep analysis objects to match encryption compression methods for different types of data packets to be compressed, thereby improving overall processing efficiency while ensuring data compression integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 is a system block diagram of Embodiment 1 of the present invention;

[0033] Figure 2 This is a flow chart of the method of Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1: Figure 1 As shown, a data compression storage system based on Internet mass data management includes a server, and the server is communicatively connected with a type management module, an encryption management module, a critical analysis module, a compression processing module and a database.

[0036] The type management module is used to manage and analyze the storage status of different types of data in the database: the data type of the data compression package stored in the database before encryption is marked as a management object, the data compression package corresponding to the management object is marked as a subordinate object of the management object, a management cycle is generated, and the frequency modulation value DP, loss value DS and retrieval value ZH of the management object are obtained at the end of the management cycle. The process of obtaining the frequency modulation value DP includes: obtaining the number of calls of the subordinate objects of the management object within the management cycle, summing up the number of calls of all subordinate objects corresponding to the management object and taking the average value to obtain the frequency modulation value DP of the management object; the process of obtaining the loss value DS includes: obtaining the ratio of the number of lost bytes of the subordinate objects of the management object during encryption processing to the number of bytes of the data packet before encryption and marking it as the lost data of the subordinate objects, summing up the lost data of all subordinate objects corresponding to the management object and taking the average value to obtain the management The lost value DS of the object; the process of obtaining the recovery value ZH includes: obtaining the number of data retrieval requests of the subordinate objects of the management object within the management cycle and marking it as the recovery data of the subordinate objects, summing up the recovery data of all subordinate objects corresponding to the management object and taking the average value to obtain the recovery value ZH of the management object; obtaining the management coefficient GL of the management object within the management cycle through the formula GL=k1*DS+k2*DS-k3*ZH, wherein k1, k2 and k3 are all proportional coefficients, and k1>k2>k3>1; sending the management coefficients GL of all management objects to the encryption management module through the server; managing and analyzing the storage status of different types of data in the database, comprehensively analyzing and calculating multiple management parameters of the data type within the management cycle to obtain the management coefficient, and using the management coefficient to provide feedback on the suitability of the data type for encryption compression processing using a lossless data compression method.

[0037] The encryption management module is used to manage and analyze the encryption compression method of the compressed data packet: obtain the management threshold GLmax through the database, and compare the management coefficient GL of the management object within the management cycle with the management threshold GLmax: if the management coefficient GL is less than the management threshold GLmax, the management object is marked as a lossless object; if the management coefficient GL is greater than or equal to the management threshold GLmax, the corresponding management object is marked as a deep analysis object, and the deep analysis object is sent to the critical analysis module through the server; the management object is differentiated according to the management coefficient, and when the management object is marked as a lossless object, it is directly matched with a lossless data compression method to ensure the data security of the data packet with high data integrity requirements after encryption compression is completed.

[0038] The critical analysis module is used to analyze the decision critical value of the deep analysis object: all subordinate objects of the deep analysis object that complete the data encryption, compression and storage process within the management cycle are marked as monitoring objects, the data encryption, compression and storage process performed by the monitoring object within the management cycle is marked as the monitoring process, the number of bytes of the data packet of the monitoring object before executing the monitoring process is marked as the volume value of the monitoring object, the execution time of the monitoring process corresponding to the monitoring object is marked as the execution value, and the ratio of the volume value to the execution value is marked as the processing coefficient of the monitoring object and the monitoring process. The processing coefficients corresponding to all monitoring processes whose encryption and compression methods are lossy data compression constitute a processing set, and a comparative analysis is performed on the processing set: the variance of the processing set is calculated to obtain the processing deviation value, the processing deviation threshold is obtained through the database, and the processing deviation value is compared with the processing deviation threshold :If the processing deviation value is less than the processing deviation threshold, the minimum value of the volume value of the monitoring process corresponding to all elements in the processing set is marked as the decision critical value; if the processing deviation value is greater than or equal to the processing deviation threshold, the minimum element and the maximum element in the processing set are eliminated, and then the processing deviation value of the processing set is recalculated, and so on, until the processing deviation value is less than the processing deviation threshold; the decision critical value of the lossy compression method is sent to the database through the server for storage; the volume value, execution value and processing coefficient of all monitored objects are statistically analyzed, and then the decision critical value is obtained based on the comparison analysis. When the deep analysis object is encrypted and compressed by the lossy data compression method through the decision critical value, the encryption efficiency is significantly improved. The encryption compression method of the deep analysis object is rationally screened according to the decision critical value.

[0039] The compression processing module is used to perform encryption and compression processing on different types of data packets to be compressed: mark the data packet to be compressed as a processing object, and determine whether the data type of the processing object is marked as a lossless object in the previous management cycle: if so, directly use lossless compression to encrypt and compress the processing object; if not, obtain the number of bytes of the data packet of the processing object and mark it as the processing value of the processing object, call the decision critical value of the previous management cycle, and compare the processing value with the decision critical value: if the processing value is less than or equal to the decision critical value, directly use lossless compression to encrypt and compress the processing object; if the processing value is greater than the decision critical value, directly use lossy compression to encrypt and compress the processing object; combine the marking situation of the lossless object and the decision critical value of the deep analysis object to match the encryption and compression methods of different types of data packets to be compressed, while ensuring the integrity of data compression and improving the overall processing efficiency.

[0040] Embodiment 2: Figure 2 As shown, the data compression and storage method based on Internet mass data management includes the following steps:

[0041] Step 1: Perform management analysis on the storage status of different types of data in the database: mark the data type of the data compression package stored in the database before encryption as a management object, generate a management cycle, and obtain the management coefficient GL of the management object at the end of the management cycle;

[0042] Step 2: manage and analyze the encryption compression method of the compressed data packet, and mark the managed object as a lossless object or a deep analysis object through the management coefficient GL;

[0043] Step 3: Analyze the decision critical value of the lossy compression method: mark all subordinate objects of the deep analysis object that complete the data encryption compression storage process within the management cycle as monitoring objects, and calculate the execution value and volume value of all monitoring objects to obtain the decision critical value;

[0044] Step 4: Encrypt and compress different types of data packets to be compressed.

[0045] The data compression storage system based on the management of Internet mass data, when working, marks the data type of the data compression package stored in the database before encryption as a management object, generates a management cycle, and obtains the management coefficient GL of the management object at the end of the management cycle; manages and analyzes the encryption compression method of the compressed data packet, and marks the management object as a lossless object or a deep analysis object through the management coefficient GL; marks all subordinate objects of the deep analysis object that complete the data encryption, compression and storage process within the management cycle as monitoring objects, calculates the execution values ​​and volume values ​​of all monitoring objects to obtain the decision critical value; and encrypts and compresses different types of data packets to be compressed.

[0046] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

[0047] The above formulas are obtained by collecting a large amount of data for software simulation and selecting a formula close to the real value. The coefficients in the formula are set by technicians in this field according to the actual situation; such as: formula GL = k1*DS + k2*DS - k3*ZH; technicians in this field collect multiple groups of sample data and set corresponding management coefficients for each group of sample data; substitute the set management coefficients and the collected sample data into the formula, any three formulas constitute a three-variable linear equation group, screen the calculated coefficients and take the average, and obtain the values ​​of k1, k2 and k3 as 3.52, 2.84 and 2.23 respectively;

[0048] The size of the coefficient is to quantify each parameter to obtain a specific value for subsequent comparison. The size of the coefficient depends on the amount of sample data and the initial setting of the corresponding management coefficient for each group of sample data by technical personnel in this field; as long as it does not affect the proportional relationship between the parameter and the quantized value, such as the management coefficient is proportional to the value of the missing value.

[0049] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0050] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A data compression storage system based on Internet mass data management, characterized in that: It includes a server, wherein the server is communicatively connected with a type management module, an encryption management module, a critical analysis module, a compression processing module and a database; The type management module is used to manage and analyze the storage status of different types of data in the database: mark the data type of the data compression package stored in the database before encryption as a management object, mark the data compression package corresponding to the management object as a subordinate object of the management object, generate a management cycle, obtain the management coefficient GL of the management object at the end of the management cycle, and send the management coefficients GL of all management objects to the encryption management module through the server; The encryption management module is used to manage and analyze the encryption compression method of the compressed data packet and mark the management object as a lossless object or a deep analysis object; The critical analysis module is used to analyze the critical value of the decision of the deep analysis object; the critical value of the decision of the deep analysis object is sent to the database through the server for storage; The compression processing module is used to perform encryption compression processing on different types of data packets to be compressed.

2. The data compression storage system based on Internet mass data management according to claim 1 is characterized in that: The process of obtaining the management coefficient GL of the management object includes: obtaining the frequency modulation value DP, loss value DS and recovery value ZH of the management object; obtaining the management coefficient GL of the management object during the management period through the formula GL=k1*DS+k2*DS-k3*ZH, where k1, k2 and k3 are all proportional coefficients, and k1>k2>k3>1.

3. The data compression storage system based on Internet mass data management according to claim 2 is characterized in that: The process of obtaining the frequency modulation value DP includes: obtaining the number of calls of the subordinate objects of the management object within the management cycle, summing up the number of calls of all subordinate objects corresponding to the management object and taking the average value to obtain the frequency modulation value DP of the management object; the process of obtaining the loss value DS includes: obtaining the ratio of the number of lost bytes of the subordinate objects of the management object during encryption processing to the number of bytes of the data packet before encryption and marking it as the lost data of the subordinate objects, summing up the lost data of all subordinate objects corresponding to the management object and taking the average value to obtain the lost value DS of the management object; the process of obtaining the retrieval value ZH includes: obtaining the number of data retrieval requests of the subordinate objects of the management object within the management cycle and marking it as the retrieval data of the subordinate objects, summing up the retrieval data of all subordinate objects corresponding to the management object and taking the average value to obtain the retrieval value ZH of the management object.

4. The data compression storage system based on Internet mass data management according to claim 3 is characterized in that: The specific process of marking a management object as a lossless object or a deep analysis object includes: obtaining the management threshold GLmax through the database, comparing the management coefficient GL of the management object within the management period with the management threshold GLmax: if the management coefficient GL is less than the management threshold GLmax, the management object is marked as a lossless object; if the management coefficient GL is greater than or equal to the management threshold GLmax, the corresponding management object is marked as a deep analysis object, and the deep analysis object is sent to the critical analysis module through the server.

5. The data compression storage system based on Internet mass data management according to claim 4 is characterized in that: The process of obtaining the decision critical value includes: marking all subordinate objects of the deep analysis objects that complete the data encryption, compression and storage process within the management cycle as monitoring objects, marking the data encryption, compression and storage process performed by the monitoring objects within the management cycle as the monitoring process, marking the number of bytes of the data packet of the monitoring object before executing the monitoring process as the volume value of the monitoring object, marking the execution time of the monitoring process corresponding to the monitoring object as the execution value, marking the ratio of the volume value to the execution value as the processing coefficient of the monitoring object and the monitoring process, forming a processing set with the processing coefficients corresponding to the monitoring processes of all the encryption and compression methods for lossless analysis objects, and performing comparative analysis on the processing set to obtain the decision critical value.

6. The data compression storage system based on Internet mass data management according to claim 5 is characterized in that: The specific process of comparative analysis of the processing set includes: performing variance calculation on the processing set to obtain a processing deviation value, obtaining a processing deviation threshold through a database, and comparing the processing deviation value with the processing deviation threshold: if the processing deviation value is less than the processing deviation threshold, then the minimum value of the volume value of the monitoring process corresponding to all elements in the processing set is marked as the decision critical value; if the processing deviation value is greater than or equal to the processing deviation threshold, then the minimum element and the maximum element in the processing set are eliminated, and then the processing deviation value of the processing set is recalculated, and so on, until the processing deviation value is less than the processing deviation threshold.

7. The data compression storage system based on Internet mass data management according to claim 6 is characterized in that: The specific process of performing encryption and compression processing on different types of data packets to be compressed includes: marking the data packet to be compressed as a processing object, and determining whether the data type of the processing object is marked as a lossless object in the previous management cycle: If yes, the object is directly encrypted and compressed using a lossless compression method; If not, obtain the number of bytes in the data packet of the processing object and mark it as the processing value of the processing object, call the decision critical value of the previous management cycle, and compare the processing value with the decision critical value: if the processing value is less than or equal to the decision critical value, directly use lossless compression to encrypt and compress the processing object; if the processing value is greater than the decision critical value, directly use lossy compression to encrypt and compress the processing object.

8. The data compression storage system based on Internet mass data management according to any one of claims 1 to 7, wherein the working method of the data compression storage system based on Internet mass data management comprises the following steps: Step 1: Perform management analysis on the storage status of different types of data in the database: mark the data type of the data compression package stored in the database before encryption as a management object, generate a management cycle, and obtain the management coefficient GL of the management object at the end of the management cycle; Step 2: manage and analyze the encryption compression method of the compressed data packet, and mark the managed object as a lossless object or a deep analysis object through the management coefficient GL; Step 3: Analyze the decision critical value of the lossy compression method: mark all subordinate objects of the deep analysis object that complete the data encryption compression storage process within the management cycle as monitoring objects, and calculate the execution value and volume value of all monitoring objects to obtain the decision critical value; Step 4: Encrypt and compress different types of data packets to be compressed.

Images