A method and system for accelerating data compression performance

By monitoring binlog information and combining it with data processing rules, data is collected and processed in real time, the storage location is determined, and new and old data are integrated. This solves the problems of data redundancy and latency in existing technologies and achieves efficient and real-time data compression and updates.

CN123068422APending Publication Date: 2026-11-10HANGZHOU NEW WINDOWS INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611271310.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-20
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies lack effective mechanisms for data merging, dirty data removal, and structured compression and regularization in real-time big data synchronization, incremental data transfer in databases, and archiving and storing massive amounts of business data. This results in large amounts of redundant data entering the database, low storage compression efficiency, and a lack of batch optimization in traditional data processing workflows, which can easily lead to synchronization link blockage and data compression delays.

Method used

By monitoring the binlog information at the required data collection location and combining it with the data processing rules in the configuration information, binlog data is collected and processed in real time to determine the data storage location. While writing data at this location, the historical location is replaced. The new and old data are integrated by using data hash values ​​and storage methods, and zero-value data is removed to achieve efficient data compression and real-time updates.

Benefits of technology

It improves the efficiency and real-time performance of data compression, reduces redundant computation, enhances the flexibility and reliability of data updates, avoids errors caused by blindly removing or accumulating data with the same hash value, and improves the efficiency and accuracy of data storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068422A_ABST
    Figure CN123068422A_ABST
Patent Text Reader

Abstract

This invention relates to a method and system for accelerating data compression performance, belonging to the field of data processing technology. It includes: responding to a preset start signal to acquire configuration information and required data acquisition locations; acquiring binlog information in real time at the required data acquisition locations and analyzing it to obtain binlog data; performing data processing operations on the binlog data according to data processing rules in the configuration information to obtain data to be stored; acquiring an original data table and historical data acquisition locations; using the data to be stored as a search term to find the corresponding data storage location in the original data table; performing compression and writing operations on the data to be stored according to the data storage location to obtain an updated data table; and updating the historical data acquisition locations based on the required data acquisition location to obtain the updated data location. This invention improves the real-time performance and efficiency of data storage updates.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for accelerating data compression performance. Background Technology

[0002] In data processing scenarios such as real-time big data synchronization, incremental data flow in databases, and archiving and storing massive amounts of business data, real-time parsing, regularization and compression, storage, and rapid querying and retrieval of massive incremental data are the core foundation for ensuring data synchronization accuracy, storage utilization, and data service efficiency.

[0003] Currently, conventional data synchronization solutions in the industry simply complete log parsing and record-by-record entry, lacking effective data merging, dirty data removal, and structured compression and normalization mechanisms. This results in large amounts of redundant data entering the database and low storage compression efficiency. At the same time, traditional data processing workflows lack batch optimization and grouping and normalization capabilities. The amount of data written in a single batch is small, and the database interaction frequency is high, which can easily cause problems such as synchronization link blockage and data compression and landing delays.

[0004] Regarding the aforementioned technologies, in the current data transmission and processing process, data is usually collected using designated tools after receiving the corresponding signal, or data is collected periodically, processed, stored, and updated. This results in data relying on existing settings and not being able to select the appropriate processing method effectively. Therefore, there is still room for improvement in the real-time performance and efficiency of data storage and updates. Summary of the Invention

[0005] To improve the real-time performance and efficiency of data storage and updates, this invention provides a method and system for accelerating data compression performance.

[0006] In a first aspect, the present invention provides a method for accelerating data compression performance, employing the following technical solution: A method for accelerating data compression performance includes: Step 1: In response to the preset start signal, obtain configuration information and the required data collection location; Step 2: Collect binlog information in real time at the required data collection location and analyze it to obtain binlog data; Step 3: Perform data processing operations on the binlog data according to the data processing rules in the configuration information to obtain the data to be stored; Step 4: Obtain the original data table and historical data collection locations; Step 5: Use the data to be stored as a search term to find the corresponding data storage location in the original data table; Step 6: Perform a compressed write operation on the data to be stored according to the data storage location to obtain the updated data table; Step 7: Update the historical data collection locations based on the required data collection locations to obtain the updated data locations.

[0007] By adopting the above technical solution, binlog information is monitored at the demand data collection location to obtain binlog data. Then, the data to be stored is obtained by combining the data processing rules in the configuration information. The corresponding position is obtained in the data table, and the data is written to the position. At the same time, the demand data collection location is replaced with the historical position, which makes the data compression and update more comprehensive and improves the data compression efficiency and real-time performance.

[0008] Optionally, methods for performing data processing operations on binlog data according to the data processing rules in the configuration information to obtain the data to be stored include: Step 30: Extract field information from the binlog data; Step 31: Use the field information as search terms to find the corresponding data transformation type, and perform data transformation operations on the binlog data according to the data transformation type to obtain standard type data; Step 32: Perform data filtering on the standard data type to obtain standard data; Step 33: Analyze the field information to obtain the grouping identifiers corresponding to the standard data; Step 34: Select standard data with the same group identifier according to the preset continuity rules and integrate them to obtain the data of the same group; Step 35: Integrate the data in the same group according to the continuity rule to obtain the data to be stored.

[0009] By adopting the above technical solution, the corresponding data conversion type is found through field information to obtain standard type data, then filtered to obtain standard data, and finally the grouping identifier of the standard data is determined according to the field information and the standard data is integrated to obtain the data to be stored. This reduces the redundant calculation amount of data compression and improves the data compression efficiency.

[0010] Optionally, methods for obtaining updated data tables include: Step 60: Obtain storage requirement information; Step 61: Analyze the data to be stored based on the storage requirement information to obtain the data hash value; Step 62: Extract the storage method from the storage requirement information; Step 63: Search for the corresponding historical data in the original data table based on the data hash value and data storage location; Step 64: Integrate the data to be stored with historical data according to the storage method to obtain a comprehensive data table; Step 65: Remove zero-value data from the comprehensive data table according to the preset division rule to obtain the updated data table.

[0011] By adopting the above technical solution, the data hash value is obtained by storing the demand information and the storage method is extracted. The historical data in the original data table is retrieved by combining the hash value and the storage location. The new and old data are integrated according to the storage rules to obtain a comprehensive data table. After removing zero-value redundant data, an updated data table is obtained. This reduces the space occupied by invalid data in the data table and improves the data compression efficiency and the real-time performance of data updates.

[0012] Optionally, methods for integrating the data to be stored with historical data to obtain a comprehensive data table according to the storage method include: Step 640: Compare the data hash values ​​to find the same hash value; Step 6410: When the same hash value does not exist, integrate the data to be stored with the historical data according to the storage method to obtain a comprehensive data table; Step 6411: When the same hash value exists, determine the corresponding data type based on the data to be stored corresponding to the same hash value; Step 6412: Use data type as a search term to find the corresponding data correction range; Step 6413: Extract the data information from the data hash values ​​corresponding to the same hash value and compare them to obtain the information difference data; Step 64140: If the information difference data does not fall within the data correction range, integrate all the data to be stored corresponding to the same hash value with the historical data according to the storage method to obtain a comprehensive data table; Step 64141: If the information difference data falls within the data correction range, select the latest data based on the preset time rules and integrate it with historical data according to the storage method to obtain a comprehensive data table.

[0013] By adopting the above technical solution, the data hash values ​​are compared to find the same hash value. When the same hash value is found, the data correction range is matched with the data type. After comparing the data information to obtain the information difference data, the data difference is compared with the correction range to determine how to obtain the comprehensive data table. This avoids the problem of blindly removing or accumulating data with the same hash value, which may cause errors in subsequent data, and improves the reliability of data compression.

[0014] Optionally, methods for integrating all data to be stored corresponding to the same hash value with historical data according to storage method to obtain a comprehensive data table include: Step 641400: Use the data to be stored with the same hash value as the search term to find the corresponding historical comprehensive data; Step 641401: Analyze historical aggregate data to obtain historical difference data sets; Step 641402: If a historical difference data group exists, directly integrate all the data to be stored corresponding to the same hash value with the historical data according to the storage method to obtain a comprehensive data table; Step 641403: If the historical difference data group does not exist, store all the data to be stored corresponding to the same hash value to the data storage location according to the storage method to obtain the comprehensive data table.

[0015] By adopting the above technical solution, historical comprehensive data can be retrieved and analyzed using data to be stored with the same hash value to obtain historical difference data groups. Different data integration methods are used depending on whether historical difference data groups exist, which also avoids the problem of data errors caused by blind accumulation and improves the real-time performance and reliability of data updates.

[0016] Optionally, it also includes a method for obtaining an updated data table when the data storage location does not exist, the method including: Step 66: Obtain the historical hash value group; Step 67: Select any historical hash value from the historical hash value group and compare it with the data hash value to analyze the data existence status. The data existence status includes the historical existence status and the new existence status. Step 680: When the data exists in a historical state, use the historical hash value as a search term to find the corresponding historical data location; Step 681: Compress and write the data to be stored corresponding to the data hash value according to the storage method and historical data location to obtain the updated data table; Step 682: When the data exists in a new state, select a new historical hash value and execute steps 67 to 681 until the data hash value is compared with all historical hash values ​​to obtain the final state of the data, or until the data exists in a historical state. Step 683: When the final state of the data is still a newly existing state, define the data to be stored as new stored data and perform a compression write operation according to the storage method to obtain an updated data table.

[0017] By adopting the above technical solution, the existence status of data is determined by comparing historical hash value groups with data hash values. Corresponding compression writing methods are then performed on the historical existence status and the new existence status, which solves the problem of data writing failure caused by missing storage locations and improves the efficiency of data compression.

[0018] Optionally, methods for defining the data to be stored as new storage data and performing a compressed write operation according to the storage method to obtain an updated data table include: Step 6830: Obtain the data acquisition time; Step 6831: Use the data type as a search term to find the corresponding location and save the time; Step 6832: Use the data hash value as a search term to find the corresponding data deletion time and combine it with the location storage time to calculate the storage time range; Step 6833: If the data acquisition time falls within the storage time range, compress and write the data to be stored corresponding to the data hash value according to the storage method and historical data location to obtain the updated data table; Step 6834: If the data acquisition time does not fall within the storage time range, define the data to be stored as new storage data and perform a compression write operation according to the storage method to obtain the updated data table.

[0019] By adopting the above technical solution, the storage time range is calculated by combining the location storage time with the data deletion time. Then, it is analyzed whether the data acquisition time falls within the storage time range to perform differentiated compression writing. This allows the original location to be saved in a short time even when the data is 0, avoiding the need to obtain a new location every time it is new data, thus improving the efficiency of data compression.

[0020] Optionally, methods for integrating the data to be stored with historical data according to the storage method to obtain a comprehensive data table also include: Step 642: Obtain the historical adjacent data corresponding to the data to be stored and compare it with the data to be stored; Step 643: If the data to be stored is consistent with historical adjacent data and a preset abnormal signal is received, the data to be stored will be removed. Step 644: If the data to be stored is consistent with the historical adjacent data, but no abnormal signal is received, integrate the data to be stored with the historical data according to the storage method to obtain a comprehensive data table.

[0021] By adopting the above technical solution, the cause of duplicate data is distinguished by comparing the data to be stored with historical adjacent data and combining abnormal signals. Faulty duplicate data is removed and normal duplicate data is integrated, avoiding redundant duplicate data generated when the system is abnormal, reducing the participation of invalid data in compression operations, and improving the accuracy and efficiency of data compression and updates.

[0022] Optional, also includes: Step 8: Obtain standard report information; Step 9: Obtain the report requirement information and use it to determine the corresponding supplementary report information in the standard report information. The supplementary report information includes report template information and report field information. Step 10: Extract data viewing information from the report requirements information; Step 11: Filter and compress data in the updated data table according to the data viewing information; Step 12: Integrate the compressed data according to the report template information and report field information to form a data report and output it.

[0023] By adopting the above technical solution, the information to be supplemented in the report is determined by combining standard report information with report requirement information. Data viewing information is extracted and used to filter and update compressed data in the data table. The data report is generated and output by combining report template information and report field information, realizing the visual output of compressed data and improving the real-time performance and practicality of data updates.

[0024] Secondly, the present invention provides a data compression performance acceleration system, which adopts the following technical solution: A data compression performance acceleration system includes: The acquisition module is used to acquire configuration information, data collection locations, storage requirements, and report requirements. A memory for storing a program for a data compression performance acceleration method as described above; The processor loads and executes programs from memory.

[0025] By adopting the above technical solution, the data such as configuration information is collected uniformly by the acquisition module, the corresponding method program is stored in the memory, and the program is loaded and run by the processor, so as to realize the systematic and automated execution of the data compression acceleration method, improve the real-time performance and stability of the overall data compression processing, and the overall efficiency of the system's data processing.

[0026] In summary, the present invention has at least one of the following beneficial technical effects: 1. By listening to binlog information at the demand data collection location to obtain binlog data, and then combining it with the data processing rules in the configuration information to obtain the data to be stored, the corresponding position is obtained in the data table. While writing the data to this position, the demand data collection location is replaced with the historical position, making the data compression and update more comprehensive and improving the data compression efficiency and real-time performance. 2. By comparing the differences between data with the same hash value in the same batch, it can be determined whether the data is being updated or accumulated, thereby deciding how to generate a compressed table, which improves data compression efficiency and data update flexibility; 3. By analyzing the state of the data, determine whether to directly perform cumulative compression and storage at the corresponding location or to create new rows for compression and storage, thereby improving the data compression efficiency and real-time performance. Attached Figure Description

[0027] Figure 1 This is a flowchart of a data compression performance acceleration method according to an embodiment of this application; Figure 2 This is a flowchart of a method for integrating data to be stored with historical data according to a storage method to obtain a comprehensive data table, according to an embodiment of this application. Figure 3This is a flowchart of a method for obtaining an updated data table when the data storage location does not exist, according to an embodiment of this application. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0029] This invention discloses a method for accelerating data compression performance. (Refer to...) Figure 1 One method to accelerate data compression performance includes: Step 1: In response to the preset start signal, obtain configuration information and the location of required data collection.

[0030] The startup signal refers to the signal that starts the data compression acceleration system, enabling Binlog listening and data compression storage. The startup signal is activated by an electrical button on the system; pressing the button triggers the startup signal.

[0031] Configuration information refers to a collection of rules and configuration files that manage Binlog listening, data processing, and database writes. This configuration information is obtained during system startup when the configuration management component simultaneously reads and aggregates various configuration files and database configuration tables, such as application.properties, ns_canal_config_item, and ns_canal_config_field. The required data collection location refers to the reading point of the Binlog logs. This location is obtained during system startup by concatenating the source table name, database address, and other information to form a Redis key. The Binlog file and offset values ​​stored in Redis are then read to determine the required data collection location.

[0032] Step 2: Collect binlog information in real time at the required data collection location and analyze it to obtain binlog data.

[0033] Binlog information refers to the raw binary log data captured in real time by the data listening component from the MySQL database. The method for collecting this Binlog information is as follows: the system connects to MySQL through the data listening component and continuously captures the raw binary log content generated in real time according to the Binlog points read from Redis.

[0034] Binlog data refers to the data obtained by parsing the original Binlog information. Here, Binlog data is obtained by the system using a data monitoring component to perform binary decoding on the collected Binlog information, extracting the field information before and after changes, and event types, which are then encapsulated into a BinlogData object to obtain the Binlog data.

[0035] Step 3: Perform data processing operations on the binlog data according to the data processing rules in the configuration information to obtain the data to be stored.

[0036] Data processing rules refer to the pre-defined standard constraints for performing operations such as field extraction, type conversion, condition filtering, and grouping / merging on Binlog data. Examples include field mapping rules, data type conversion matching rules, multi-condition logical filtering rules, and continuous data merging rules. These rules are obtained by the system integrating the sourceField configuration, filtering configuration, and data merging configurations from the configuration information within the data processing component. Field mapping rules bind fields; data type conversion rules pre-define the matching relationships between standard data types such as text, numeric, and time and binlog fields; logical filtering rules support custom filtering thresholds and conditions using AND, OR, and SpEL expressions; and continuous rules define the time continuity threshold and data row continuity criteria for data in the same group.

[0037] Data processing operations refer to standard processing actions performed on Binlog data, such as field extraction and field type conversion. The execution method for these data processing operations is as follows: the system uses data processing components to sequentially perform specified field extraction, match field types for format conversion, filter valid data according to rules, and then merge data according to group identifiers and continuous Binlog requirements to execute the data processing operation. These rules include, for example, AND, OR, and SpEL. Specifically, the system matches data conversion types based on extracted field information to automatically convert original heterogeneous binlog fields to system-unified standard fields. Based on AND and OR multi-condition combination rules and SpEL expressions, custom filtering logic is used to remove invalid data such as null values, out-of-limit values, and abnormal garbled characters. The continuity of standard data within the same group is determined by combining time continuity thresholds to complete batch data integration. Data to be stored refers to the data that can be written to the compressed table after processing the Binlog data. This data to be stored is obtained by the system using data processing components to sequentially perform field extraction, field type conversion, compliance filtering, and merging of continuous Binlog data within the same group.

[0038] Step 4: Obtain the original data table and the location of historical data collection.

[0039] The raw data table refers to the compressed MySQL source business data table that needs to continuously write new business data. The raw data table is obtained by the system reading the `tableName` configuration item from the configuration information and splitting the comma-separated table name content to obtain the complete raw data table. The historical data collection position refers to the Binlog record point stored in Redis after the previous round of successful batch data insertion. The historical data collection position is obtained by the system constructing a Redis key based on the raw data table, MySQL host address, port, etc., and reading the log parameters stored within the key to obtain the historical data collection position.

[0040] Step 5: Use the data to be stored as a search term to find the corresponding data storage location in the original data table.

[0041] The data storage location refers to the location within the original data table where the data to be stored is matched, used for adding or accumulating data. The data storage location is found by the system generating a group identifier based on all character fields of the data to be stored and generating an MD5 primary key hash. This primary key hash is then used to retrieve the corresponding data storage row from the original data table to obtain the data storage location.

[0042] Step 6: Perform a compressed write operation on the data to be stored according to the data storage location to obtain an updated data table.

[0043] Compressed write operation refers to the batch writing of data to be stored into a compressed table according to the data storage location. Here, the compressed write operation is executed by the system using a data writing component to perform a batch upsert based on the primary key hash matching result. The InnoDB database engine automatically compresses the disk file and removes invalid rows with numeric fields of 0 to complete the compressed write operation. Updated data table refers to the compressed table obtained after the data compression write and invalid data removal are completed, with updated field contents and storage files. Here, the updated data table is obtained by the system completing the writing, updating, insertion, zero-value deletion, and underlying disk compression of the data to be stored.

[0044] Step 7: Update the historical data collection locations based on the required data collection locations to obtain the updated data locations.

[0045] The data update position refers to the new Binlog log read point updated to Redis after the Binlog data has been processed and stored in the database. The data update position is obtained by the system extracting the latest Binlog file in the current processing batch and its offset information, combining it with the original data table and MySQL address, and then concatenating the Redis key to overwrite the original point to obtain the data update position.

[0046] The methods for performing data processing operations on binlog data according to the data processing rules in the configuration information to obtain the data to be stored include: Step 30: Extract field information from binlog data.

[0047] Field information refers to the content of various business fields within the BinlogData object, such as basic fields, amount fields, and statistical fields. The field information is extracted by the system using a data processing component that, based on the target field list provided by sourceField in the configuration, filters the corresponding field names and values ​​from the BinlogData generated by parsing Binlog to obtain the field information.

[0048] Step 31: Use the field information as search terms to find the corresponding data conversion type, and perform data conversion operations on the binlog data according to the data conversion type to obtain standard type data.

[0049] Data conversion type refers to a predefined category that categorizes various fields into suitable formats, such as string, long, and BigDecimal. The data conversion type is retrieved by the system matching the categorization tags corresponding to the field information based on the `type` parameter of the `sourceField` in the configuration.

[0050] Data transformation operations refer to actions such as formatting and numerical conversion of field values ​​according to the data transformation type. Here, the data transformation operation is executed by the system based on the processing field corresponding to the data transformation type, uniformly formatting dates, converting values ​​to BigDecimal, replacing null values ​​with -1, and performing positive / negative conversions on UPDATE event values. Standard type data refers to standardized field data whose fields are formatted uniformly according to the data transformation type and adapted to subsequent requirements. Here, standard type data is obtained by the system processing the binlog data according to the matching data transformation type, handling formatting, numerical values, and null values ​​to obtain standard type data.

[0051] Step 32: Perform a data filtering operation on the standard type data to obtain standard data.

[0052] Data filtering refers to the validation process of selecting compliant and valid data and removing invalid data. The data filtering operation is performed by the system validating standard data types, such as AND, ORS, and Conditions, according to configured rules. Data is retained if it meets any OR condition, all AND conditions, or SpEL expressions. Simultaneously, the system differentiates between INSERT, UPDATE, and DELETE events and validates the corresponding fields to complete the data filtering operation. Standard data refers to standardized data that has undergone multi-dimensional filtering and validation and has been confirmed as valid and usable. Standard data is obtained by the system performing a complete filtering validation, retaining data that passes the validation, and removing data that does not meet the conditions.

[0053] Step 33: Analyze the field information to obtain the grouping identifier corresponding to the standard data.

[0054] Group identifiers are labels derived from character fields within standard data. These group identifiers are obtained by the system recognizing all character fields in the standard data, summarizing their values, and generating group identifiers from the complete content formed by all character fields.

[0055] Step 34: Select standard data with the same group identifier according to the preset continuous rules and integrate them to obtain the same group data.

[0056] A continuity rule refers to the constraint condition used to determine whether multiple standard data items with the same grouping identifier belong to a continuous event. The continuity rule is obtained by the system retrieving the relevant configuration for data merging from the configuration information and reading the judgment logic used to determine the temporal continuity of Binlog events.

[0057] Data in the same group refers to a standard set of data with consistent group identifiers and Binlog event time sequences that meet the continuity rule. This data in the same group is obtained by the system using continuity rules to filter and integrate standard data with consecutive time sequences under the same group identifier.

[0058] Step 35: Integrate the data in the same group according to the continuity rule to obtain the data to be stored.

[0059] The data to be stored is obtained by the system performing a merging process on the standard data in the same group according to the continuous rules. Character fields keep their original values ​​unchanged, numeric fields are summed, and fields such as project ID and Binlog time are updated to obtain the data to be stored.

[0060] The methods for obtaining updated data tables include: Step 60: Obtain storage requirement information.

[0061] Storage requirement information refers to the collection of various requirements for storing data in the database, such as storage mode and batch write limit. This storage requirement information is obtained by the system reading relevant configuration items such as write mode, hash generation, and division by zero rules from the configuration information to obtain complete storage requirement information.

[0062] Step 61: Analyze the data to be stored based on the storage requirements information to obtain the data hash value.

[0063] A data hash value is an MD5 identifier calculated from the character fields of the data to be stored. The data hash value is obtained by the system extracting all character fields of the data to be stored according to the hash generation rules in the storage requirements, concatenating them, and then calculating the MD5 hash value.

[0064] Step 62: Extract the storage method from the storage requirement information.

[0065] The storage method refers to the way the data to be stored is written into the original data table. Here, the storage method is extracted by the system reading information such as the write mode identifier and compression switch from the storage requirement information.

[0066] Step 63: Find the corresponding historical data in the original data table based on the data hash value and data storage location.

[0067] Historical data refers to old data stored at the data storage location corresponding to the data hash value in the original data table. The method for finding historical data here is for the system to locate the data storage location using the data hash value, retrieve the original data table, and obtain the original business data already stored in that row to obtain the historical data.

[0068] Step 64: Integrate the data to be stored with historical data according to the storage method to obtain a comprehensive data table.

[0069] A comprehensive data table refers to a data table formed after field fusion and value merging according to the storage method. Here, the comprehensive data table is obtained by the system executing the corresponding data fusion logic based on the storage method. If there is no historical data, the data to be stored is used directly. If historical data exists, character fields use the historical data, and numeric fields are overlaid with the data to be stored. All field contents are then integrated to obtain the comprehensive data table.

[0070] Step 65: Remove zero-value data from the comprehensive data table according to the preset division rule to obtain the updated data table.

[0071] A division-by-zero rule is a constraint that removes rows where the value of a numeric field is 0. Here, the division-by-zero rule is obtained by the system retrieving pre-configured zero-value determination logic from the stored requirement information.

[0072] The updated data table is obtained by the system traversing each row of the comprehensive data table, removing data entries where all numeric fields are equal to 0, and then combining the remaining data with the underlying database compression processing to obtain the updated data table.

[0073] Reference Figure 2 Methods for integrating the data to be stored with historical data to obtain a comprehensive data table according to the storage method include: Step 640: Compare the data hash values ​​to obtain the same hash value.

[0074] Same hash value refers to MD5 strings where the hash values ​​of the data to be stored are completely identical. The same hash value is obtained by comparing the hash values ​​of each data item to be stored; when the two hash values ​​match exactly, they are considered to have the same hash value.

[0075] Step 6410: When the same hash value does not exist, integrate the data to be stored with the historical data according to the storage method to obtain a comprehensive data table.

[0076] When the same hash value does not exist, it means that there is no possibility of duplicate data. Each hash value corresponds to a new data and is compressed and stored. Therefore, the data to be stored and historical data are integrated according to the storage method to obtain a comprehensive data table.

[0077] Step 6411: When the same hash value exists, determine the corresponding data type based on the data to be stored corresponding to the same hash value.

[0078] Data type refers to the category of data to be stored corresponding to the same hash value. The data type is determined by the system classifying the data to be stored with the same hash value according to preset configurations.

[0079] When the same hash value exists, it means that the same data exists. It is necessary to find out whether it is multiple collections accumulated at the same time or duplicate data with errors. Therefore, the corresponding data type is determined based on the data to be stored corresponding to the same hash value.

[0080] Step 6412: Use the data type as the search term to find the corresponding data correction range.

[0081] The data correction range refers to the allowable error range for data fluctuation under a corresponding data type. Different data types have different data correction ranges, which are pre-defined by professionals in the field based on experiments for each type of data and then input into the system. When the system receives a data type, it automatically retrieves the corresponding data correction range. The data correction range can be set using three standardized methods: fixed absolute value threshold, dynamic percentage threshold, and standard deviation fluctuation threshold. Configurations are differentiated according to the business data type: absolute value error range for measurement numerical data, percentage error range for statistical business data, and standard deviation fluctuation range for time-series data, etc. All correction ranges conform to industry data acquisition error standards, equipment acquisition accuracy thresholds, and experimental calibration of allowable business fluctuation ranges, and are stored in the system. For example, a ±0.05 absolute value correction range is set for routinely acquired numerical data, a ±3% percentage correction range for total statistical data, and a one-standard-deviation fluctuation correction range for continuous time-series data. The system can retrieve the corresponding judgment threshold based on the matched data type to obtain the data correction range.

[0082] Step 6413: Extract the data information from the data hash values ​​corresponding to the same hash value and compare them to obtain the information difference data.

[0083] Data information refers to the parameter information of the data to be stored corresponding to the same hash value. The data information is extracted by the system reading all the data to be stored corresponding to the same hash value and completely extracting all the collected content carried by each data entry. Information difference data refers to the set of parameter differences obtained by comparing the data to be stored corresponding to the same hash value with historical data. This information difference data is obtained by the system comparing the two sets of data item by item, filtering out information with inconsistent parameter values ​​and collected content, and extracting the corresponding differences.

[0084] Step 64140: If the information difference data does not fall within the data correction range, integrate all the data to be stored corresponding to the same hash value with the historical data according to the storage method to obtain a comprehensive data table.

[0085] The comprehensive data table is obtained by the system accumulating and merging all data to be stored with the same hash value and the historical data in the original data table according to the storage method. Character fields use the existing historical data content, and numerical fields are overlaid and summarized. All field information is integrated to obtain the comprehensive data table.

[0086] If the discrepancy in the data does not fall within the scope of data correction, it means that the presence of two or more copies of the data does not necessarily indicate that correction is required. It may indeed be normal data from two different instances. Therefore, all data to be stored corresponding to the same hash value are integrated with historical data according to the storage method to obtain a comprehensive data table.

[0087] Step 64141: If the information difference data falls within the data correction range, select the latest data based on the preset time rules and integrate it with historical data according to the storage method to obtain a comprehensive data table.

[0088] The time rule refers to the criteria for selecting the most recent data. Here, the time rule is obtained by pre-configuring a filtering logic based on the order of data collection. The system directly retrieves this preset rule, compares the data collection times, and selects the most recent valid data from the latest collection time.

[0089] The latest data refers to the data with the latest timestamp among the data to be stored with the same hash value. The latest data is selected by the system comparing the timestamps of all data to be stored with the same hash value according to time rules, and then filtering out the data with the latest collection time. The comprehensive data table is obtained by the system filtering out the latest data to be stored from multiple data entries according to time rules, and then merging this latest data with historical data according to the storage method. Character fields use the historical data, and numeric fields are merged and calculated to obtain the comprehensive data table.

[0090] If the discrepancy in the data falls within the data correction range, it means that the two sets of data exist because one set of data is incorrect and needs to be corrected. Therefore, the latest data is selected based on the preset time rules and integrated with the historical data according to the storage method to obtain a comprehensive data table.

[0091] Among them, the method of integrating all data to be stored corresponding to the same hash value with historical data according to the storage method to obtain a comprehensive data table includes: Step 641400: Use the data to be stored with the same hash value as the search term to find the corresponding historical comprehensive data.

[0092] Historical aggregated data refers to the set of all data in the original data table that match the current hash value. The method for finding historical aggregated data here is as follows: the system uses the data to be stored with the same hash value as the search criteria to retrieve the original data table, and simultaneously retrieves the final summary data of the hash matches, as well as the original scattered collected data that participated in the integration, to obtain the historical aggregated data.

[0093] Step 641401: Analyze historical aggregate data to obtain historical difference data sets.

[0094] Historical difference data sets refer to data sets extracted from historical aggregated data where the input compression discrepancies exceed the correction range. The analysis and acquisition method for historical difference data sets involves the system disassembling all original, scattered data entries from the historical aggregated data, and integrating data that were input simultaneously, or in the same batch, and whose discrepancies exceed the correction range to form historical difference data sets.

[0095] Step 641402: If a historical difference data group exists, directly integrate all the data to be stored corresponding to the same hash value with the historical data according to the storage method to obtain a comprehensive data table.

[0096] If a historical difference data group exists, it means that there was a previous instance of incomplete data collection, which was supplemented in the same batch. In this case, the same situation applies. Therefore, all the data to be stored corresponding to the same hash value should be integrated with the historical data according to the storage method to obtain a comprehensive data table.

[0097] Step 641403: If the historical difference data group does not exist, store all the data to be stored corresponding to the same hash value to the data storage location according to the storage method to obtain the comprehensive data table.

[0098] The comprehensive data table is obtained by the system merging each piece of data to be stored under the same hash value with the corresponding historical data one by one according to the storage method. Character fields use the corresponding historical content, and numerical fields are calculated independently. Multiple pieces of data are stored in the database at the same time to form a comprehensive data table.

[0099] If the historical difference data group does not exist, it means that there were no cases of supplementing in the same batch. In order to avoid blind accumulation and data errors, all data to be stored corresponding to the same hash value are stored in the data storage location according to the storage method to obtain a comprehensive data table.

[0100] Reference Figure 3 It also includes a method for obtaining an updated data table when the data storage location does not exist, the method including: Step 66: Obtain the historical hash value group.

[0101] Historical hash value groups refer to the collection of all previously stored and archived MD5 hash codes within the original data table. The historical hash value groups are obtained by the system iterating through the existing records of the original data table and extracting all complete hash codes of the stored data in batches.

[0102] Step 67: Select any historical hash value from the historical hash value group and compare it with the data hash value to analyze the data existence status.

[0103] Historical hash values ​​refer to the MD5 hash code of any record within a historical hash value group. The historical hash values ​​are selected randomly by the system from any hash code within the historical hash value group.

[0104] Data existence status refers to the classification of data persistence obtained by comparing hash codes and verifying the retention of data in the original data table. This includes historical existence status and new existence status. The analysis method for data existence status is that the system compares the current data hash value with the selected historical hash value. If they are the same, it is determined to be a historical existence status; if the hashes are inconsistent, or the hashes are consistent but the corresponding data has been deleted due to all values ​​being zeroed out, or the hash has never been stored in the data table, it is determined to be a new existence status. Historical existence status refers to a hash value that is completely consistent with the selected historical hash value, and the corresponding data exists in the original data table, without being removed due to division by zero. The determination method for historical existence status is that the system completes hash matching verification and simultaneously checks to confirm that the corresponding data has not been deleted due to division by zero; if both conditions are met, it is determined to be a historical existence status. New existence status refers to situations where the hash codes do not match, or the hashes are consistent but the corresponding data has been deleted due to division by zero, or the hash value has never appeared before. The method for determining a new existing state is as follows: if the hash comparison is different, the data matching the hash is deleted by dividing by zero, or there is no record in the database, it is determined to be a new existing state.

[0105] Step 680: When the data exists in a historical state, use the historical hash value as a search term to find the corresponding historical data location.

[0106] Historical data location refers to the storage address information of the data corresponding to the hash value within the original data table. The historical data location is found by the system using the currently matched historical hash value as the search keyword to locate the specific partition, row number, and storage path of the data corresponding to that hash value within the compressed table.

[0107] When the data exists in a historical state, it means that the data can be directly accumulated to the position corresponding to the historical hash value. At this time, the historical hash value can be used as a search term to find the corresponding historical data position.

[0108] Step 681: Compress and write the data to be stored corresponding to the data hash value according to the storage method and historical data location to obtain the updated data table.

[0109] The updated data table is obtained by the system retrieving the original data stored in the historical data location according to the storage method, merging the data to be stored with the original data, using the original historical data content for character fields, and summarizing and overlaying the numerical fields, and then compressing and writing the updated data table at the historical data location.

[0110] Step 682: When the data exists in a new state, select a new historical hash value and execute steps 67 to 681 until the data hash value is compared with all historical hash values ​​to obtain the final state of the data, or until the data exists in a historical state.

[0111] The final state of the data refers to the final state obtained after the current data hash value has been traversed and compared with all historical hash codes within the historical hash value group. The final state is obtained by the system traversing all historical hash values ​​to complete one round of comparison. If no hash match is found throughout the process, or if the data corresponding to a matching hash has been removed by dividing by zero, the final state is fixed as the new existing state. If a valid, undeleted hash value is matched during the traversal, the final state is fixed as the historical existing state. The final state of the data is obtained by locking the existing category after the traversal is completed.

[0112] When the data exists in a new state, it means that the hash value of the data to be stored is not related to the hash value being compared. Therefore, a new historical hash value is selected and steps 67 to 681 are executed until the data hash value is compared with all historical hash values ​​to obtain the final state of the data, or until the data exists in a historical state.

[0113] Step 683: When the final state of the data is still a newly existing state, define the data to be stored as new stored data and perform a compression write operation according to the storage method to obtain an updated data table.

[0114] New storage data refers to the data to be stored that has not matched a valid hash value after traversing all historical hash values. The updated data table is obtained by the system compressing the new storage data as a whole according to the storage method and writing it to the archived updated data table in a completely new storage path.

[0115] When the final state of the data is still a newly existing state, it means that the data is newly introduced or needs to be rearranged. Therefore, the data to be stored is defined as new storage data and a compression write operation is performed according to the storage method to obtain an updated data table.

[0116] The method of defining the data to be stored as new storage data and performing a compressed write operation according to the storage method to obtain an updated data table includes: Step 6830: Obtain the data collection time.

[0117] Data acquisition time refers to the standard timestamp of the business parameters collected by the acquisition terminal. The data acquisition time is obtained by the system reading the acquisition timestamp field inherent in the data to be stored and retrieving the standard time recorded by the device at the instant acquisition is completed.

[0118] Step 6831: Use the data type as a search term to find the corresponding location and save the time.

[0119] Location retention time refers to the maximum time that data can be stored in a given location. The method for finding the location retention time is as follows: different data types have different location retention times, which are set and entered into the system by professionals in the field according to actual needs. When the system receives a data type, it automatically retrieves the corresponding location retention time.

[0120] Step 6832: Use the data hash value as a search term to find the corresponding data deletion time and combine it with the location storage time to calculate the storage time range.

[0121] The data deletion time refers to the specific timestamp when data in the compressed table was removed due to its value reaching zero. The data deletion time is retrieved by the system using the data hash value as the search keyword to query the compressed table and obtain the deletion time associated with that hash value.

[0122] The retention time range refers to the valid time interval during which data is allowed to remain at the corresponding location when its value is 0. The retention time range is calculated by starting the timeline from the data deletion time based on the location retention time, obtaining the final location retention time threshold, and then extracting the time interval within this threshold.

[0123] Step 6833: If the data acquisition time falls within the storage time range, compress and write the data to be stored corresponding to the data hash value according to the storage method and historical data location to obtain the updated data table.

[0124] If the data acquisition time falls within the storage time range, it means that the dedicated location of the data is still saved and has not been occupied by other data. Therefore, at this time, the data to be stored corresponding to the data hash value is compressed and written according to the storage method and the historical data location to obtain the updated data table.

[0125] Step 6834: If the data acquisition time does not fall within the storage time range, define the data to be stored as new storage data and perform a compression write operation according to the storage method to obtain the updated data table.

[0126] If the data collection time does not fall within the storage time range, it means that the data has not been collected for a long time and its corresponding position has been filled by other data. Therefore, the data to be stored is defined as new storage data and a compression write operation is performed according to the storage method to obtain an updated data table.

[0127] The methods for integrating the data to be stored with historical data according to the storage method to obtain a comprehensive data table also include: Step 642: Obtain the historical adjacent data corresponding to the data to be stored and compare it with the data to be stored.

[0128] Historical adjacent data refers to historical data that, under the same hash value and sorted by collection time, is immediately preceding the collection time of the data to be stored. The method for obtaining historical adjacent data is as follows: the system searches the compressed table using the hash value and collection time of the current data to be stored as filtering criteria, sorts all historical data with the same hash value by timestamp, and extracts historical records whose time is immediately adjacent to the current collection time to obtain historical adjacent data.

[0129] Step 643: If the data to be stored is consistent with historical adjacent data and a preset abnormal signal is received, the data to be stored will be removed.

[0130] An abnormal signal refers to a signal indicating abnormal disturbance in the system's operation, such as a system restart or power outage. The abnormal signal is received by the system in real-time monitoring of alarm indicators in the hardware, program, and transmission channels. The system automatically triggers and receives the abnormal signal whenever it captures a list of anomalies pre-entered by personnel in the field. Examples of anomalies include: momentary power outages, hardware restarts, offline alarms from data acquisition devices, abnormal exits of data processing processes, and program freezes and restarts.

[0131] If the data to be stored is consistent with historical adjacent data and a preset abnormal signal is received, it indicates that the system has malfunctioned. The consistency of the data may be due to duplicate input, so the data to be stored will be removed.

[0132] Step 644: If the data to be stored is consistent with the historical adjacent data, but no abnormal signal is received, integrate the data to be stored with the historical data according to the storage method to obtain a comprehensive data table.

[0133] If the data to be stored is consistent with the historical adjacent data, but no abnormal signal is received, it means that the data may have been collected twice exactly. In this case, the data to be stored and the historical data are integrated according to the storage method to obtain a comprehensive data table.

[0134] If the data to be stored is inconsistent with the historical adjacent data, it means that the two data are completely different sets of data. In this case, normal compression and writing to the compression table is sufficient.

[0135] This also includes: Step 8: Obtain standard report information.

[0136] Standard report information refers to pre-configured and standardized report template information, including field formats, statistical definitions, and layout rules. This standard report information is obtained by the system retrieving pre-entered report template files from the local configuration library and fully reading the report's fixed fields, statistical rules, and format requirements to obtain the standard report information.

[0137] Step 9: Obtain the report requirement information and use it to determine the corresponding supplementary report information in the standard report information.

[0138] Report requirement information refers to the set of relevant requirements a user has when viewing a report. This report requirement information is obtained by the system receiving and integrating user input regarding their report viewing needs through the corresponding control terminal.

[0139] The "Report Information to be Supplemented" refers to the complete set of report-related content that requires data entry. This includes report template information and report field information. The method for determining this information is as follows: the system matches the user's report requirements, filters suitable layout frameworks and statistical rules from standard report information to obtain report template information, filters columns that need data entry to obtain report field information, and integrates the above to obtain the report information to be supplemented. The report template information refers to the overall report framework data filtered from standard report information. This information is determined by the system matching standard reports according to report requirements, extracting suitable layout structure, statistical logic, overall layout rules, etc., to obtain report template information. The report field information refers to the field-related information that requires data entry. This information is determined by the system combining the user's report requirements, extracting all field-related content such as field names, unit, and order of measured data and statistical values ​​from the matched standard reports to obtain report field information.

[0140] Step 10: Extract the data viewing information from the report requirements information.

[0141] Data viewing information is a collection of data-related information that needs to be viewed. The method for extracting this data viewing information is as follows: the system breaks down the report requirement information, separately filters out the time range, data type, filter conditions, hash number, and other information of the data to be viewed, and then integrates them to obtain the data viewing information.

[0142] Step 11: Filter the compressed data in the updated data table according to the data viewing information.

[0143] Compressed data refers to the data set stored in the updated data table. The compressed data is filtered by the system based on the time range, data type, hash number, and filter conditions specified in the data viewing information, searching the updated data table and matching all compressed and stored business data that meets the criteria.

[0144] Step 12: Integrate the compressed data according to the report template information and report field information to form a data report and output it.

[0145] A data report is a visual file obtained by filling compressed data into a report. Here, the data report is generated by the system arranging the compressed data according to the row and column layout and calculation formulas of the report template, and then filling in the various business values ​​and collected information into the corresponding positions according to the report fields. The data report is output by the system saving it as a preset format file, displaying it on the user terminal page, or downloading it locally.

[0146] Based on the same inventive concept, embodiments of the present invention provide a data compression performance acceleration system.

[0147] One of the data compression performance acceleration systems includes: The acquisition module is used to acquire configuration information, data collection locations, storage requirements, and report requirements. Memory used to store a program for a method to accelerate data compression performance; The processor loads and executes programs from memory.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0149] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for accelerating data compression performance, characterized in that, include: Step 1: In response to the preset start signal, obtain configuration information and the required data collection location; Step 2: Collect binlog information in real time at the required data collection location and analyze it to obtain binlog data; Step 3: Perform data processing operations on the binlog data according to the data processing rules in the configuration information to obtain the data to be stored; Step 4: Obtain the original data table and historical data collection locations; Step 5: Use the data to be stored as a search term to find the corresponding data storage location in the original data table; Step 6: Perform a compressed write operation on the data to be stored according to the data storage location to obtain the updated data table; Step 7: Update the historical data collection locations based on the required data collection locations to obtain the updated data locations.

2. The data compression performance acceleration method according to claim 1, characterized in that, The methods for performing data processing operations on binlog data according to the data processing rules in the configuration information to obtain the data to be stored include: Step 30: Extract field information from the binlog data; Step 31: Use the field information as search terms to find the corresponding data transformation type, and perform data transformation operations on the binlog data according to the data transformation type to obtain standard type data; Step 32: Perform data filtering on the standard data type to obtain standard data; Step 33: Analyze the field information to obtain the grouping identifiers corresponding to the standard data; Step 34: Select standard data with the same group identifier according to the preset continuity rules and integrate them to obtain the data of the same group; Step 35: Integrate the data in the same group according to the continuity rule to obtain the data to be stored.

3. The data compression performance acceleration method according to claim 2, characterized in that, Methods for obtaining updated data tables include: Step 60: Obtain storage requirement information; Step 61: Analyze the data to be stored based on the storage requirement information to obtain the data hash value; Step 62: Extract the storage method from the storage requirement information; Step 63: Search for the corresponding historical data in the original data table based on the data hash value and data storage location; Step 64: Integrate the data to be stored with historical data according to the storage method to obtain a comprehensive data table; Step 65: Remove zero-value data from the comprehensive data table according to the preset division rule to obtain the updated data table.

4. The data compression performance acceleration method according to claim 3, characterized in that, Methods for integrating data to be stored with historical data to obtain a comprehensive data table according to storage method include: Step 640: Compare the data hash values ​​to find the same hash value; Step 6410: When the same hash value does not exist, integrate the data to be stored with the historical data according to the storage method to obtain a comprehensive data table; Step 6411: When the same hash value exists, determine the corresponding data type based on the data to be stored corresponding to the same hash value; Step 6412: Use data type as a search term to find the corresponding data correction range; Step 6413: Extract the data information from the data hash values ​​corresponding to the same hash value and compare them to obtain the information difference data; Step 64140: If the information difference data does not fall within the data correction range, integrate all the data to be stored corresponding to the same hash value with the historical data according to the storage method to obtain a comprehensive data table; Step 64141: If the information difference data falls within the data correction range, select the latest data based on the preset time rules and integrate it with historical data according to the storage method to obtain a comprehensive data table.

5. The data compression performance acceleration method according to claim 4, characterized in that, Methods for integrating all data to be stored corresponding to the same hash value with historical data according to storage method to obtain a comprehensive data table include: Step 641400: Use the data to be stored with the same hash value as the search term to find the corresponding historical comprehensive data; Step 641401: Analyze historical aggregate data to obtain historical difference data sets; Step 641402: If a historical difference data group exists, directly integrate all the data to be stored corresponding to the same hash value with the historical data according to the storage method to obtain a comprehensive data table; Step 641403: If the historical difference data group does not exist, store all the data to be stored corresponding to the same hash value to the data storage location according to the storage method to obtain the comprehensive data table.

6. The data compression performance acceleration method according to claim 3, characterized in that, It also includes a method for obtaining an updated data table when the data storage location does not exist, the method including: Step 66: Obtain the historical hash value group; Step 67: Select any historical hash value from the historical hash value group and compare it with the data hash value to analyze the data existence status. The data existence status includes the historical existence status and the new existence status. Step 680: When the data exists in a historical state, use the historical hash value as a search term to find the corresponding historical data location; Step 681: Compress and write the data to be stored corresponding to the data hash value according to the storage method and historical data location to obtain the updated data table; Step 682: When the data exists in a new state, select a new historical hash value and execute steps 67 to 681 until the data hash value is compared with all historical hash values ​​to obtain the final state of the data, or until the data exists in a historical state. Step 683: When the final state of the data is still a newly existing state, define the data to be stored as new stored data and perform a compression write operation according to the storage method to obtain an updated data table.

7. The data compression performance acceleration method according to claim 6, characterized in that, Methods for defining the data to be stored as new storage data and performing a compressed write operation according to the storage method to obtain an updated data table include: Step 6830: Obtain the data acquisition time; Step 6831: Use the data type as a search term to find the corresponding location and save the time; Step 6832: Use the data hash value as a search term to find the corresponding data deletion time and combine it with the location storage time to calculate the storage time range; Step 6833: If the data acquisition time falls within the storage time range, compress and write the data to be stored corresponding to the data hash value according to the storage method and historical data location to obtain the updated data table; Step 6834: If the data acquisition time does not fall within the storage time range, define the data to be stored as new storage data and perform a compression write operation according to the storage method to obtain the updated data table.

8. The data compression performance acceleration method according to claim 3, characterized in that, The method of integrating the data to be stored with historical data to obtain a comprehensive data table according to the storage method also includes: Step 642: Obtain the historical adjacent data corresponding to the data to be stored and compare it with the data to be stored; Step 643: If the data to be stored is consistent with historical adjacent data and a preset abnormal signal is received, the data to be stored will be removed. Step 644: If the data to be stored is consistent with the historical adjacent data, but no abnormal signal is received, integrate the data to be stored with the historical data according to the storage method to obtain a comprehensive data table.

9. The data compression performance acceleration method according to claim 1, characterized in that, Also includes: Step 8: Obtain standard report information; Step 9: Obtain the report requirement information and use it to determine the corresponding supplementary report information in the standard report information. The supplementary report information includes report template information and report field information. Step 10: Extract data viewing information from the report requirements information; Step 11: Filter and compress data in the updated data table according to the data viewing information; Step 12: Integrate the compressed data according to the report template information and report field information to form a data report and output it.

10. A data compression performance acceleration system, characterized in that, include: The acquisition module is used to acquire configuration information, data collection locations, storage requirements, and report requirements. A memory for storing a program of a data compression performance acceleration method as described in any one of claims 1 to 9; The processor loads and executes programs from memory.