Efficient lossless data compression method and system based on parallel computing

By calling the JNI mechanism and parallel computing optimization algorithm in Java 8 and above services, the problem of data compression performance bottleneck in Java services is solved, 5-10 times the compression performance is improved, and API compatibility is maintained.

CN119420365APending Publication Date: 2025-02-11SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411306560.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-19
Publication Date
2025-02-11

AI Technical Summary

Technical Problem

Services in Java 8 and above have difficulties in using optimized compression libraries, which leads to inability to enjoy the benefits of performance improvements, especially when processing large amounts of data, compression speed becomes a bottleneck.

Method used

By calling the JNI mechanism of the underlying compression library function, parallel computing optimization algorithm is integrated to improve memory data compression rate and ensure API compatibility to improve data compression performance.

Benefits of technology

Under the same compression task, the compression performance is 5-10 times higher than that of the native Java class library, significantly improving data processing efficiency, and ensuring the compatibility of interface and data format with the native Java class library.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119420365A_ABST
    Figure CN119420365A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, in particular to an efficient lossless data compression method and system based on parallel computing, and the method comprises the following steps: calling a JNI mechanism of a bottom layer compression library function; integrating a parallel computing optimization algorithm; the memory data compression rate is improved; the API compatibility is ensured; testing and verifying; the efficient lossless data compression method and the efficient lossless data compression system based on parallel computing have the beneficial effects that the performance of a Java platform is improved by integrating a parallel computing optimization algorithm and transforming a compression library. Through actual testing, under the same compression task, the compression performance is improved by 5-10 times compared with the original Java class library, so that the data processing efficiency is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a high-efficiency lossless data compression method and system based on parallel computing. Background Art

[0002] Traditional lossless data compression technologies, such as zlib and gzip, perform well in many scenarios, but their compression speed becomes a bottleneck when processing large amounts of data. In particular, with the development of big data and high-speed networks, the demand for data compression speed is growing. However, some optimization methods are mostly concentrated at the library level and are limited by specific environments and API compatibility. Therefore, the market urgently needs a more general and efficient compression system that can make full use of computing resources, especially parallel computing capabilities, to adapt to the growing data processing needs.

[0003] Data compression technology plays a vital role in the computer field. It can effectively reduce the cost of data storage and transmission and is widely used in various scenarios, including network transmission, file transfer, database management, and operating systems.

[0004] Traditional data compression technologies are mainly divided into two categories: lossless compression and lossy compression. Lossless compression technology can ensure the integrity of data, that is, the original data can be restored after decompression; while lossy compression technology will discard some data information during the compression process to obtain a higher compression rate, but the original data cannot be completely restored after decompression.

[0005] In Java applications, commonly used compression tools include zip and gzip, as well as the compression function library zlib. These tools and libraries play an important role in processing HTTP requests, message queue services, and database reading and writing, and are especially indispensable in business scenarios involving big data transmission and storage.

[0006] In the past few decades, data compression technology has gone through several important development stages. From the 1950s to the 1980s, Claude Shannon founded information theory, which laid the theoretical foundation for data compression technology. During this period, many classic algorithms were born, such as Huffman coding and LZ series coding.

[0007] In 1989, Phil Katz launched the file archiving software PKZIP, which disclosed all the technical parameters of the zip format and the data compression algorithm deflate used. Since then, the zip format has become the de facto standard in the field of data compression.

[0008] In 1995, Jean-loup Gailly and Mark Adler implemented the compression library zlib based on the deflate algorithm in the gzip source code. zlib provides general compression / decompression capabilities for other scenarios through library function calls. The three data compression formats of DEFLATE, ZLIB, and GZIP were also included in the RFC standard in the same year.

[0009] In addition, with the continuous development of Internet technology, major Internet companies have successively opened up new compression algorithms and implementations, such as Apple's LZFSE, Google's Brotli, and Facebook's Zstandard. These algorithms have achieved varying degrees of improvement in compression speed and compression ratio.

[0010] However, despite the existence of many excellent compression algorithms and implementations, Java 8 and above services still have some difficulties in using optimized compression libraries, resulting in the inability to enjoy the benefits of performance improvement. Therefore, the present invention aims to solve the problem of data compression performance bottleneck in Java services to improve the compression rate and overall system performance. Summary of the invention

[0011] The purpose of the present invention is to provide an efficient lossless data compression method and system based on parallel computing to solve the problems raised in the above background technology.

[0012] To achieve the above object, the present invention provides the following technical solution: an efficient lossless data compression method based on parallel computing, the method comprising the following steps:

[0013] The JNI mechanism for calling the underlying compression library function;

[0014] Integrate parallel computing optimization algorithms;

[0015] Improve memory data compression rate;

[0016] Ensure API compatibility;

[0017] Test verification: Design and execute compatibility tests to verify the correctness and stability of the system and native Java class libraries when the interfaces are cross-used; conduct comprehensive performance tests to prove that the compression performance of the system is significantly improved by 5-10 times compared with the native Java class library under the same compression task; perform necessary system tuning based on the test results to ensure that the system can meet performance indicators and user needs in different application scenarios.

[0018] Preferably, the specific operations of the JNI mechanism for calling the underlying compression library function include:

[0019] Establish an interactive interface between the Java platform and the underlying compression library through the Java Native Interface (JNI) mechanism;

[0020] Implement JNI methods for transferring data between the Java platform and the underlying compression library;

[0021] Call the functions of the underlying compression library to perform data compression and decompression operations;

[0022] Ensure the stability and reliability of JNI method calls to maintain the overall performance and stability of the system.

[0023] Preferably, the specific operations of the integrated parallel computing optimization algorithm include:

[0024] Introducing Intel's ISA-L library to transform and optimize traditional zlib or other compression libraries;

[0025] Use the parallel computing optimization algorithm in the ISA-L library to accelerate the data compression and decompression process;

[0026] Apply SSE / AVX / AVX2 / AVX512 extended instruction sets to parallelize underlying functions and improve computing efficiency;

[0027] When optimizing algorithm design, ensure that the integrity and accuracy of data compression and decompression are not affected.

[0028] Preferably, the specific operations for improving the memory data compression rate include:

[0029] Based on the parallel computing optimization algorithm of integrated parallel computing optimization algorithm, the data in memory is efficiently compressed and decompressed;

[0030] Through experimental testing, the effectiveness of the optimization algorithm in improving data compression rate was verified, ensuring a 5-10 times performance improvement;

[0031] Perform performance analysis on memory data compression and decompression operations, and continuously optimize to explore potential performance improvements.

[0032] Preferably, specific operations to ensure API compatibility include:

[0033] During the optimization process, we maintain high compatibility with the native Java class library API, ensuring that users can migrate seamlessly without modifying existing code.

[0034] Ensure the compatibility of external interfaces and data formats to support users to seamlessly integrate and use the system;

[0035] Provides detailed technical documentation and sample codes to guide users to quickly understand and apply the various functions and features of this system.

[0036] An efficient lossless data compression system based on parallel computing, the system comprising:

[0037] The JNI interaction module interacts with the underlying compression library through the Java Native Interface (JNI) mechanism to achieve data transmission and processing between the Java platform and the underlying compression library;

[0038] Parallel computing optimization module, which introduces Intel's ISA-L library to transform and optimize zlib or other underlying compression libraries;

[0039] The memory data compression rate enhancement module performs efficient lossless compression and decompression operations on the data in the memory based on the integrated parallel computing optimization algorithm;

[0040] API compatibility assurance module: when optimizing the system, it fully considers the API compatibility with the native Java class library to ensure the stability of the external interface and data format;

[0041] Test verification module, used to design and implement compatibility tests and performance tests.

[0042] Preferably, the JNI interaction module passes the data on the Java platform to the underlying compression library through the JNI method, and calls the function of the underlying compression library to complete the lossless compression and decompression operations of the data, while ensuring that the calling process of the JNI method is stable and reliable and does not have a negative impact on the stability and performance of the system.

[0043] Preferably, the parallel computing optimization module uses the parallel computing optimization algorithm in the ISA-L library to accelerate the compression and decompression operations of data, and implements parallel computing of underlying functions by using the SSE, AVX, AVX2, and AVX512 modern processor extended instruction sets to improve data processing efficiency;

[0044] In the design process of the optimization algorithm, the characteristics of data compression and decompression are fully considered to ensure that the integrity and accuracy of the data are maintained while improving performance.

[0045] Preferably, the API compatibility assurance module ensures that users can seamlessly switch to this system without modifying existing codes or workflows. At the same time, detailed technical documentation and sample codes are provided to help users quickly get started and correctly use the various functions and features of the system.

[0046] Preferably, the test verification module verifies the correctness and stability of the system and the native Java class library when performing compression / decompression operations in the interface crossover, and verifies the effectiveness and reliability of the compression performance of the system being 5-10 times higher than that of the native Java class library under the same compression task;

[0047] Based on the test results, the test verification module tunes and optimizes the system to ensure that the system can meet the expected performance indicators and user needs in various application scenarios.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The invention proposes an efficient lossless data compression method and system based on parallel computing. By integrating parallel computing optimization algorithms and transforming compression libraries, the system improves the performance of the Java platform. After actual testing, under the same compression task, the compression performance is improved by 5-10 times compared with the native Java class library, thereby greatly improving data processing efficiency.

[0050] Optimization is performed while ensuring API compatibility, ensuring the same interface and data format as the native Java class library, allowing users to seamlessly switch to this system without modifying existing code or workflow.

[0051] The present invention is applicable to computer services of Java 8 and above, and can be widely used in various data processing businesses. Whether it is network transmission, file transmission or database management, the efficiency and performance of data compression can be improved by using this system.

[0052] Through the implementation of the present invention, technical experience in data compression optimization on the Java platform is accumulated, which provides useful experience and inspiration for solving similar problems in the future and promotes the further development and application of data compression technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of the native JDK compression / decompression function call chain of the present invention;

[0054] Figure 2 This is a schematic diagram of a compression / decompression function call link of the present invention;

[0055] Figure 3 This is a native zlib function call flow chart of the present invention;

[0056] Figure 4 This is a schematic diagram of a compression / decompression function call link of the present invention;

[0057] Figure 5 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] For example, see Figures 1 to 5 The present invention provides a technical solution: an efficient lossless data compression method based on parallel computing, the method comprising the following steps:

[0060] S1. JNI mechanism for calling underlying compression library functions:

[0061] S101. Interact with the underlying compression library through the Java Native Interface (JNI) mechanism.

[0062] S102, implement the JNI method to pass the data of the Java platform to the underlying compression library for processing.

[0063] S103, calling the function of the underlying compression library to complete data compression and decompression operations.

[0064] S104. Ensure that the calling process of the JNI method is stable and reliable and does not affect the stability and performance of the system.

[0065] S2. Integrated parallel computing optimization algorithm:

[0066] S201. Introduce Intel's ISA-L library to complete the transformation and optimization of zlib.

[0067] S202. Utilize the parallel computing optimization algorithm in the ISA-L library to accelerate the compression and decompression operations.

[0068] S203. By using extended instructions such as SSE / AVX / AVX2 / AVX256, parallel operations on underlying functions are realized.

[0069] S204. The design of the optimization algorithm must fully consider the characteristics of data compression and decompression to ensure that the integrity and accuracy of the data are not affected while improving performance.

[0070] S3, achieve memory data compression rate improvement effect:

[0071] S301. Based on the integrated parallel computing optimization algorithm, the compression and decompression operations on the memory data are completed.

[0072] S302. Verify the effectiveness and reliability of the optimization algorithm in improving data compression rate through experimental testing.

[0073] S303. Ensure that the memory data compression rate improvement effect meets expectations and achieves a 5-10 times performance improvement.

[0074] S304. Perform performance analysis on the compression and decompression operations of memory data to find potential optimization space and further improve the performance and stability of the system.

[0075] S4. Ensure API compatibility:

[0076] S401. During the optimization process, full consideration is given to API compatibility with native Java class libraries.

[0077] S402. Ensure compatibility with external interfaces and data formats so that users can seamlessly switch to the present system without modifying existing codes or workflows.

[0078] S403. Provide detailed technical documentation and sample codes to help users quickly get started and correctly use the functions and features of this system.

[0079] S5. Test verification:

[0080] S501. Design and implement compatibility testing to verify the correctness and stability of compression / decompression operations performed by the system and the native Java class library interface.

[0081] S502. Perform performance testing to verify that under the same compression task, the compression performance of this system is 5-10 times more effective and reliable than that of the native Java class library.

[0082] S503. Tune and optimize the system according to the test results to ensure that the system can meet the expected performance indicators and user needs in various scenarios.

[0083] Embodiment 2, based on embodiment 1, proposes an efficient lossless data compression system based on parallel computing, the system comprising:

[0084] The JNI interaction module interacts with the underlying compression library through the Java Native Interface (JNI) mechanism to realize data transfer and processing between the Java platform and the underlying compression library. It passes the data on the Java platform to the underlying compression library through the JNI method, and calls the functions of the underlying compression library to complete the lossless compression and decompression operations of the data. At the same time, it ensures that the calling process of the JNI method is stable and reliable, and does not have a negative impact on the stability and performance of the system.

[0085] The parallel computing optimization module introduces Intel's ISA-L library to transform and optimize zlib or other underlying compression libraries; the parallel computing optimization algorithm in the ISA-L library is used to accelerate the compression and decompression operations of data, and the parallel computing of underlying functions is realized by using the SSE, AVX, AVX2, and AVX512 modern processor extended instruction sets to improve data processing efficiency;

[0086] In the design process of the optimization algorithm, the characteristics of data compression and decompression are fully considered to ensure that the integrity and accuracy of the data are maintained while improving performance.

[0087] The memory data compression rate enhancement module performs efficient lossless compression and decompression operations on the data in the memory based on the integrated parallel computing optimization algorithm;

[0088] The API compatibility assurance module fully considers the API compatibility with the native Java class library when optimizing the system to ensure the stability of the external interface and data format; it ensures that users can seamlessly switch to this system without modifying existing codes or workflows, and provides detailed technical documentation and sample codes to help users quickly get started and correctly use the various functions and features of the system.

[0089] The test verification module is used to design and implement compatibility tests and performance tests; verify the correctness and stability of the system and the native Java class library when performing compression / decompression operations at the interface crossover, and verify the effectiveness and reliability of the compression performance of the system being 5-10 times higher than that of the native Java class library under the same compression task;

[0090] Based on the test results, the test verification module tunes and optimizes the system to ensure that the system can meet the expected performance indicators and user needs in various application scenarios.

[0091] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An efficient lossless data compression method based on parallel computing, characterized in that: The method comprises the following steps: The JNI mechanism for calling the underlying compression library function; Integrate parallel computing optimization algorithms; Improve memory data compression rate; Ensure API compatibility; Test verification: Design and execute compatibility tests to verify the correctness and stability of the system and native Java class libraries when the interfaces are cross-used; conduct comprehensive performance tests to prove that the compression performance of the system is significantly improved by 5-10 times compared with the native Java class library under the same compression task; perform necessary system tuning based on the test results to ensure that the system can meet performance indicators and user needs in different application scenarios.

2. The method for efficient lossless data compression based on parallel computing according to claim 1, characterized in that: The specific operations of the JNI mechanism for calling the underlying compression library function include: Establish an interactive interface between the Java platform and the underlying compression library through the Java Native Interface (JNI) mechanism; Implement JNI methods for transferring data between the Java platform and the underlying compression library; Call the functions of the underlying compression library to perform data compression and decompression operations; Ensure the stability and reliability of JNI method calls to maintain the overall performance and stability of the system.

3. The method for efficient lossless data compression based on parallel computing according to claim 1, characterized in that: The specific operations of the integrated parallel computing optimization algorithm include: Introducing Intel's ISA-L library to transform and optimize traditional zlib or other compression libraries; Use the parallel computing optimization algorithm in the ISA-L library to accelerate the data compression and decompression process; Apply SSE / AVX / AVX2 / AVX512 extended instruction sets to parallelize underlying functions and improve computing efficiency; When optimizing algorithm design, ensure that the integrity and accuracy of data compression and decompression are not affected.

4. The method for efficient lossless data compression based on parallel computing according to claim 3, characterized in that: The specific operations to improve the memory data compression rate include: Based on the parallel computing optimization algorithm of integrated parallel computing optimization algorithm, the data in memory is efficiently compressed and decompressed; Through experimental testing, the effectiveness of the optimization algorithm in improving data compression rate was verified, ensuring a 5-10 times performance improvement; Perform performance analysis on memory data compression and decompression operations, and continuously optimize to explore potential performance improvements.

5. The method for efficient lossless data compression based on parallel computing according to claim 1, characterized in that: Specific operations to ensure API compatibility include: During the optimization process, we maintain high compatibility with the native Java class library API, ensuring that users can migrate seamlessly without modifying existing code. Ensure the compatibility of external interfaces and data formats to support users to seamlessly integrate and use the system; Provides detailed technical documentation and sample codes to guide users to quickly understand and apply the various functions and features of this system.

6. An efficient lossless data compression system based on parallel computing according to the efficient lossless data compression method based on parallel computing according to any one of claims 1 to 5, characterized in that: The system comprises: The JNI interaction module interacts with the underlying compression library through the Java Native Interface (JNI) mechanism to achieve data transmission and processing between the Java platform and the underlying compression library; Parallel computing optimization module, which introduces Intel's ISA-L library to transform and optimize zlib or other underlying compression libraries; The memory data compression rate enhancement module performs efficient lossless compression and decompression operations on the data in the memory based on the integrated parallel computing optimization algorithm; API compatibility assurance module: when optimizing the system, it fully considers the API compatibility with the native Java class library to ensure the stability of the external interface and data format; Test verification module, used to design and implement compatibility tests and performance tests.

7. The high-efficiency lossless data compression system based on parallel computing according to claim 6, characterized in that: The JNI interaction module passes the data on the Java platform to the underlying compression library through the JNI method, and calls the functions of the underlying compression library to complete the lossless compression and decompression operations of the data, while ensuring that the calling process of the JNI method is stable and reliable and does not have a negative impact on the stability and performance of the system.

8. The high-efficiency lossless data compression system based on parallel computing according to claim 6, characterized in that: The parallel computing optimization module uses the parallel computing optimization algorithm in the ISA-L library to accelerate the compression and decompression operations of data, and implements parallel computing of underlying functions by using the SSE, AVX, AVX2, and AVX512 modern processor extended instruction sets to improve data processing efficiency. In the design process of the optimization algorithm, the characteristics of data compression and decompression are fully considered to ensure that the integrity and accuracy of the data are maintained while improving performance.

9. The high-efficiency lossless data compression system based on parallel computing according to claim 6, characterized in that: The API compatibility assurance module ensures that users can seamlessly switch to this system without modifying existing codes or workflows. It also provides detailed technical documentation and sample codes to help users quickly get started and correctly use the various functions and features of the system.

10. The high-efficiency lossless data compression system based on parallel computing according to claim 6, characterized in that: Test verification module, verify the correctness and stability of this system and native Java class library when performing compression / decompression operations in the interface crossover, and verify the effectiveness and reliability of this system's compression performance being 5-10 times higher than that of native Java class library under the same compression task; Based on the test results, the test verification module tunes and optimizes the system to ensure that the system can meet the expected performance indicators and user needs in various application scenarios.