Block compression method of marine survey data, electronic equipment and storage medium

Abstract

The invention provides a block compression method for marine survey data, electronic equipment and a storage medium. The method comprises the following steps: acquiring marine survey data, wherein the marine survey data comprises text data and image data; according to the data type of the marine survey data, partitioning the marine survey data to obtain a plurality of original data blocks, and partitioning text data in the marine survey data according to a first partitioning size; partitioning image data in the marine survey data according to a second partitioning size; respectively carrying out data compression on each data block in the plurality of original data blocks to obtain a plurality of compressed data blocks, and adopting different compression strategies on repeated character strings and non-repeated character strings in each data block in the plurality of original data blocks; and performing global assembly on the plurality of compressed data blocks according to the sequence of the plurality of original data blocks to obtain target compressed data. The method can give consideration to high compression rate, high compression speed and low resource consumption.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a method for block compression of marine survey data, an electronic device, and a storage medium. Background Technology

[0002] As marine survey activities advance, the amount of marine survey data in marine areas continues to increase. This data includes information such as air temperature, air pressure, wind speed, seawater temperature, salinity, and depth. The real-time acquisition and transmission of this data is crucial for supporting the efficient operation and intelligent development of deep-sea engineering projects. In related technologies, to adapt to the ever-increasing volume of marine survey data and improve data transmission efficiency, it is usually necessary to compress the marine survey data before transmission. However, existing compression algorithms struggle to balance high compression ratio, compression speed, and parallel processing capabilities, resulting in poor compression performance and consequently affecting the data transmission efficiency of marine survey data.

[0003] In view of this, the present invention is hereby proposed. Summary of the Invention

[0004] The present invention is proposed in view of the above-mentioned problems. According to one aspect of the present invention, a method for block compression of marine survey data is provided, comprising: Acquire marine survey data, which includes text data and image data; Based on the data type of the marine survey data, the marine survey data is divided into blocks to obtain multiple raw data blocks. Specifically, text data in the marine survey data is divided into blocks according to a first block size, and image data in the marine survey data is divided into blocks according to a second block size. Each data block in the plurality of original data blocks is compressed to obtain a plurality of compressed data blocks. For each data block in the plurality of original data blocks, different compression strategies are used for repeating strings and non-repeating strings in the data block. The multiple compressed data blocks are globally assembled according to the order of the multiple original data blocks to obtain the target compressed data.

[0005] For example, the step of compressing each data block in the plurality of original data blocks includes: For each of the plurality of original data blocks The data block is preprocessed to determine the repeating strings in the data block, wherein, for the repeating strings in the data block, a matching sequence for the repeating strings is constructed, the matching sequence including at least the number of matching bytes and the matching position of the repeating strings; For the repeated strings in the data block, variable-length integer encoding technology is used to process the number of matching bytes and the matching position of the repeated strings to obtain the compression result of the repeated strings; For the non-repeating strings in the data block, dynamic Huffman coding is used to process the non-repeating strings to obtain the compressed result of the non-repeating strings; The compression results of each string are combined according to the character order in the data block to obtain the compressed data block corresponding to the data block.

[0006] For example, the preprocessing of the data block includes: Define a sliding window and a look-ahead buffer, wherein the look-ahead buffer is used to store unprocessed data in the data block, and the sliding window is used to store historical data that has been processed; According to the character order in the data block, the duplicate character search operation is performed repeatedly using the sliding window and the look-ahead buffer until the last character in the data block is processed. According to the character order in the data block, the character processing results obtained from each execution of the duplicate character search operation are combined to obtain the preprocessed result.

[0007] For example, the duplicate character search operation includes: The longest substring that is exactly the same as the beginning of the lookahead buffer is found in the sliding window, and the first character of the longest substring is the first character in the lookahead buffer; When there is a longest substring in the sliding window that is completely consistent with the beginning part of the lookahead buffer, the offset and matching length of the longest substring in the sliding window are output to construct the matching sequence of the longest substring. The number of matching bytes of the longest substring is the matching length of the longest substring in the sliding window, and the matching position of the longest substring is the offset of the longest substring in the sliding window. If there is no string in the sliding window that is exactly the same as the beginning of the lookahead buffer, output the first character in the lookahead buffer.

[0008] Exemplarily, the method further includes: Perform the following update operation: After each duplicate character search operation, if there is a longest substring in the sliding window that is completely consistent with the beginning part of the lookahead buffer, then move the longest substring in the lookahead buffer to the sliding window; otherwise, move the first character in the lookahead buffer to the sliding window. The next duplicate character search operation is performed based on the updated sliding window and look-ahead buffer.

[0009] For example, the look-ahead buffer stores a first preset number of characters starting with the character at the current processing position, and the sliding window stores a second preset number of characters before the current processing position; wherein, the character at the current processing position is the first character to be processed in the current round.

[0010] For example, the step of processing the non-repeating string using dynamic Huffman coding to obtain the compressed result of the non-repeating string includes: Dynamic Huffman coding is used to process the non-repeating string to obtain the initial compressed data; Calculate the storage size of the initial compressed data and the storage size of the non-repeating string, respectively; Compare the initial compressed data with the storage size of the non-repeating string; When the storage size of the non-repeating string is less than or equal to the storage size of the initial compressed data, the compression result of the non-repeating string is determined to be the non-repeating string itself. Otherwise, the compression result of the non-repeating string is determined to be the initial compressed data.

[0011] For example, combining the compression results of each string according to the character order in the data block includes: Based on the data type of the compressed result of each string, the compressed result of each string is assembled separately. Specifically, when the data type is a raw literal, the data type identifier and the total number of bytes of the compressed result are written sequentially before the compressed result; when the data type is dynamic Huffman coding, the data type identifier and the Huffman frequency table of the compressed result are written sequentially before the compressed result; when the data type is a matching sequence, the data type identifier of the compressed result is written before the compressed result. According to the character order in the data block, the assembly fragments corresponding to the compression results of each string are concatenated and combined sequentially.

[0012] According to another aspect of the present invention, an electronic device is provided, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method as described above.

[0013] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program / instructions that, when executed by a processor, implement the method described above.

[0014] Compared with the prior art, the present invention has at least the following technical effects: 1. By taking into account the different characteristics of the two core data types, text and images, in marine survey data, we use appropriate first and second block sizes for targeted block division. This not only effectively avoids the problem of poor adaptability of traditional single block division to heterogeneous data, laying the foundation for subsequent efficient compression, but also makes full use of the advantages of distributed systems to independently compress each original data block, achieving parallel processing and improving compression efficiency. 2. This block-based, parallel processing strategy effectively reduces the memory pressure and CPU burden on individual nodes. In large-scale distributed scenarios, the advantages of this strategy are even more pronounced, significantly reducing the overall resource consumption of the system. While ensuring high throughput and low latency—the two core requirements—it can further optimize energy efficiency. Especially for scenarios like real-time marine data monitoring, its energy-saving and consumption-reducing effects are very significant, helping to lower the overall operating costs of the business. 3. By implementing differentiated compression strategies for repetitive and non-repetitive strings within each original data block, we can significantly reduce redundant data through efficient encoding of repetitive strings, while ensuring data accuracy by using appropriate compression logic for non-repetitive strings, thus achieving a balance between "compression efficiency" and "data fidelity". 4. At the same time, whether a string is repeated or not is relative to the data block in which it is located and is unrelated to other data blocks. This helps to build an independent dictionary for each block, break through the serial limitation of "traditional global dictionary depends on historical data", eliminate cross-block dependency, and support parallel processing of different original data blocks. 5. By performing global assembly according to the original data block order, it is ensured that the compressed data can be completely restored according to the original logic, avoiding data corruption; 6. In summary, this method can solve the problems of high storage costs, numerous duplicate fields, low transmission efficiency, and difficulties in real-time processing faced by massive text log data (such as observation logs, sensor records, ship operation logs, etc.) and image data (such as sonar data or underwater camera images) during marine surveys. Applying this method to marine survey data can significantly improve the compression efficiency of marine survey data, reduce the space occupied by data storage and the consumption of transmission bandwidth, while ensuring the integrity and availability of compressed data, providing technical support for the efficient management, transmission, and subsequent analysis and application of marine survey data.

[0015] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0016] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0017] Figure 1 A schematic flowchart illustrating a method for block compression of marine survey data according to an embodiment of the present invention is shown. Figure 2 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0019] As mentioned above, existing compression algorithms struggle to balance high compression ratios, compression speeds, and parallel processing capabilities. Specifically, the main compression algorithms used in related technologies include GZIP / DEFLATE, LZO, Snappy, and LZ4. GZIP / DEFLATE, relying primarily on LZ77 and Huffman coding, achieves high compression ratios but suffers from slow compression speeds, high memory consumption, and lacks parallel processing support, making it unsuitable for real-time processing in large-scale data processing. Snappy and LZ4, while offering excellent speed and suitable for real-time data transmission, still suffer from storage waste when dealing with massive datasets due to their lower compression ratios. While LZO achieves a balance between compression speed and ratio, its lack of flexibility makes dynamic optimization based on data characteristics difficult. Advanced compression technologies such as Brotli and Zstandard have also been used in data analysis and artificial intelligence, but these methods typically focus on single-dimensional optimization and struggle to balance compression ratios, speed, and parallel processing requirements. Furthermore, most existing algorithms rely on global dictionaries or static encoding methods, failing to fully leverage the advantages of distributed systems. This results in limited scalability when processing large datasets and potential bottlenecks during parallel decompression. In view of this, this invention provides a block compression method, electronic device, and storage medium for marine survey data. This method achieves triple optimization—high compression ratio, high speed, and distributed adaptation—through adaptive block segmentation of heterogeneous data and hybrid dynamic encoding selection, particularly suited to the heterogeneous data characteristics of marine survey data, which is primarily text-based with supplementary image data. The method, electronic device, and storage medium are described in detail below.

[0020] According to one aspect of the present invention, a method for block compression of marine survey data is provided. Figure 1 A schematic flowchart illustrating a method for block compression of marine survey data according to an embodiment of the present invention is shown. Figure 1 As shown, the method may include the following steps: S110, S120, S130 and S140.

[0021] In step S110, marine survey data is acquired, including text data and image data.

[0022] In this article, marine survey data includes, but is not limited to, text-based observation logs, meteorological and marine environmental monitoring data, sonar / acoustic detection data, images, and video data.

[0023] Text-based observation logs can include sensor logs, navigation / operation logs, etc. Sensor logs can include data such as temperature, salinity, depth (CTD data), dissolved oxygen, turbidity, and ocean current velocity. These logs are usually recorded in text or semi-structured formats (JSON, CSV, etc.), and while individual files may be small, the sheer number of logs is enormous. Navigation / operation logs can include the operational status, route records, fault information, and communication logs of oceanographic research vessels or underwater robots (ROVs, AUVs). This data is mostly structured or semi-structured text and often contains a large number of repetitive fields.

[0024] Meteorological and marine environmental monitoring data can include weather station records and sea state monitoring data. Weather station records can include data such as wind speed, air pressure, humidity, and precipitation, which can be presented in text or tabular form. Sea state monitoring can include wave height, tide observations, sea surface temperature, sea ice distribution, etc., and can contain a large amount of numerical data, some of which is stored in formats such as NetCDF and HDF5, and may also be exported as text or binary files.

[0025] Sonar / acoustic detection data can include sonar echo data and acoustic Doppler current profiler (ADCP) data. Sonar echo data consists of raw waveform data or processed echo files used to detect seabed topography and target objects; it is typically in binary format and can be very large in volume. Acoustic Doppler current profiler (ADCP) data is continuous data measuring water flow velocity, usually stored in binary or proprietary formats.

[0026] Image and video data can be obtained through underwater camera footage and satellite remote sensing data. Underwater camera data can include videos or images of seabed life and the seabed environment taken by remotely operated underwater vehicles (ROVs), and the format may be JPG, PNG, TIFF, or video streams (MP4, etc.). Satellite remote sensing data consists of ocean surface images taken by satellites, with high resolution and large data volume, and is usually in formats such as GeoTIFF and HDF.

[0027] In step S120, the marine survey data is divided into blocks according to the data type of the marine survey data to obtain multiple raw data blocks. Specifically, text data in the marine survey data is divided into blocks according to the first block size, and image data in the marine survey data is divided into blocks according to the second block size.

[0028] Research has revealed that different types of data possess varying redundancy characteristics. Given that marine survey data is primarily text-based with supplementary image data, existing compression algorithms cannot simultaneously adapt to the redundancy characteristics of different data types. Therefore, this invention considers dynamically adjusting the block size based on the data type. In some embodiments, the first block size can be 64KB, and the second block size can be 16KB. This allows for larger blocks (e.g., 64KB) for text data and smaller blocks (e.g., 16KB) for binary data such as images, ensuring optimal memory usage and processing efficiency while maintaining compression effectiveness. This approach not only adapts to the redundancy characteristics of different data types but also provides a foundation for parallel processing.

[0029] In this example, when dividing marine survey data into multiple raw data blocks, if the size of the last raw data block is smaller than its corresponding preset block size, it can be padded with 0 bytes (0x00). For example, if the number of raw data blocks is K, the first K-1 data blocks can be directly cut according to the preset block size, and if the last data block is insufficient, it can be padded with 0 bytes (0x00) at the end.

[0030] In one specific embodiment, step S130 may include: dividing the marine survey data into blocks using the following formula to obtain multiple raw data blocks: ; in, Indicates the first Each block of data, D Indicates the size of the original input data byte stream. S This indicates the preset block size. When the original input data byte stream is text data, the preset block size is the first block size. When the original input data byte stream is image data, the preset block size is the second block size.

[0031] In some embodiments, prior to step S120, the method may further include converting the marine survey data into a byte stream format. Subsequent steps may process the byte stream format marine survey data. As described above, marine survey data may include text data, image data, etc. In this embodiment, converting the marine survey data into byte stream data facilitates subsequent data processing.

[0032] In step S130, each data block in the plurality of original data blocks is compressed to obtain a plurality of compressed data blocks. For each data block in the plurality of original data blocks, different compression strategies are used for repeating strings and non-repeating strings in the data block.

[0033] In this example, when compressing each data block in multiple original data blocks, the compression process of different data blocks can be carried out in parallel, and the compression process of each original data block is independent of each other. This eliminates the need for cross-block data dependencies. In practice, it can support multi-threaded and distributed processing frameworks such as HDFS, Spark, and Flink. This not only helps to improve the overall compression efficiency, but also has good random access and local decompression characteristics, providing a solid foundation for the real-time processing and efficient access of large-scale ocean data.

[0034] When compressing each original data block, the inventors considered independently distinguishing between repeating and non-repeating strings within each block and employing different compression strategies for each. On one hand, the repeating string is relative to its original data block and unrelated to data in other blocks. This allows for the creation of an independent dictionary for each block, overcoming the serial limitation of "traditional global dictionaries relying on historical data," eliminating cross-block dependencies, and supporting parallel processing of different original data blocks. On the other hand, applying differentiated compression strategies to repeating and non-repeating strings enables more accurate and efficient data compression. Specifically, for repetitive strings, compression algorithms more suited to repetitive patterns, such as dictionary encoding, differential encoding, and matching sequences, can be used to significantly reduce the storage space occupied by repetitive data and improve the compression ratio. For non-repetitive strings, compression methods more suitable for irregular data can be selected, ensuring compression effectiveness while avoiding resource waste caused by invalid compression operations. This achieves a balance between high compression efficiency, compression speed, and low resource consumption, reducing data storage costs and saving bandwidth and time costs for subsequent data transmission and retrieval. This is especially suitable for data scenarios such as marine survey data, which contain a large number of repetitive fragments and scattered non-repetitive content.

[0035] In step S140, multiple compressed data blocks are globally assembled according to the order of multiple original data blocks to obtain the target compressed data.

[0036] The core of global assembly is to embed key metadata through a standardized file header structure to achieve data security, random access, and system compatibility. In this example, the compressed data of each block can be arranged sequentially according to block number to obtain a global compressed data stream. Then, a standardized file header is embedded before the global compressed data stream to obtain the target compressed data. In a specific embodiment, the embedding method can be as follows: a file header area is allocated at the very beginning of the global compressed data stream, and metadata is compactly stored in the order of magic number, block configuration information, frequency table index, and reserved extension bits. The core representation content, embedding format, and position of each key metadata are shown in the table below: Table 1. Detailed Explanation of Key Metadata

[0037] This embodiment's solution incorporates a flexible file header structure designed during the global assembly phase, embedding key information such as magic numbers, block sizes, and frequency tables. This supports multi-version compatibility and subsequent expansion, while ensuring seamless transmission and decompression of compressed data across different platforms (e.g., the zero-copy decompression requirement of distributed computing frameworks like Spark and Flink). It addresses the limitations of traditional compression schemes in terms of metadata management and scalability, and meets the multiple requirements of marine survey data processing for data security, random access, and system compatibility.

[0038] The above-described global assembly method is merely an example. Those skilled in the art can use other methods to assemble the various compressed data blocks in sequence to obtain the final target compressed data as needed, which will not be elaborated further.

[0039] In the above technical solution, by taking into account the different characteristics of the two core data types, text and images, in marine survey data, appropriate first and second block sizes are used for targeted block division. This effectively avoids the problem of poor adaptability to heterogeneous data in traditional single-block methods, laying the foundation for subsequent efficient compression. It also makes full use of the advantages of distributed systems to independently compress each original data block, thereby meeting the memory constraints and distributed processing requirements in large-scale data environments. Moreover, this block-based, parallel processing strategy can effectively reduce the memory pressure and CPU burden of individual nodes. In large-scale distributed scenarios, the advantages of this strategy are even more prominent. It can significantly reduce the overall resource consumption of the system, ensuring both high throughput and low latency while further optimizing energy efficiency. This is particularly evident in scenarios such as real-time marine data monitoring, where its energy-saving and consumption-reducing effects are very significant, helping to lower the overall operating costs of the business. By implementing differentiated compression strategies for repetitive and non-repetitive strings within each original data block, it can significantly reduce redundant data through efficient encoding of repetitive strings, while ensuring data accuracy by using appropriate compression logic for non-repetitive strings, achieving a balance between compression efficiency and data fidelity. Furthermore, whether a string is repetitive or not is relative to its own data block and independent of other data blocks. This helps to establish an independent dictionary for each block, breaking through the serial limitation of "traditional global dictionary relying on historical data," eliminating cross-block dependencies, and supporting parallel processing of different original data blocks. By performing global assembly according to the original data block order, it ensures that the compressed data can be completely restored according to the original logic, avoiding data corruption. In summary, this method can solve the problems of high storage costs, numerous duplicate fields, low transmission efficiency, and difficulties in real-time processing faced by massive text log data (such as observation logs, sensor records, and ship operation logs) and image data (such as sonar data or underwater camera images) during marine surveys. Applying this method to marine survey data can significantly improve the compression efficiency of marine survey data, reduce the space occupied by data storage and the consumption of transmission bandwidth, while ensuring the integrity and availability of compressed data, and providing technical support for the efficient management, transmission, and subsequent analysis and application of marine survey data.

[0040] For example, data compression is performed on each data block in a plurality of original data blocks, including: for each data block in the plurality of original data blocks, preprocessing the data block to determine the repeating strings in the data block, wherein, for the repeating strings in the data block, a matching sequence of the repeating strings is constructed, the matching sequence including at least the number of matching bytes and the matching position of the repeating strings; for the repeating strings in the data block, variable-length integer encoding is used to process the number of matching bytes and the matching position of the repeating strings to obtain the compression result of the repeating strings; for the non-repeating strings in the data block, dynamic Huffman encoding is used to process the non-repeating strings to obtain the compression result of the non-repeating strings; and the compression results of each string are combined according to the character order in the data block to obtain the compressed data block corresponding to the data block.

[0041] In some embodiments, the matching sequence may also include a matching data type identifier, that is, a triple consisting of a matching data type identifier, the number of matching bytes, and the matching position may be used as the matching sequence.

[0042] In this example, we first consider finding duplicate strings in the data block. That is, we can sequentially traverse each character in the original data block from front to back. When at least one character at the current position has appeared before, we can determine that the string at the current position is a duplicate string and determine the number of matching bytes and the matching position based on the position and length of the first occurrence of the string.

[0043] After preprocessing, all repeating and non-repeating strings in the original data block can be identified. For repeating strings, variable-length integer encoding (such as Snappy) can be used to encode the number of matching bytes and the matching position to improve compression speed. For non-repeating strings, dynamic Huffman coding can be used to process the literal of the non-repeating string (i.e., the string itself), and the compression result can be obtained based on the processing result. For example, the dynamic Huffman coding result can be directly used as the compression result for the non-repeating string. Alternatively, the encoding result can be compared with the storage overhead of the original string, and the result with the smaller overhead can be selected as the compression result.

[0044] In the above technical solution, for repeated strings in the data block, preprocessing is used to accurately identify and construct a matching sequence containing the number of matching bytes and the matching position. Then, variable-length integer encoding technology is used to process the key information of the sequence. This not only preserves the core features of repeated strings to the greatest extent to ensure the accuracy of subsequent decompression, but also reduces the storage redundancy of repeated information through the flexibility of variable-length encoding, significantly reducing the space occupied by repeated content. For non-repeating strings, dynamic Huffman coding is used. This coding technology can dynamically adjust the encoding length according to the frequency of character occurrence, so that high-frequency characters get shorter codes, further improving the compression efficiency of non-repeating content. Finally, the compression results of various strings are combined according to the original character order. While ensuring the integrity of the data logical order, the final compressed data block has the advantages of high compression ratio and low information loss, effectively reducing the data storage volume and improving the data transmission efficiency. At the same time, it lays a solid foundation for the accurate restoration of data in the subsequent decompression process. In summary, this solution dynamically selects the compression encoding strategy according to whether the string is repeated or not, ensuring compact data storage without significantly increasing the computational burden, thereby achieving dual optimization of compression ratio and speed. Meanwhile, the compression process within each data block is completely self-contained, with no cross-block data dependencies. It supports parallel compression and random read / decompression in multi-threaded and distributed environments, greatly improving the operating efficiency of big data processing platforms such as HDFS, Spark, and Flink.

[0045] For example, preprocessing the data block includes: defining a sliding window and a look-ahead buffer, wherein the look-ahead buffer is used to store unprocessed data in the data block, and the sliding window is used to store processed historical data; repeatedly performing duplicate character lookup operations using the sliding window and look-ahead buffer according to the character order in the data block until the last character in the data block is processed; and combining the character processing results obtained from each duplicate character lookup operation according to the character order in the data block to obtain the preprocessed result.

[0046] In this example, a sliding window and lookahead buffer are used to find duplicate strings in the original data block. The duplicate string is replaced with a pointer (in this example, the offset and matching length) to the first occurrence of the duplicate string within the sliding window, thus eliminating redundant data and generating literals and matching sequences. Specifically, a sliding window and lookahead buffer are constructed on the string corresponding to the original data block. The sliding window stores processed characters, and the lookahead buffer stores unprocessed characters. Then, it is checked whether the sliding window includes the beginning portion of the string in the lookahead buffer. If it exists, a duplicate string is identified, and the number of matching bytes and the matching position of the duplicate string are output. The sliding window and lookahead buffer are moved synchronously, and the process continues with the next string. If the duplicate string does not exist, the non-duplicate string is output, and the sliding window and lookahead buffer are moved synchronously, and the process continues with the next string. This duplicate character search operation is repeated until all characters have been processed.

[0047] In the above technical solution, the sliding window retains processed historical data to provide contextual support, while the lookahead buffer temporarily stores unprocessed data to ensure the continuity of character reading. By defining a collaborative mechanism between the sliding window and the lookahead buffer, the duplicate character search operation based on both is repeatedly executed according to the character order of the data block until all characters are processed. Finally, the character processing results of each search are combined sequentially to form the preprocessed result. This method can efficiently and accurately identify and extract duplicate characters in the data block. Leveraging the historical data context advantage of the sliding window reduces redundant calculations and improves search accuracy. At the same time, the temporary storage mechanism of the lookahead buffer ensures that the character processing process is uninterrupted and guarantees data integrity. Overall, it significantly optimizes the efficiency and accuracy of data preprocessing, providing a high-quality preprocessed data foundation for subsequent data compression and effectively reducing the computational cost and data redundancy of subsequent stages. Moreover, in this process, both the historical data in the sliding window and the data to be processed in the lookahead buffer are only included in the current data block. This helps to overcome the serial limitation of "traditional global dictionary depending on historical data" and eliminate cross-block dependencies, thereby facilitating the parallel compression of various data blocks and improving compression efficiency.

[0048] For example, the duplicate character search operation includes: searching in a sliding window for the longest substring that is exactly the same as the beginning of the lookahead buffer, where the first character of the longest substring is the first character in the lookahead buffer; when a longest substring that is exactly the same as the beginning of the lookahead buffer exists in the sliding window, outputting the offset and matching length of the longest substring in the sliding window to construct a matching sequence of the longest substring, where the number of matching bytes of the longest substring is the matching length of the longest substring in the sliding window, and the matching position of the longest substring is the offset of the longest substring in the sliding window; when no substring that is exactly the same as the beginning of the lookahead buffer exists in the sliding window, outputting the first character in the lookahead buffer.

[0049] As described above, the longest substring in this example refers to the substring that starts with the first character in the lookahead buffer. In this example, the scheme searches for the longest substring in the sliding window W that has the same beginning portion as the lookahead buffer L, ensuring the length of the matching substring is as long as possible. Then, it returns the matching length and the offset of the matching substring within the sliding window. This process can be represented as: ; in, The function name for the longest match search. This indicates finding the longest substring in the sliding window that is identical to the beginning of the lookahead buffer. s .

[0050] For example, in some embodiments, the sliding window is W and the lookahead buffer is L. If W = [a,b,d,c] and L = [a,b,d], then first, the first character 'a' in L is searched, and W contains 'a'. Then, the string [a,b] in L is searched in W, and W contains this string. The search continues in L, and the string [a,b,d] in L is searched in W, and W contains this string. In this case, the longest substring is [a,b,d]. The offset of this longest substring in W (i.e., the matching position of the longest substring) is 3, and the matching length (also called the number of characters or the matching position of the longest substring) is 3, so the matching sequence (3,3) can be output. In other embodiments, the sliding window is W and the lookahead buffer is L. If W = [a,b,d,c] and L = [e,b,d], then first, the first character 'e' in L is searched, and W does not contain this character. In this case, the literal 'e' is directly output.

[0051] The above scheme, by clearly defining the search logic centered on a sliding window and a look-ahead buffer, can achieve accurate and efficient identification of duplicate strings in the data. On the one hand, by using the rule of "finding the longest substring that is completely consistent with the beginning of the look-ahead buffer" and anchoring the matching starting point with the first character, it can capture the longest repeating structure in the data to the maximum extent, avoiding the omission of duplicate information due to incomplete matching. At the same time, by constructing a matching sequence by outputting the offset and matching length of the longest substring in the sliding window, it can clearly record the position and scale information of the repeating strings, providing accurate basic data support for subsequent processing based on the repeating data. On the other hand, for scenarios where there is no matching substring in the sliding window, the first character of the look-ahead buffer is directly output, ensuring the continuity of data processing and avoiding the problem of interruption in the search process. The overall operation logic not only ensures the accuracy and completeness of duplicate character search, but also improves the stability and efficiency of the search process through a clear branching mechanism. It can be adapted to scenarios with a large number of duplicate fields in marine survey data, improving data compression speed and compression rate.

[0052] For example, the method further includes performing the following update operation: after each duplicate character search operation, if there is a longest substring in the sliding window that is exactly the same as the beginning of the lookahead buffer, then the longest substring in the lookahead buffer is moved to the sliding window; otherwise, the first character in the lookahead buffer is moved to the sliding window; wherein the next duplicate character search operation is completed based on the updated sliding window and the lookahead buffer.

[0053] For example, let the sliding window be W and the lookahead buffer be L. If W = [a,b,d,c] and L = [a,b,e], then the longest substring in L found in W is [a,b]. In this case, the matching sequence (2,2) can be output. Afterward, the processed substring can be moved into the sliding window to update L and W. The updated W = [a,b,d,c,a,b] and L = [e].

[0054] The above solution, by dynamically updating the sliding window, can significantly improve the processing efficiency of long texts or highly repetitive character sequences while enhancing the accuracy and targeting of duplicate character searches.

[0055] The data preprocessing operation of the present invention is described below through a detailed embodiment. In this embodiment, the original string T=[a,d,b,a,d,b,c] corresponds to the original data block. The sliding window and lookahead buffer data and character processing results for each round in this scheme are shown in Table 2 below.

[0056] Table 2 Pretreatment Process

[0057] At this point, the preprocessing result can be represented as [a,d,b,(3,3),c].

[0058] For example, the look-ahead buffer stores a first preset number of characters starting with the character at the current processing position, and the sliding window stores a second preset number of characters before the current processing position; wherein, the character at the current processing position is the first character to be processed in the current round.

[0059] It is understandable that, due to hardware memory limitations, the length of the longest string typically has an upper limit. Therefore, the size of the sliding window and lookahead buffer can be set according to this upper limit. For example, in some embodiments, both the first and second preset quantities can be 258 bytes. This avoids unlimited data searching, which would negatively impact efficiency.

[0060] For example, processing the non-repeating string using dynamic Huffman coding to obtain the compression result of the non-repeating string includes: processing the non-repeating string using dynamic Huffman coding to obtain initial compressed data; calculating the storage size of the initial compressed data and the storage size of the non-repeating string respectively; comparing the storage size of the initial compressed data and the storage size of the non-repeating string; if the storage size of the non-repeating string is less than or equal to the storage size of the initial compressed data, determining that the compression result of the non-repeating string is the non-repeating string itself; otherwise, determining that the compression result of the non-repeating string is the initial compressed data.

[0061] In this example, the non-repeating string is first processed using dynamic Huffman coding to obtain the initial compressed data. Specifically, a Huffman tree needs to be constructed first, i.e., a min-heap initialization: ; Here, f(b) represents the frequency of byte b in the literal; Len() is a function that gets the length, that is, the length of the binary string C(b); Bits is used to quantize the storage size of the encoded data.

[0062] Then, the two nodes with the lowest frequency are merged in a loop to generate a new node, until only the root node remains in the heap. Finally, the tree is traversed, with the left branch marked as 0 and the right branch marked as 1, to generate the encoding table C(b).

[0063] Next, the optimal solution between compression ratio and metadata overhead is dynamically selected by calculating the overhead. The method for calculating the initial storage size of the compressed data is as follows: ; ; in, C ( b ) represents a byteb Huffman coding, i.e., binary strings; O verhead indicates the storage overhead of the Huffman table; m The amount of data representing a literal; If the initial compressed data obtained by Huffman coding is used as the compression result, otherwise, the non-repeating string itself is used as the compression result.

[0064] The above technical solution evaluates the compression benefits of literal data in real time and intelligently switches between Huffman coding and original storage, enabling the algorithm to achieve a compression ratio close to GZIP while maintaining Snappy-level speed.

[0065] In some embodiments, the variable-length integer encoding process is as follows: For the input integer, it is divided into groups of 7 bits, and a flag bit is added to the high-order bits of each group, as shown in the following formula: ; in, n Represents the input integer, This represents a variable-length encoded byte sequence. This method can effectively reduce data storage space and has an extremely fast encoding speed.

[0066] For example, the compression results of each string are combined according to the character order in the data block, including: assembling the compression results of each string according to the data type of the compression results of each string, wherein, when the data type is a raw literal, the data type identifier and the total number of bytes of the compression result are written sequentially before the compression result; when the data type is dynamic Huffman coding, the data type identifier and the Huffman frequency table of the compression result are written sequentially before the compression result; when the data type is a matching sequence, the data type identifier of the compression result is written before the compression result; and concatenating and combining the assembled fragments corresponding to the compression results of each string according to the character order in the data block.

[0067] In some embodiments, the output result of the current data block after selection by hybrid dynamic encoding can be read first, and split into three categories according to data type: raw literals, Huffman-coded literals, and matching sequences, to ensure that the classification is complete and accurately matches the corresponding identifier type. The core encoding status within the block is recorded synchronously, including whether Huffman encoding is used, the distribution of the number of matching sequences, and the total length of the literals. At the same time, the binary byte stream buffer is initialized to provide a unified storage medium for subsequent data assembly and avoid format confusion.

[0068] Then, for literals whose encoding overhead is higher than that of the original data, a 1-byte identifier 0x00 is first written to the buffer to clarify that the data type is an unencoded original literal. Subsequently, the total number of bytes of the literal is written using variable-length integer encoding to ensure that it conforms to the preset encoding rules and that the length value is accurate. Finally, the unencoded original literal byte stream is directly written without adding any additional auxiliary information, reducing redundant overhead while ensuring data integrity.

[0069] For literals compressed using Huffman coding, a 1-byte identifier (0x01) is first written to distinguish the coding type. Next, a sequence of (ASCII code, variable-length coding frequency) tuples is written to the Huffman frequency table to ensure successful reconstruction of the Huffman tree at the decoding end. Then, a 1-byte padding bit count is written to record the padding information of the encoded bitstream, ensuring the value matches the actual number of padding bits. Finally, the Huffman-coded bitstream is converted into a byte stream and written to the buffer, immediately following the padding bits, to prevent data misalignment.

[0070] For the matching sequence encoded with variable-length integers, a 1-byte identifier 0x02 is first written to the buffer to clarify the data attributes. Then, the length of the matching substring and the backtracking distance are written in a fixed order of length and distance using Snappy-style variable-length integer encoding. This ensures that the encoding rules are compliant and the values ​​are accurate, without the need to add additional frequency tables or padding bits or other auxiliary information. This simplifies the storage structure while ensuring that the decoding end can parse it correctly.

[0071] Finally, following the literal and matching sequence order, the various tagged data fragments are sequentially concatenated into the buffer, ensuring that the concatenation order is consistent with the original processing order, without any errors or omissions. After concatenation, the data in the buffer is traversed to check the consistency between the tagged data and the data. For example, the 0x01 tagged data must be immediately followed by the frequency table, padding bits, and encoded data. Additionally, a 1-byte XOR checksum can be added to the end of the block to improve the reliability of data transmission and storage.

[0072] The above scheme transforms the compressed results (literals, matching sequences) of each block into standard, decomposable compressed data through unified identification standards, supplementary auxiliary information, and data format encapsulation. On the one hand, by formulating differentiated assembly rules for the compressed results of different data types (raw literals, dynamic Huffman coding, matching sequences)—pre-identifying the data type and total number of bytes for raw literals, pre-identifying the data type and Huffman frequency table for dynamic Huffman coding, and pre-identifying the data type for matching sequences—it ensures that each compressed result carries clear "identity information" and key auxiliary data, thus avoiding the mixing of different types of compression. Data obfuscation provides a precise basis for subsequent decompression, effectively reducing identification errors and logical complexity during decompression. On the other hand, strictly adhering to the original character order in the data block and sequentially splicing the assembled fragments of each string compression result can completely preserve the sequence logic of the original data, ensuring that the data maintains the same sequential correlation as the source data after compression and integration. At the same time, the standardized assembly and splicing method also improves the overall structure of the compressed data block, facilitating data management during subsequent storage and transmission, and ultimately achieving the unity of orderly integration of compressed data, accurate parsing, and sequence integrity assurance.

[0073] The effectiveness of the present invention is verified through a specific embodiment below. In this embodiment, the Yangtze River waterway maintenance and dredging project is taken as the research object. Data including hydrology, meteorology, water depth, and resources are collected on-site. Different operational data are then used as test datasets, divided into six data groups according to data volume (each data group can be referred to as a set of marine survey data). The data is then compared and analyzed with traditional data compression algorithms. The compression scheme is comprehensively evaluated using three indicators: storage space, transmission time, and compression ratio. The compression ratio is calculated as follows: ; In the formula, This represents the compressed data storage space. This indicates the storage space for the original data. This indicates the compression ratio. The lower the compression ratio, the smaller the compressed data file, and the better the compression effect. Therefore, the compression ratio can intuitively reflect the compression algorithm's ability to remove data redundancy.

[0074] The proposed solution (which can be referred to as HYBRID for ease of description) was used to process these data sets. The storage space and transmission time before and after compression were recorded sequentially, and the results are summarized in Table 3 for a direct comparison of performance before and after compression. During this process, each data set was preprocessed into a byte stream format, ensuring that the original data was presented in binary format. This process guarantees the consistency and accuracy of the data during subsequent compression.

[0075] Table 3 Comparison of data before and after compression

[0076] As shown in Table 3, for the largest set of test data (Set 6), the original data required 4,581,425 bytes of storage, while after HYBRID compression, only 1,236,984 bytes were needed. This means the compressed data is only about 27% of the original data, successfully compressing approximately 73% of redundant information. The resulting benefit is a 73.1% reduction in storage space usage, indicating that the HYBRID solution performs exceptionally well in data compression. On the other hand, from a transmission time perspective, research shows that to ensure a basic user experience, file transmission time should ideally be controlled within 50ms. For Set 6 test data, the original data transmission time exceeded 40ms, and latency could further increase under conditions of limited network bandwidth or a large number of files. However, after HYBRID compression, the transmission time was effectively reduced to within 10ms, approximately 24.5% of the original transmission time. This not only significantly shortened transmission latency but also provided users with a smoother experience.

[0077] Furthermore, it's worth noting that while HYBRID achieves a slightly lower compression ratio compared to GZIP, GZIP typically employs single-threaded compression, making it difficult to fully utilize the advantages of multi-core CPUs. This leads to a significant decrease in efficiency under large file or high-concurrency scenarios. In contrast, the solution of this invention employs a design philosophy that dynamically adjusts block size based on data type, compresses each block independently, and dynamically adjusts the compression strategy based on character repetition. This meets the needs of parallel processing, significantly improving compression speed and reducing memory usage without sacrificing compression performance. Moreover, the size of each data block can be flexibly set according to data type, enabling dynamic block division and achieving optimal compression performance under different data distributions, while ensuring a balance between memory usage and processing efficiency.

[0078] In summary, by comparing with traditional data compression algorithms in the prior art, it is easy to see that the solution of the present invention can achieve a balance between compression ratio, compression speed and parallel processing requirements, which helps to further improve the compression efficiency and effect of marine survey data, thereby helping to improve the transmission efficiency of marine survey data.

[0079] According to another aspect of the present invention, an electronic device is also provided. Figure 2 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Figure 2 As shown, the electronic device 200 includes a processor 210 and a memory 220. The memory 220 stores a computer program, which the processor 210 executes to implement the method described above.

[0080] According to another aspect of the present invention, a computer-readable storage medium is also provided. The storage medium stores a computer program / instructions that, when executed by a processor, implement the method described above. The storage medium may, for example, include a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0081] Those skilled in the art will readily understand the implementation structure, working principle, and beneficial effects of electronic devices and computer-readable storage media by reading the above methods. For the sake of brevity, further details will not be elaborated here.

[0082] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0084] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0085] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0086] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, in the description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof.

[0087] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0088] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0089] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the electronic device according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0090] The above description is merely a specific embodiment of the present invention or an explanation of the specific embodiment. The scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for block compression of marine survey data, characterized in that, include: Acquire marine survey data, which includes text data and image data; Based on the data type of the marine survey data, the marine survey data is divided into blocks to obtain multiple raw data blocks. Specifically, text data in the marine survey data is divided into blocks according to a first block size, and image data in the marine survey data is divided into blocks according to a second block size. Each data block in the plurality of original data blocks is compressed to obtain a plurality of compressed data blocks. For each data block in the plurality of original data blocks, different compression strategies are used for repeating strings and non-repeating strings in the data block. The multiple compressed data blocks are globally assembled according to the order of the multiple original data blocks to obtain the target compressed data; The step of compressing each data block in the plurality of original data blocks includes: For each of the plurality of original data blocks The data block is preprocessed to determine the repeating strings in the data block, wherein, for the repeating strings in the data block, a matching sequence for the repeating strings is constructed, the matching sequence including at least the number of matching bytes and the matching position of the repeating strings; For the repeated strings in the data block, variable-length integer encoding technology is used to process the number of matching bytes and the matching position of the repeated strings to obtain the compression result of the repeated strings; For the non-repeating strings in the data block, dynamic Huffman coding is used to process the non-repeating strings to obtain the compressed result of the non-repeating strings; The compression results of each string are combined according to the character order in the data block to obtain the compressed data block corresponding to the data block.

2. The block compression method according to claim 1, characterized in that, The preprocessing of the data block includes: Define a sliding window and a look-ahead buffer, wherein the look-ahead buffer is used to store unprocessed data in the data block, and the sliding window is used to store historical data that has been processed; According to the character order in the data block, the duplicate character search operation is performed repeatedly using the sliding window and the look-ahead buffer until the last character in the data block is processed. According to the character order in the data block, the character processing results obtained from each execution of the duplicate character search operation are combined to obtain the preprocessed result.

3. The block compression method according to claim 2, characterized in that, The duplicate character search operation includes: The longest substring that is exactly the same as the beginning of the lookahead buffer is found in the sliding window, and the first character of the longest substring is the first character in the lookahead buffer; When there is a longest substring in the sliding window that is completely consistent with the beginning part of the lookahead buffer, the offset and matching length of the longest substring in the sliding window are output to construct the matching sequence of the longest substring. The number of matching bytes of the longest substring is the matching length of the longest substring in the sliding window, and the matching position of the longest substring is the offset of the longest substring in the sliding window. If there is no string in the sliding window that is exactly the same as the beginning of the lookahead buffer, output the first character in the lookahead buffer.

4. The block compression method according to claim 2, characterized in that, The method further includes: Perform the following update operation: After each duplicate character search operation, if there is a longest substring in the sliding window that is completely consistent with the beginning part of the lookahead buffer, then move the longest substring in the lookahead buffer to the sliding window; otherwise, move the first character in the lookahead buffer to the sliding window. The next duplicate character search operation is performed based on the updated sliding window and look-ahead buffer.

5. The block compression method according to any one of claims 2-4, characterized in that, The look-ahead buffer stores a first preset number of characters starting with the character at the current processing position, and the sliding window stores a second preset number of characters preceding the current processing position; wherein, the character at the current processing position is the first character to be processed in the current round.

6. The block compression method according to claim 1, characterized in that, The process of using dynamic Huffman coding to process the non-repeating string to obtain the compressed result of the non-repeating string includes: Dynamic Huffman coding is used to process the non-repeating string to obtain the initial compressed data; Calculate the storage size of the initial compressed data and the storage size of the non-repeating string, respectively; Compare the initial compressed data with the storage size of the non-repeating string; When the storage size of the non-repeating string is less than or equal to the storage size of the initial compressed data, the compression result of the non-repeating string is determined to be the non-repeating string itself. Otherwise, the compression result of the non-repeating string is determined to be the initial compressed data.

7. The block compression method according to claim 1, characterized in that, The step of combining the compressed results of each string according to the character order in the data block includes: Based on the data type of the compressed result of each string, the compressed result of each string is assembled separately. Specifically, when the data type is a raw literal, the data type identifier and the total number of bytes of the compressed result are written sequentially before the compressed result; when the data type is dynamic Huffman coding, the data type identifier and the Huffman frequency table of the compressed result are written sequentially before the compressed result; when the data type is a matching sequence, the data type identifier of the compressed result is written before the compressed result. According to the character order in the data block, the assembly fragments corresponding to the compression results of each string are concatenated and combined sequentially.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method as claimed in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The system stores a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images