Bitmap data processing method, apparatus, device, and medium
By performing feature extraction and parallel compression on bitmap data, the problems of low accuracy in sparse module identification and poor adaptability of compression algorithms are solved, achieving efficient bitmap data processing and improving query performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-01-08
- Publication Date
- 2026-05-29
AI Technical Summary
Existing bitmap index compression technologies have low sparse module identification accuracy and poor compatibility of compression algorithms, resulting in compressed files that cannot meet query requirements.
By extracting features from the original bitmap data, sparse modules and ordinary modules are selected. The sparse modules are compressed using a parallel compression algorithm, and all modules are integrated to generate a compressed bitmap file.
It improves the accuracy of sparse module identification, reduces data compression time, reduces the space occupied by bitmap data, and improves data query efficiency.
Smart Images
Figure CN122115595A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of big data technology and data processing technology, and in particular to a bitmap data processing method, apparatus, device and medium. Background Technology
[0002] In the field of data processing, bitmap indexes are widely used in databases, data warehouses, and other scenarios due to their high query performance. For example, in the medical field, they are used to assist doctors in quickly locating similar cases, verifying lesion characteristics, and improving diagnostic efficiency. In the financial field, they are used to verify the authenticity of image information, screen for fraud risks, and improve business processing efficiency. However, existing bitmap index compression technologies have many shortcomings in practical applications and cannot meet the growing demand for bitmap data processing. Furthermore, existing technologies have low accuracy in identifying sparse modules, and the compression algorithms have poor adaptability, resulting in compressed files that cannot meet query requirements. Therefore, there is an urgent need for a bitmap data processing method to solve these problems. Summary of the Invention
[0003] This invention provides a bitmap data processing method, apparatus, device, and medium to solve the technical problems of low recognition accuracy of sparse modules and poor adaptability of compression algorithms in the prior art.
[0004] A bitmap data processing method includes: Feature extraction is performed on the original bitmap data to obtain feature information corresponding to each initial module; Based on all the aforementioned feature information, sparse modules and ordinary modules are selected from all the initial modules, and a list of sparse modules including compression algorithm identifiers and a list of ordinary modules including encapsulated data of all the ordinary modules are determined. Based on the sparse module list, the bitmap data in all the sparse modules are compressed in parallel to obtain compressed data corresponding to each sparse module. When the integrity of all the compressed data passes the verification, the data of all the sparse modules and all the ordinary modules are integrated to obtain a compressed bitmap file corresponding to the original bitmap data.
[0005] A bitmap data processing apparatus, comprising: The feature extraction module is used to extract features from the original bitmap data to obtain feature information corresponding to each initial module; The module filtering module is used to filter out sparse modules and ordinary modules from all the initial modules based on all the feature information, and determine a list of sparse modules including compression algorithm identifiers, and a list of ordinary modules including encapsulation data of all the ordinary modules. The data compression module is used to perform parallel compression of bitmap data in all the sparse modules based on the sparse module list, so as to obtain compressed data corresponding to each sparse module. The data integration module is used to integrate the data of all the sparse modules and all the ordinary modules when the integrity of all the compressed data passes the verification, so as to obtain a compressed bitmap file corresponding to the original bitmap data.
[0006] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being used to perform the bitmap data processing method described above.
[0007] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the bitmap data processing method described above.
[0008] The aforementioned bitmap data processing method, apparatus, device, and medium, in this invention, achieves module division of the original bitmap data and extraction of feature information from each initial module by extracting features from the original bitmap data. This enables the screening of sparse modules, improves the accuracy of sparse module identification, and avoids the problem of missing sparse modules due to low identification accuracy. Furthermore, it enables the determination of the sparse module list and the ordinary module list. Based on the sparse module list, the bitmap data in all sparse modules is compressed in parallel, achieving compression of bitmap data in each sparse module and parallel compression of all sparse modules, thereby reducing data compression time and improving compression efficiency. By integrating the data from all sparse modules and all ordinary modules, the compressed data is integrated, thereby enabling the acquisition of compressed bitmap files, reducing the space occupied by bitmap data, and improving the efficiency of subsequent data querying. Attached Figure Description
[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 This is a flowchart of a bitmap data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart of step S30 of a bitmap data processing method in one embodiment of the present invention; Figure 3 This is a schematic diagram of a parts cost prediction device according to an embodiment of the present invention. Detailed Implementation
[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0012] In one embodiment, such as Figure 1 As shown, a bitmap data processing method is provided, including the following steps: S10: Extract features from the original bitmap data to obtain feature information corresponding to each initial module.
[0013] In essence, raw bitmap data refers to a data structure that uses binary bits to represent information or state. An initial module refers to a portion of the bitmap data obtained after partitioning it; that is, the initial module includes a portion of the raw bitmap data. Feature information includes the number (density) of 1s (binary bits) in the initial module, the length distribution of consecutive 0 (binary bits) sequences, the length distribution of consecutive 1 sequences, and the start and end positions of the module. Feature information refers to the module density / length distribution and position information extracted from each module.
[0014] Specifically, the original bitmap data is first divided into modules to obtain at least one initial module. Then, feature extraction is performed on the number of 1s in each initial module, as well as the distribution of consecutive 0 and 1 sequences. This involves calculating the module density and length distribution of the initial modules to obtain the feature information corresponding to each initial module. In one embodiment, the initial module size is set to 32 bits. If the module density changes significantly between two or more consecutive modules, for example, exceeding a preset density change threshold, the initial module size can be adjusted to 16 bits or 64 bits for re-division and feature extraction.
[0015] S20: Based on all the feature information, sparse modules and ordinary modules are selected from all the initial modules, and a list of sparse modules including the compression algorithm identifier and a list of ordinary modules including the encapsulation data of all the ordinary modules are determined.
[0016] In essence, a sparse module refers to a module whose module density and bit distribution do not meet the threshold requirements. A regular module refers to a module whose module density and bit distribution meet the threshold requirements. The sparse module list includes compression algorithm identifiers and their corresponding sparse modules. The regular module list includes regular modules and their corresponding encapsulation data.
[0017] Specifically, the module density, length distribution of consecutive 0 sequences, and length distribution of consecutive 1 sequences in the feature information of each initial module are compared with corresponding thresholds. Modules whose module density and distribution do not meet the threshold requirements are identified as sparse modules, while those that do are identified as ordinary modules. Then, compression algorithm matching is performed on each sparse module to find compression algorithms that match the binary bit distribution in that sparse module. These are then identified as target compression algorithms for each sparse module, and the compression algorithm identifier corresponding to the target algorithm is associated with the corresponding sparse module, resulting in a list of sparse modules including the compression algorithm identifier. Similarly, a preset storage format is obtained, and the bitmap data in ordinary modules is encapsulated using this preset storage format to obtain a list of ordinary modules. During the encapsulation process, module header information, module type, and length distribution data can be added.
[0018] S30: Based on the sparse module list, perform parallel compression on the bitmap data in all the sparse modules to obtain compressed data corresponding to each sparse module.
[0019] In essence, compressed data refers to the data obtained by compressing the bitmap data of each sparse module. This may include information used for integrity verification, such as checksums. It may also include the compression status during the compression process, such as compression time and compression ratio.
[0020] Specifically, the target compression algorithm corresponding to each sparse module is called based on the compression algorithm identifier in the sparse module list. Then, using a distributed framework, the bitmap data in all sparse modules is compressed in parallel using all target compression algorithms, thereby obtaining compressed data corresponding to each sparse module. In one embodiment, the compression status is recorded during the compression process and added to the compressed data. Verification information is calculated for each compressed data and added to the compressed data.
[0021] S40: When the integrity of all the compressed data passes the verification, the data of all the sparse modules and all the ordinary modules are integrated to obtain a compressed bitmap file corresponding to the original bitmap data.
[0022] Understandably, compressed bitmap files contain global header information, compressed data streams, and index structures, which are used to query the location of modules and algorithms later.
[0023] Specifically, a checksum is first calculated for each compressed data item to obtain a new checksum. This new checksum is then compared with the checksum corresponding to the same compressed data item. If the two checksums match, the compressed data is considered complete. If they do not match, the compressed data is marked as erroneous and recompressed until the integrity of all compressed data passes the integrity check. Next, all sparse modules and all regular modules are integrated. This involves sorting the bitmap data in both sparse and regular modules according to their position within the original bitmap data, forming a continuous compressed data stream. A module index and an algorithm index are then constructed, and the continuous compressed data stream, along with the constructed module and algorithm indices, constitutes the compressed bitmap file. In one embodiment, global header information is added during data integration, including metadata such as the length of the original bitmap, the total number of compressed modules, the number of sparse modules, the number of regular modules, and the compression time.
[0024] In one embodiment, in the financial field, when banks batch process check bitmaps, the system first segments the image into multiple initial modules, extracting features from each module, such as the pixel grayscale concentration of text and number modules (including account number, amount, and date), the color and texture features of the seal module, and the pixel consistency of the blank background module. After filtering, the text areas containing the amount and account number, as well as the seal area, are classified as ordinary modules, which are encapsulated and included in the ordinary module list. The blank areas at the check edges and the repetitive texture areas without information in the background are classified as sparse modules, and they are labeled with JPEG algorithm identifiers suitable for high compression ratios, forming a sparse module list. Parallel compression is initiated for all sparse modules, and the blank and redundant background areas are quickly compressed using the JPEG algorithm; ordinary modules only undergo light lossless compression to ensure the clarity of text and seals. After compression, it is verified that no information is lost in each module, and then all modules are integrated to generate a compressed check bitmap file.
[0025] In one embodiment, in the medical field, the resolution of pathological slide bitmaps can reach hundreds of millions of pixels. The system segments the tissue regions of the slides into initial modules such as tumor lesion modules, normal cell modules, and blank slide modules. Cell density and texture complexity features of each module are extracted, such as the disorder of cell arrangement in the lesion module and the pixel indistinction features in the blank slide module. The tumor lesion module and normal cell module, as the diagnostic core, are classified as ordinary modules and encapsulated; the blank slide area and the cell-free blank areas at the edge of the slide are sparse modules, labeled with a TIFF lossless compression algorithm (to preserve the integrity of module boundaries and avoid splicing defects during integration). Lossless compression is performed in parallel on all sparse modules of the slides, while near-lossless compression is used for ordinary modules to preserve cell morphology details. After all modules pass compression verification, they are integrated into a complete compressed slide file.
[0026] In the bitmap data processing method of this invention, feature extraction is performed on the original bitmap data to achieve module division of the original bitmap data and extraction of feature information of each initial module. This enables the screening of sparse modules, improves the accuracy of sparse module identification, and avoids the problem of missing sparse modules due to low identification accuracy. Furthermore, it enables the determination of the sparse module list and the ordinary module list. Based on the sparse module list, the bitmap data in all sparse modules is compressed in parallel, achieving compression of bitmap data in each sparse module and parallel compression of all sparse modules, thereby reducing data compression time and improving compression efficiency. By integrating the data of all sparse modules and all ordinary modules, the integration of all compressed data is achieved, thereby enabling the acquisition of compressed bitmap files, reducing the space occupied by bitmap data, and improving the efficiency of subsequent data query.
[0027] In one embodiment, in step S10, the feature extraction of the original bitmap data to obtain feature information corresponding to each initial module includes: S101, the original bitmap data is segmented according to a preset sliding window to obtain at least one initial module and bitmap data corresponding to each initial module.
[0028] S102, count all the bitmap data in each of the initial modules to obtain a list of module density and length distributions corresponding to each of the initial modules.
[0029] S103, based on the module density, the length distribution list, and the start and end positions corresponding to each initial module, determine the feature information corresponding to each initial module.
[0030] Understandably, a preset sliding window refers to a pre-defined, sliding area, such as a 3x3 area. The start and end positions are used to record the position of each initial module in the original bitmap data. Module density refers to the ratio of the number of 1s in the segmented data to the size of the preset sliding window. The length distribution list includes the length distribution of 0 sequences and the length distribution of 1 sequences, such as 0(9, 6, 3, 2), where the longest consecutive 0 sequence is 9 and the shortest is 2; and 1(8, 5, 3, 1), where the longest consecutive 1 sequence is 8 and the shortest is 1. Bitmap data refers to binary data, used to construct the underlying information set of the bitmap.
[0031] Specifically, a preset sliding window is obtained, and the original bitmap data is segmented using this window. This means that the region of each preset sliding window in the original bitmap data is defined as an initial module, and the binary data within each initial module is defined as bitmap data. This yields all the initial modules and their corresponding bitmap data in the original bitmap data. Then, all the bitmap data within each initial module is counted. This involves first determining the number of 1s in all the bitmap data and calculating the module density of each initial module. Then, the length distribution of all consecutive 0 sequences and the length distribution of all consecutive 1 sequences in the bitmap data are statistically analyzed to obtain a length distribution list.
[0032] In one embodiment, if the module density of all initial modules in the original bitmap data varies significantly, the size of a preset sliding window is dynamically adjusted to regulate the module density and prevent large variations in module density. For example, if the change in module density between two consecutive initial modules exceeds a density change threshold, it is determined that the sliding window size needs to be adjusted.
[0033] Next, the position of each initial module in the original bitmap data is determined, and its start and end positions are identified. In another embodiment, the boundary positions of the preset sliding serial port when segmenting the original bitmap data are determined as the start and end positions. Finally, the module density, length distribution list, start and end positions corresponding to the same initial module are determined as the feature information corresponding to the initial module data. In this way, the feature information corresponding to each initial module can be obtained.
[0034] In this embodiment, a preset sliding window is used to segment the original bitmap data, obtain the initial modules, and determine the bitmap data. Data statistics are used to calculate the module density in each initial module and obtain a length distribution list, thereby enabling the extraction of feature information.
[0035] In one embodiment, in step S20, the step of filtering sparse modules and ordinary modules from all the initial modules based on all the feature information includes: S201, determine whether the module density in the feature information corresponding to each initial module exceeds a preset density threshold.
[0036] S202, when the module density of the initial module is less than the preset density threshold, detect whether the length distribution list in the feature information corresponding to each initial module satisfies the preset sparsity condition.
[0037] S203, when the length distribution list of the initial modules meets the preset sparsity condition, the initial modules are determined as sparse modules, and all initial modules other than sparse modules are determined as ordinary modules.
[0038] Understandably, a preset density threshold refers to a pre-set density threshold used to evaluate sparse modules and normal modules. Preset sparsity conditions include thresholds used to evaluate the length distribution of consecutive 0 sequences and the length distribution of consecutive 1 sequences.
[0039] Specifically, a preset density threshold is obtained, and the module density in the feature information of each initial module is compared with the preset density threshold. When the module density of the initial module is greater than or equal to the preset density threshold, the initial module is determined to be a normal module. Then, when the module density of the initial module is less than the preset density threshold, a preset sparsity condition is obtained, and the length distribution list in the feature information of the initial module is compared with the preset sparsity condition. That is, if the longest consecutive 0 sequence length in the feature information of the initial module is greater than the preset consecutive 0 length threshold, and the average length of the consecutive 0 sequences is greater than the average length threshold, while the maximum length of the consecutive 1 sequences is less than the consecutive 1 length threshold, then the initial module is determined to meet the preset sparsity condition, and the initial module is determined to be a sparse module. In this way, all initial modules except sparse modules are determined to be normal modules. Among them, when the length distribution list does not meet the preset sparsity condition, the initial module is determined to be a normal module.
[0040] In this embodiment, by setting a density threshold and a sparsity condition, the sparse module is accurately identified, avoiding the problem of missing sparse modules due to low identification accuracy, thereby improving the accuracy of the compression algorithm matching.
[0041] In one embodiment, in step S20, determining the list of sparse modules including the compression algorithm identifier and the list of ordinary modules including the encapsulation data of all the ordinary modules includes: S204, Match the feature information of all the sparse modules with the preset algorithms in the preset algorithm library to obtain the target compression algorithm corresponding to each sparse module.
[0042] S205, associate the compression algorithm identifier corresponding to the target compression algorithm with the sparse module to obtain a list of sparse modules including the compression algorithm identifier.
[0043] S206, the bitmap data in all the ordinary modules is encapsulated according to a preset storage format to obtain encapsulated data corresponding to each ordinary module, and a list of ordinary modules is determined based on all the encapsulated data and all the ordinary modules.
[0044] Understandably, a preset algorithm refers to a pre-defined compression algorithm, such as run-length encoding or differential encoding. A target compression algorithm refers to a preset algorithm that matches each sparse module. A compression algorithm identifier refers to a symbol or mark used to identify the compression algorithm. A packaged ordinary module refers to an ordinary module whose feature information has been converted to a preset storage format.
[0045] Specifically, after filtering out sparse modules and ordinary modules, a preset algorithm library is obtained. The feature information of each sparse module is then matched against the preset algorithm library, comparing the feature information with the usage conditions of the preset algorithms to find the preset algorithm that best suits the sparse module. This preset algorithm is then designated as the target compression algorithm. For example, for sparse modules with long and relatively uniformly distributed consecutive 0 sequences, run-length encoding is chosen, as this algorithm has a significant compression effect on continuous repetitive data. For sparse modules where the positions of 1s are relatively scattered, but the intervals between adjacent 1s have a certain regularity, differential encoding is chosen, reducing the amount of data by storing the difference between adjacent 1 positions. For sparse modules with extremely low data density (e.g., density below 0.01) and no obvious continuous sequence characteristics, variable-length integer encoding is chosen to more efficiently store the position information of a small number of 1s.
[0046] Then, the compression algorithm identifier corresponding to each target compression algorithm is obtained, and the compression algorithm identifiers corresponding to the same target compression algorithm are associated with and integrated with the sparse modules to obtain a list of sparse modules including the compression algorithm identifiers. Next, a preset storage format is obtained, and the feature information of each ordinary module is encapsulated using the preset storage format to obtain encapsulated data corresponding to each ordinary module. The encapsulated data is then associated with and integrated with its corresponding ordinary module to obtain a list of ordinary modules.
[0047] In this embodiment, by matching feature information with a preset algorithm, the compression algorithm for each sparse module is determined, thereby establishing the association between the compression algorithm identifier and the sparse module, and determining the list of sparse modules, which facilitates subsequent calls to the compression algorithm. Through data encapsulation, the bitmap data in ordinary modules conforms to the storage format, thus enabling the retrieval of the list of ordinary modules.
[0048] In one embodiment, such as Figure 2 As shown, in step S30, the parallel compression of bitmap data in all sparse modules based on the sparse module list to obtain compressed data corresponding to each sparse module includes: S301, the sparse module list is divided into multiple compression subtasks, and each compression subtask contains several sparse modules.
[0049] S302, according to the independent thread corresponding to each compression subtask, the bitmap data in all the sparse modules is compressed by the target compression algorithm corresponding to each compression algorithm identifier to obtain the compressed data corresponding to each sparse module.
[0050] Understandably, a compression subtask comprises several sparse modules. An independent thread refers to an execution path that can run independently of the main thread or other threads. Compressed data is the simplified data form obtained by processing raw bitmap data using a compression algorithm.
[0051] Specifically, after obtaining the list of sparse modules, the compression complexity of each sparse module is determined. For example, the complexity is calculated by combining the length of each sparse module with the computational cost of the target compression algorithm. Then, based on the compression complexity and the amount of computing resources (such as the number of CPU cores), all sparse modules in the list are divided, ensuring a balanced computational load across the compression subtasks, resulting in multiple compression subtasks, each containing several sparse modules. Next, an independent computation thread or process is allocated to each compression subtask. Each thread or process, based on the compression algorithm identifier corresponding to the sparse module, calls the appropriate target compression algorithm to compress the bitmap data within the sparse module. During compression, the compression status of each sparse module (such as compression progress and error detection) is recorded in real time, and verification information (such as CRC checksums) is added to the compressed data to ensure data integrity. In this way, compressed data corresponding to each sparse module is obtained.
[0052] In this embodiment, a sparse module list is used to divide the compression tasks of all sparse modules into different compression subtasks. Parallel compression of all compression subtasks is achieved through independent threads, thereby improving compression efficiency and preventing the impact of excessive compression time on real-time data loading and querying.
[0053] In one embodiment, step S40, which involves integrating the data of all the sparse modules and all the ordinary modules to obtain a compressed bitmap file corresponding to the original bitmap data, includes: S401, based on the start and end positions in all the feature information, sort all the compressed data and all the encapsulated data and add global header information to obtain a compressed data stream.
[0054] S402, determine the module index table based on the starting offset, module length, module type, and compression algorithm identifier of all the sparse modules and all the ordinary modules in the compressed data stream.
[0055] S403, determine the algorithm index table based on the compression algorithm name, compression algorithm version, and entry address of the decompression function corresponding to each compression algorithm identifier.
[0056] S404, determine the compressed bitmap file based on the compressed data stream, the module index table, and the algorithm index table.
[0057] Understandably, the global header information includes metadata such as the length of the original bitmap, the total number of compressed modules, the number of sparse modules, the number of regular modules, and the compression time. A data stream refers to a sequential, continuous sequence of data. A compressed data stream refers to a sequence of all consecutive compressed data. A module index table is an index table used to query sparse or regular modules. An algorithm index table is an index table used to query algorithms. An index table is a data structure used to quickly locate data positions; it establishes a mapping relationship between keywords and data storage locations.
[0058] Specifically, based on the start and end positions in all feature information, all sparse modules and all ordinary modules are sorted and global header information is added. That is, according to the start and end positions in the feature information of each module in the original bitmap data, the compressed data in the sparse modules and the encapsulated data in the ordinary modules are arranged in sequence and global header information is added to form a continuous compressed data stream.
[0059] Then, based on the starting offset, module length, module type (sparse or normal), and compression algorithm identifier of all sparse modules and all normal modules in the compressed data stream, a module index table is constructed. Normal modules do not have an identifier. Similarly, an algorithm index table is constructed based on the compression algorithm name, compression algorithm version, and entry address of the decompression function corresponding to each compression algorithm identifier. The module index table and algorithm index table can be defined as the index structure. Next, based on the compressed data stream, module index table, and algorithm index table, the compressed bitmap file is determined; that is, the compressed data stream, module index table, and algorithm index table are defined as the compressed bitmap file.
[0060] In this embodiment, the sparse modules and ordinary modules are sorted by the start and end positions, thereby enabling the determination of the compressed data stream, the construction of the module index table and algorithm index table, and the determination of the compressed bitmap file. This facilitates subsequent queries of each module and algorithm and improves query performance.
[0061] In one embodiment, in step S40, the integrity of all the compressed data passes verification, including: S405, Perform checksum processing on all the compressed data to obtain an intermediate checksum corresponding to each of the compressed data.
[0062] S406, when the intermediate checksum and the initial checksum corresponding to the same compressed data are consistent, it is determined that the integrity of all the compressed data has passed the verification.
[0063] Understandably, the intermediate checksum refers to the new checksum after recalculating the compressed data. The initial checksum refers to the checksum calculated on the compressed data.
[0064] Specifically, all compressed data undergoes checksum processing. First, the 32-bit CRC register is initialized to 0xFFFFFFFF. Then, each byte of compressed data is processed bit-by-bit (from most significant bit to least significant bit): the most significant bit of the register is XORed with the current data bit. If the result is 1, the register is shifted left by 1 bit and then XORed with the polynomial 0xEDB88320; if the result is 0, it is only shifted left by 1 bit. After all bitmap data processing is complete, the register value is XORed with 0xFFFFFFFF to obtain the intermediate checksum corresponding to each compressed data. Then, the intermediate checksum corresponding to the same compressed data is compared with the initial checksum in the compressed data for consistency checking to determine if the two checksums are completely identical. If the intermediate checksum and the initial checksum match, the integrity check of the compressed data is considered passed. Thus, if all intermediate checksums and all initial checksums match, the integrity check of all compressed data is considered passed.
[0065] In this process, if the intermediate checksum and the initial checksum of at least one compressed data are inconsistent, the compressed data is marked as erroneous. The feature information corresponding to the erroneous data is then recompressed using the target compression algorithm to obtain new compressed data. This process continues until the integrity of all compressed data passes the integrity check. In this embodiment, by calculating the compressed data, the intermediate checksum is obtained, and the compressed data is recalculated. By comparing the intermediate checksum with the initial checksum, the consistency of the checksum is checked, thereby ensuring the integrity of the compressed data and effectively avoiding data loss or deviation problems caused by defects in the compression algorithm or data processing errors in the prior art.
[0066] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0067] In one embodiment, a bitmap data processing apparatus is provided, which corresponds one-to-one with the bitmap data processing methods described in the above embodiments. For example... Figure 3 As shown, the bitmap data processing device includes a feature extraction module 10, a module filtering module 20, a data compression module 30, and a data integration module 40. Detailed descriptions of each functional module are as follows: Feature extraction module 10 is used to extract features from the original bitmap data to obtain feature information corresponding to each initial module; The module filtering module 20 is used to filter out sparse modules and ordinary modules from all the initial modules based on all the feature information, and determine a list of sparse modules including compression algorithm identifiers, and a list of ordinary modules including encapsulation data of all the ordinary modules. The data compression module 30 is used to perform parallel compression of bitmap data in all the sparse modules based on the sparse module list to obtain compressed data corresponding to each sparse module. The data integration module 40 is used to integrate the data of all the sparse modules and all the ordinary modules when the integrity of all the compressed data passes the verification, so as to obtain a compressed bitmap file corresponding to the original bitmap data.
[0068] In one embodiment, the feature extraction module 10 includes: A data segmentation unit is used to segment the original bitmap data according to a preset sliding window to obtain at least one initial module and bitmap data corresponding to each initial module. A module information determination unit is used to count all bitmap data in each initial module to obtain a module density and length distribution list corresponding to each initial module; The feature information unit is used to determine the feature information corresponding to each of the initial modules based on the module density, the length distribution list, and the start and end positions corresponding to each of the initial modules.
[0069] In one embodiment, the module filtering module 20 includes: A density determination unit is used to determine whether the module density in the feature information corresponding to each initial module exceeds a preset density threshold. The condition judgment unit is used to detect whether the length distribution list in the feature information corresponding to each initial module satisfies the preset sparsity condition when the module density of the initial module is less than the preset density threshold. The module filtering unit is used to determine the initial module as a sparse module when the length distribution list of the initial modules meets the preset sparsity condition, and to determine the initial modules other than the sparse modules as ordinary modules.
[0070] In one embodiment, the module filtering module 20 further includes: The algorithm matching unit is used to match the feature information of all the sparse modules with the preset algorithms in the preset algorithm library to obtain the target compression algorithm corresponding to each of the sparse modules. An associated storage unit is used to associate the compression algorithm identifier corresponding to the target compression algorithm with the sparse module to obtain a list of sparse modules including the compression algorithm identifier; The data encapsulation unit is used to encapsulate the bitmap data in all the ordinary modules according to a preset storage format to obtain encapsulated data corresponding to each ordinary module, and to determine a list of ordinary modules based on all the encapsulated data and all the ordinary modules.
[0071] In one embodiment, the data compression module 30 includes: A task partitioning unit is used to divide the sparse module list into multiple compression subtasks, each of which contains several sparse modules. The data compression unit is used to compress the bitmap data in all the sparse modules according to the independent thread corresponding to each of the compression subtasks and the target compression algorithm corresponding to each of the compression algorithm identifiers, so as to obtain compressed data corresponding to each of the sparse modules.
[0072] In one embodiment, the data integration module 40 includes: The data stream unit is used to sort all the compressed data and all the encapsulated data according to the start and end positions in all the feature information and add global header information to obtain a compressed data stream; The module index unit is used to determine the module index table based on the starting offset, module length, module type, and compression algorithm identifier of all the sparse modules and all the ordinary modules in the compressed data stream; The algorithm index unit is used to determine the algorithm index table based on the compression algorithm name, compression algorithm version, and entry address of the decompression function corresponding to each compression algorithm identifier; The file determination unit is used to determine the compressed bitmap file based on the compressed data stream, the module index table, and the algorithm index table.
[0073] In one embodiment, the data integration module 40 further includes: A check code calculation unit is used to perform check code processing on all the compressed data to obtain an intermediate check code corresponding to each of the compressed data. The check code comparison unit is used to determine that the integrity of all the compressed data has passed the verification when the intermediate check code and the initial check code corresponding to the same compressed data are consistent.
[0074] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being used to perform the bitmap data processing method described above.
[0075] Specific limitations regarding the computer equipment, processor, and their various units and modules can be found in the limitations of the bitmap data processing method described above, and will not be repeated here. Each module in the aforementioned processor can be implemented entirely or partially through software, hardware, or a combination thereof. Understandably, the processor includes a processor, memory, network interface, and database connected via a device bus. Each module of the processor can be embedded in hardware or independent of the processor, or stored in memory in software form, so that the processor can call and execute the operations corresponding to each module. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating devices and computer programs in the non-volatile storage media. The database stores the data used in the bitmap data processing method described in the above embodiments. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a bitmap data processing method.
[0076] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the bitmap data processing method described above.
[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0079] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A bitmap data processing method, characterized in that, include: Feature extraction is performed on the original bitmap data to obtain feature information corresponding to each initial module; Based on all the aforementioned feature information, sparse modules and ordinary modules are selected from all the initial modules, and a list of sparse modules including compression algorithm identifiers and a list of ordinary modules including encapsulated data of all the ordinary modules are determined. Based on the sparse module list, the bitmap data in all the sparse modules are compressed in parallel to obtain compressed data corresponding to each sparse module. When the integrity of all the compressed data passes the verification, the data of all the sparse modules and all the ordinary modules are integrated to obtain a compressed bitmap file corresponding to the original bitmap data.
2. The bitmap data processing method as described in claim 1, characterized in that, The step of extracting features from the original bitmap data to obtain feature information corresponding to each initial module includes: The original bitmap data is segmented according to a preset sliding window to obtain at least one initial module and bitmap data corresponding to each initial module. By statistically analyzing all the bitmap data in each of the initial modules, a list of module density and length distributions corresponding to each initial module is obtained. Based on the module density, the length distribution list, and the start and end positions corresponding to each initial module, the feature information corresponding to each initial module is determined.
3. The bitmap data processing method as described in claim 2, characterized in that, The step of filtering sparse modules and ordinary modules from all the initial modules based on all the feature information includes: Determine whether the module density in the feature information corresponding to each initial module exceeds a preset density threshold. When the module density of the initial module is less than the preset density threshold, it is detected whether the length distribution list in the feature information corresponding to each initial module satisfies the preset sparsity condition. When the length distribution list of the initial modules meets the preset sparsity condition, the initial modules are determined as sparse modules, and all initial modules other than sparse modules are determined as ordinary modules.
4. The bitmap data processing method as described in claim 1, characterized in that, The determination of the sparse module list including the compression algorithm identifier and the ordinary module list including the encapsulation data of all the ordinary modules includes: The feature information of all the sparse modules is matched with the preset algorithms in the preset algorithm library to obtain the target compression algorithm corresponding to each sparse module; Associating the compression algorithm identifier corresponding to the target compression algorithm with the sparse module yields a list of sparse modules including the compression algorithm identifier; The bitmap data in all the ordinary modules is encapsulated according to a preset storage format to obtain encapsulated data corresponding to each ordinary module, and a list of ordinary modules is determined based on all the encapsulated data and all the ordinary modules.
5. The bitmap data processing method as described in claim 1, characterized in that, The step of parallelizing and compressing the bitmap data in all the sparse modules based on the sparse module list to obtain compressed data corresponding to each sparse module includes: The list of sparse modules is divided into multiple compression subtasks, and each compression subtask contains several sparse modules. Based on the independent thread corresponding to each compression subtask, the bitmap data in all the sparse modules is compressed using the target compression algorithm corresponding to each compression algorithm identifier to obtain the compressed data corresponding to each sparse module.
6. The bitmap data processing method as described in claim 1, characterized in that, The step of integrating the data of all the sparse modules and all the ordinary modules to obtain a compressed bitmap file corresponding to the original bitmap data includes: Based on the start and end positions of all the aforementioned feature information, all the compressed data and all the encapsulated data are sorted and global header information is added to obtain a compressed data stream. Based on the starting offset, module length, module type, and compression algorithm identifier of all the sparse modules and all the ordinary modules in the compressed data stream, a module index table is determined; An algorithm index table is determined based on the compression algorithm name, compression algorithm version, and decompression function entry address corresponding to each compression algorithm identifier; The compressed bitmap file is determined based on the compressed data stream, the module index table, and the algorithm index table.
7. The bitmap data processing method as described in claim 1, characterized in that, The integrity of all the compressed data passed the verification, including: Perform checksum processing on all the compressed data to obtain an intermediate checksum corresponding to each of the compressed data. If the intermediate checksum and the initial checksum corresponding to the same compressed data are consistent, it is determined that the integrity of all the compressed data has passed the verification.
8. A bitmap data processing apparatus, characterized in that, include: The feature extraction module is used to extract features from the original bitmap data to obtain feature information corresponding to each initial module; The module filtering module is used to filter out sparse modules and ordinary modules from all the initial modules based on all the feature information, and determine a list of sparse modules including compression algorithm identifiers, and a list of ordinary modules including encapsulation data of all the ordinary modules. The data compression module is used to perform parallel compression of bitmap data in all the sparse modules based on the sparse module list, so as to obtain compressed data corresponding to each sparse module. The data integration module is used to integrate the data of all the sparse modules and all the ordinary modules when the integrity of all the compressed data passes the verification, so as to obtain a compressed bitmap file corresponding to the original bitmap data.
9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being configured to perform the bitmap data processing method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the bitmap data processing method as described in any one of claims 1 to 7.