Dynamic compression reconstruction and query method and system based on semantic fingerprint perception

Abstract

The application relates to a dynamic compression reconstruction and query method and system based on semantic fingerprint perception. Target data blocks are initially compressed and packaged into evolvable compression blocks. In response to receiving a query request, a statistical fingerprint is used to prune the query request to obtain candidate evolvable compression blocks. On-demand decompression or calculation is performed based on the query request and the candidate evolvable compression blocks to obtain a target query result corresponding to the query request. The change result of the semantic fingerprint in the evolvable compression block is monitored, and when the change result meets a preset condition, adaptive switching of a compression algorithm is performed on the evolvable compression block to realize dynamic reconstruction. The application solves the problem of query performance degradation caused by the fact that a fixed compression structure cannot adapt to changes in access modes, improves query performance under dynamic loads, avoids the problem of large resource consumption caused by full-volume decompression, reduces computing resource overhead, and realizes dynamic balance between storage cost and query efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing in the next-generation information technology industry, and in particular to a dynamic compression, reconstruction, and query method and system based on semantic fingerprint awareness. Background Technology

[0002] Columnar storage, as a crucial technology in big data analytics, is widely applied in data warehousing, business intelligence, and other scenarios. Among related technologies, the collaborative operation of columnar storage and lightweight compression algorithms constructs an efficient data storage system. Specifically, this system encompasses key aspects such as data compression, storage tiering, and query optimization. It reduces space usage by employing algorithms like RLE (Run-Length Encoding) or dictionary encoding, and uses a tiered storage architecture to store hot and cold data on high-speed and low-cost media respectively.

[0003] However, existing columnar storage methods directly employ static compression structures and fixed hierarchical strategies. This can lead to mismatches between the compressed structure and query patterns, or complex queries requiring full decompression, thus impacting system performance and resource utilization efficiency. For example, when data access patterns change, the original compressed structure cannot adaptively adjust, resulting in decreased query performance. Furthermore, for complex queries involving multi-column joins, traditional solutions require full decompression, causing significant CPU (Central Processing Unit) consumption. Summary of the Invention

[0004] This application provides a dynamic compression, reconstruction, and query method and system based on semantic fingerprint awareness to solve the technical problems of low system performance and resource utilization efficiency in related technologies.

[0005] Firstly, this application provides a dynamic compression reconstruction and query method based on semantic fingerprint awareness, the method comprising: The target data block is initially compressed and encapsulated into an evolvable compressed block. The evolvable compressed block contains compressed data and metadata. The metadata includes a semantic fingerprint and a statistical fingerprint generated based on the content and access characteristics of the target data block. In response to receiving a query request, the statistical fingerprint is used to prune based on the query request to obtain candidate evolvable compressed blocks; Based on the query request and the candidate evolvable compression blocks, the compressed data is decompressed or calculated on demand to obtain the target query result corresponding to the query request; Monitor the changes in the semantic fingerprint in the evolvable compressed block, and when the changes meet preset conditions, perform adaptive switching of the compression algorithm on the evolvable compressed block to achieve dynamic reconstruction.

[0006] Optionally, the initial compression and encapsulation of the target data block into an evolvable compressed block includes: The target data block is compressed using a preset algorithm to obtain compressed data corresponding to the compressed bitstream; Determine the version information and algorithm identifier field of the current preset algorithm; Obtain the statistical feature information of the target data block, and generate the statistical fingerprint based on the statistical feature information, wherein the statistical fingerprint includes minimum / maximum extreme values, histogram bucket information, and Bloom filter bitmap; By performing hash operations on the multidimensional features of the target data block, a semantic fingerprint of the target data block is generated, and the algorithm identifier field, the statistical fingerprint, and the semantic fingerprint are determined as the metadata of the target data block. Set a status control bit to mark the lifecycle state and reconfiguration state of the evolvable compressed block, and combine the compressed data, the metadata, and the status control bit into an evolvable compressed block.

[0007] Optionally, generating a semantic fingerprint of the target data block by performing a hash operation on the multidimensional features of the target data block includes: Extract the distribution entropy of field values, the correlation matrix between fields, and the access popularity vector from the target data block; The distribution entropy, the correlation matrix, and the access popularity vector are used as multi-dimensional features input into the hash algorithm to generate the corresponding hash digest. The hash digest is determined as the semantic fingerprint of the target data block.

[0008] Optionally, the step of pruning based on the statistical fingerprint to obtain candidate evolvable compressed blocks according to the query request includes: Extract the data constraints contained in the query request, wherein the data constraints include at least one of time range, data source, or data size; Traverse the statistical fingerprints of each of the evolvable compression blocks, determine the evolvable compression blocks whose statistical fingerprints satisfy the data constraints as candidate evolvable compression blocks, and filter out the evolvable compression blocks that do not meet the conditions.

[0009] Optionally, the step of obtaining the target query result corresponding to the query request by performing on-demand decompression or calculation based on the query request and the candidate evolvable compressed blocks includes: When the query request contains an aggregation operation and uses run-length encoding (RLE), the numerical and count items of the RLE bitstream in the compressed data of the candidate evolvable compressed block are parsed, and the count item is multiplied by the corresponding numerical item as a scalar and accumulated. The resulting aggregation result is determined as the target query result corresponding to the query request. When the query request contains regular expression matching or fuzzy query, the pre-built Local Sensitive Hash (LSH) index is used to locate the matching target data range in the candidate evolvable compressed block, and the data after decompression of the target data range is determined as the target query result corresponding to the query request. When the query request includes multi-table join operations, a multi-hash bloom filter is used to remove mismatched key-value pairs in the candidate evolvable compressed blocks, and the data after decompression of the remaining candidate evolvable compressed blocks is determined as the target query result corresponding to the query request.

[0010] Optionally, the step of monitoring the changes in the semantic fingerprint within the evolvable compressed block, and adaptively switching the compression algorithm for the evolvable compressed block to achieve dynamic reconstruction when the changes meet preset conditions, includes: Monitor the rate of change of the semantic fingerprint in the evolvable compressed block; When the change rate of the semantic fingerprint is detected to exceed a preset threshold, it is determined that the change rate meets the preset condition. The target evolvable compressed block corresponding to the change rate is asynchronously read from the disk into the memory buffer through a background thread. The asynchronous reading process does not block the foreground query request. Based on the current encoding type and data distribution characteristics in the target evolvable compressed block, an appropriate reassembly strategy is selected, and the data blocks contained in the loaded target evolvable compressed block are recompressed using the reassembly strategy, and the corresponding metadata and status control bits are updated to obtain a new evolvable compressed block. The new evolvable compressed block is written to a new storage location on the disk using copy-on-write technology, and the pointer pointing to the target evolvable compressed block is atomically updated to point to the new storage location.

[0011] Optionally, the method further includes: The system collects system status information, business characteristic information, and data characteristic information in real time. The system status information includes CPU utilization, GPU utilization, memory level, and I / O queue length. The business characteristic information includes the current query type distribution. The data characteristic information includes the semantic fingerprint distribution of each level of evolvable compressed blocks. The system status information, business feature information and data feature information collected are modeled using an LSTM neural network to predict the distribution of query load types and the trend of access popularity within a future preset time window. Based on the query load type distribution, the access popularity change trend and the current system status, select the action to be executed in the predefined action space and generate a scheduling decision. The action space includes the execution path selection for the current query, the migration direction selection of data blocks between hard disk drives (HDD) and solid disk drives (SSD), and the compression algorithm selection for newly written data. The scheduling decision is executed, including switching query paths, migrating data, and adjusting compression algorithms.

[0012] Secondly, this application provides a dynamic compression reconstruction and query system based on semantic fingerprint awareness, the system comprising: A compression module is used to initially compress the target data block and encapsulate it into an evolvable compressed block. The evolvable compressed block contains compressed data and metadata. The metadata includes a semantic fingerprint and a statistical fingerprint generated based on the content and access characteristics of the target data block. A filtering module is used to respond to a received query request by pruning the statistical fingerprint based on the query request to obtain candidate evolvable compressed blocks. The query module is used to decompress or calculate the compressed data on demand based on the query request and the candidate evolvable compressed blocks, and obtain the target query result corresponding to the query request; The reconstruction module is used to monitor the changes in the semantic fingerprint in the evolvable compressed block, and when the changes meet preset conditions, to perform adaptive switching of the compression algorithm on the evolvable compressed block to achieve dynamic reconstruction.

[0013] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements any of the methods described in the first aspect.

[0014] Fourthly, this application provides a storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described in any of the first aspects.

[0015] Compared with the prior art, the technical solution provided in this application has the following advantages: the target data block is initially compressed and encapsulated into an evolvable compressed block, which contains compressed data and metadata. The metadata includes semantic fingerprints and statistical fingerprints generated based on the content and access characteristics of the target data block. In response to receiving a query request, the statistical fingerprint is used to prune based on the query request to obtain candidate evolvable compressed blocks. Based on the query request and the candidate evolvable compressed blocks, decompression or calculation is performed on demand to obtain the target query result corresponding to the query request. The change results of the semantic fingerprint in the evolvable compressed block are monitored, and when the change results meet the preset conditions, the compression algorithm is adaptively switched for the evolvable compressed block to achieve dynamic reconstruction. The method provided in this application introduces a semantic fingerprint that integrates content features and access behavior features during data compression and encapsulation. Based on the dynamic changes of the semantic fingerprint, the compression algorithm is adaptively switched to achieve reconstruction. This solves the problem of query performance degradation caused by the fixed compression structure failing to adapt to changes in access patterns in the prior art, and improves query performance under dynamic load. At the same time, by using metadata pruning and embedded indexes in compressed data to perform decompression or calculation in the compressed state, the method avoids the problem of high resource consumption caused by full decompression, reduces computational resource overhead, and achieves a dynamic balance between storage cost and query efficiency. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 A flowchart illustrating a dynamic compression, reconstruction, and query method based on semantic fingerprint awareness, provided for an embodiment of this application; Figure 2 A schematic diagram of the interaction process of a dynamic compression reconstruction and query method based on semantic fingerprint awareness provided in an embodiment of this application; Figure 3This is a schematic diagram of the structure of a dynamic compression reconstruction and query system based on semantic fingerprint awareness, provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0022] For ease of description, spatial relative terms may be used in the text to describe the relative position or movement of one element or feature relative to another element or feature, as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "below," "above," "front," "back," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure. For example, if the device in the figure undergoes a positional flip, orientation change, or change of motion, these directional indications will change accordingly. For instance, an element described as "below other elements or features" or "below other elements or features" will subsequently be oriented "above other elements or features" or "above other elements or features." Therefore, the example term "below" can include both upper and lower orientations. The device may be otherwise oriented (rotated 90 degrees or in other directions), and the spatial relative descriptors used in the text will be interpreted accordingly.

[0023] To address the low system performance and resource utilization efficiency issues in existing technologies, this application provides a dynamic compression, reconstruction, and query method based on semantic fingerprint awareness. By introducing a semantic fingerprint that integrates content features and access behavior features during data compression and encapsulation, and adaptively switching compression algorithms based on the dynamic changes of the semantic fingerprint to achieve reconstruction, this method solves the problem of decreased query performance caused by the fixed compression structure failing to adapt to changes in access patterns in existing technologies, thus improving query performance under dynamic loads. Simultaneously, by utilizing metadata pruning and embedded indexes within compressed data to perform decompression or computation in the compressed state, it avoids the high resource consumption caused by full decompression, reduces computational resource overhead, and achieves a dynamic balance between storage costs and query efficiency.

[0024] Figure 1 A flowchart illustrating a dynamic compression, reconstruction, and query method based on semantic fingerprint awareness, provided in an embodiment of this application, is shown below. Figure 1 As shown, the method may include the following steps: Step 101: Perform initial compression on the target data block and encapsulate it into an evolvable compressed block. The evolvable compressed block contains compressed data and metadata. The metadata includes semantic fingerprints and statistical fingerprints generated based on the content and access characteristics of the target data block.

[0025] In one embodiment of this application, the aforementioned evolvable compressed block has dynamic evolution capabilities and can serve as a basic unit for data storage. The evolvable compressed block may include compressed data, metadata, and status control bits. Specifically, the compressed data stores a binary bitstream processed by a specific compression algorithm. This bitstream does not contain plaintext data but retains the compressed encoded information. The metadata records the algorithm identifier, statistical fingerprint, and semantic fingerprint of the target data block. The algorithm identifier indicates the currently used compression algorithm version and parameter configuration. The statistical fingerprint provides fast filtering information such as extreme values, histogram binning information, and Bloom filter bitmaps for the target data block. The semantic fingerprint reflects the content distribution characteristics of the target data block and also incorporates the access pattern characteristics of the target data block, thereby enabling the compressed block to perceive its own state changes. The status control bits mark the current lifecycle state of the evolvable compressed block, such as cold, warm, hot, or under reconstruction, to support subsequent dynamic management.

[0026] In one embodiment of this application, the method for initially compressing and encapsulating a target data block into an evolvable compressed block may include the following steps: Step 1011: Compress the target data block using a preset algorithm to obtain the compressed data corresponding to the compressed bitstream.

[0027] In one embodiment of this application, the compressed bitstream, compressed using a preset algorithm, is identified as compressed data and stored. This compressed data does not contain any plaintext data. In one embodiment of this application, the preset algorithm may include RLE or a dictionary encoding algorithm.

[0028] In one embodiment of this application, if the target data block is detected to be a data pattern with a high repetition rate, RLE compression is selected; if the target data block is detected to be data with obvious classification attributes, dictionary encoding is used for compression; if the target data block is detected to be other sparse or mixed type data, a general compression algorithm can be selected.

[0029] For example, in one embodiment of this application, assuming RLE compression is used, the compressed bitstream is represented as [(temperature: 25, number of times: 100), (temperature: 26, number of times: 50)]).

[0030] Step 1012: Determine the version information of the current preset algorithm and the algorithm identifier field of the parameter configuration.

[0031] Step 1013: Obtain statistical feature information of the target data block and generate a statistical fingerprint based on the statistical feature information. The statistical fingerprint includes minimum / maximum extreme values, histogram binning information, and Bloom filter bitmap.

[0032] In one embodiment of this application, statistical fingerprints can be used to quickly exclude irrelevant data blocks in subsequent queries.

[0033] Step 1014: By performing hash operations on the multidimensional features of the target data block, a semantic fingerprint of the target data block is generated, and the algorithm identifier field, statistical fingerprint, and semantic fingerprint are determined as the metadata of the target data block.

[0034] In one embodiment of this application, the aforementioned multidimensional features may include the distribution entropy of field values ​​in the target data block, the correlation matrix between fields, and the recent access popularity vector; wherein, the distribution entropy characterizes the dispersion of data values, the correlation matrix reflects the association between different fields, and the access popularity vector records the historical frequency of the target data block being recently accessed by operation modes such as point-to-point query and range scan.

[0035] In one embodiment of this application, the method for generating a semantic fingerprint of a target data block by performing hash operations on the multidimensional features of the target data block may include: extracting the distribution entropy of field values, the correlation matrix between fields, and the access popularity vector in the target data block; inputting the distribution entropy, the correlation matrix, and the access popularity vector as multidimensional features into a hash algorithm to generate a corresponding hash digest; and determining the hash digest as the semantic fingerprint of the target data block.

[0036] In one embodiment of this application, distribution entropy, correlation matrix, and access popularity vector can be used as multi-dimensional features input into the SHA3-512 hash algorithm to generate a unique 512-bit hash digest, i.e., a semantic fingerprint. This semantic fingerprint not only identifies the statistical characteristics of the data content but also incorporates the data's access behavior characteristics, enabling it to proactively reflect its current usage patterns.

[0037] Step 1015: Set the status control bit, mark the lifecycle status and reconstruction status of the evolvable compressed block, and combine the compressed data, metadata and status control bit into an evolvable compressed block.

[0038] In one embodiment of this application, to support lifecycle management, the evolvable compressed block also embeds a status control bit to mark whether the evolvable compressed block is currently in a cold, warm or hot state, and whether it is in the process of reconstruction.

[0039] In one embodiment of this application, the compressed data, metadata, and status control bits are encapsulated together into an Evolvable Compressible Block (ECB) and stored in the underlying storage medium (such as HDD (Hard Disk Drive), SSD (Solid State Drive), or object storage) in an atomic write manner.

[0040] In one embodiment of this application, each data block possesses both data storage and behavior awareness capabilities through the aforementioned evolvable compressed blocks, thereby providing accurate decision-making basis for subsequent zero-cost metadata pruning and dynamic compressed state reconstruction, improving query filtering efficiency and the accuracy of data block adaptive adjustment.

[0041] Step 102: In response to receiving a query request, pruning is performed based on the query request using statistical fingerprints to obtain candidate evolvable compressed blocks.

[0042] In one embodiment of this application, when a query request is received, statistical fingerprints can be used to perform pruning filtering based on the query request to obtain candidate evolvable compressed blocks.

[0043] In one embodiment of this application, the method for obtaining candidate evolvable compressed blocks by pruning based on statistical fingerprints according to query requests may include the following steps: Step 1021: Extract the data constraints contained in the query request. The data constraints include at least one of the following: time range, data source, or data size. Step 1022: Traverse the statistical fingerprints of each evolvable compression block, determine the evolvable compression blocks whose statistical fingerprints satisfy the data constraints as candidate evolvable compression blocks, and filter out the evolvable compression blocks that do not meet the conditions.

[0044] In one embodiment of this application, the query request may include data constraints, which include at least one of time range, data source, or data size. For example, assuming the query request is SELECT SUM(temp) FROM table WHERE temp>30 AND user_id = 'X', then the corresponding data constraint in this query request is temp>30, and the data source is user_id = 'X'.

[0045] In one embodiment of this application, the statistical fingerprints of each stored evolvable compressed block are compared with the data constraints one by one. Evolvable compressed blocks with matching sources and data sizes within the constraints are marked as candidate evolvable compressed blocks. The remaining evolvable compressed blocks that do not meet the data constraints are directly filtered out, thereby avoiding subsequent compressed state queries and decompression operations on irrelevant data and reducing computational load and I / O (Input / Output) overhead.

[0046] For example, in one embodiment of this application, assuming the data constraint is temp>30, and the maximum value of the statistical fingerprint record in the evolvable compressed block is 25, it will be directly determined that there is no matching data in the evolvable compressed block and it can be directly filtered.

[0047] In one embodiment of this application, zero-cost pruning can be performed through the above steps. Only the statistical fingerprint in the metadata of the evolvable compressed block is read. Using the extreme values, histogram bucketing information or Bloom filter bitmap recorded therein, evolvable compressed blocks that do not match the query conditions can be quickly excluded. This process does not require access to the data body area or any decompression operation, thereby reducing the consumption of computing resources.

[0048] Step 103: Based on the query request and candidate evolvable compressed blocks, decompress or calculate the compressed data on demand to obtain the target query result corresponding to the query request.

[0049] In one embodiment of this application, after obtaining candidate evolvable compressed blocks through the above steps, compressed data can be decompressed or calculated on demand based on query requests and candidate evolvable compressed blocks to obtain the target query results corresponding to the query requests, thereby avoiding the problem of high resource consumption caused by full decompression and reducing computing resource overhead.

[0050] In one embodiment of this application, for evolvable compressed blocks that are not excluded by pruning, the system enters the compressed state native algebra operation stage. In this stage, the system directly performs arithmetic operations on the compressed data or uses the embedded index structure to locate the matching interval, thereby avoiding full decompression.

[0051] Specifically, in one embodiment of this application, the method for obtaining the target query result corresponding to the query request by performing on-demand decompression or calculation based on the query request and candidate evolvable compressed blocks may include the following steps: Step 1031: When the query request contains aggregation operations and uses RLE, parse the numerical and count items of the RLE bitstream in the compressed data of the candidate evolvable compressed block, multiply the count items as scalars with the corresponding numerical items and accumulate them, and determine the resulting aggregation result as the target query result corresponding to the query request.

[0052] In one embodiment of this application, assuming the query request is SELECT SUM(temperature) FROM sensor data table, the query request includes the aggregation operation SUM, the candidate evolvable compressed block adopts RLE, and the compressed bitstream is represented as [(temperature: 25, count: 100), (temperature: 26, count: 50)]). In this case, arithmetic operations can be directly performed on the compressed bitstream: SUM = (25 × 100) + (26 × 50), treating it as a scalar and directly participating in the accumulation, avoiding the process of decompressing the 150 values ​​in the evolvable compressed block into memory.

[0053] Step 1032: When the query request contains regular expression matching or fuzzy query, the pre-built Local Sensitive Hash (LSH) index is used to locate the matching target data range in the candidate evolvable compressed block, and the data after decompression of the target data range is determined as the target query result corresponding to the query request.

[0054] In one embodiment of this application, when a query request contains regular expression matching or fuzzy queries, a built-in locality-sensitive hash index is used to quickly locate the potential matching interval in compressed mode. The query conditions are compared with the bucket index corresponding to the compressed data through hash mapping to obtain the target data interval. Based on this, it is determined that the query result corresponding to the query request is located in the target data interval. Therefore, this very small portion of data can be decompressed as needed, and its corresponding plaintext format can be determined as the target query result corresponding to the query request and returned.

[0055] Step 1033: When the query request contains multi-table join operations, use a multi-hash bloom filter to remove mismatched key-value pairs in the candidate evolvable compressed blocks, and determine the decompressed data of the remaining candidate evolvable compressed blocks as the target query result corresponding to the query request.

[0056] In one embodiment of this application, when a query request includes multi-table join operations, a multi-hash bloom filter can be used to remove mismatched key-value pairs, retain candidate key-value pairs that meet the minimum probability, and determine the data after decompressing the candidate evolvable compressed blocks corresponding to the candidate key-value pairs as the target query result corresponding to the query request.

[0057] In one embodiment of this application, the above steps maximize the use of data locality in the compressed state, avoid the waste of computing resources caused by full decompression, and at the same time ensure the integrity and correctness of query semantics, and reduce data processing latency and resource consumption.

[0058] Step 104: Monitor the changes in semantic fingerprints in the evolvable compressed block, and when the changes meet preset conditions, perform adaptive switching of compression algorithms on the evolvable compressed block to achieve dynamic reconstruction.

[0059] In one embodiment of this application, during data querying via evolvable compressed blocks, the data storage structure can be dynamically and adaptively adjusted by continuously monitoring the access patterns of these blocks. This involves real-time monitoring of the access characteristics of each evolvable compressed block, including but not limited to query type, frequency, and locality changes in data access, and comparing these dynamic behavioral characteristics with semantic fingerprints recorded in the metadata. Furthermore, the semantic fingerprint, as a hash digest of the multidimensional features of the data block (such as field value distribution, relevance, and access frequency), can sensitively reflect substantial changes in data access patterns.

[0060] In one embodiment of this application, when the rate of change of the semantic fingerprint exceeds a preset threshold, it is determined that the current compression algorithm is no longer suitable for the new access mode. A background asynchronous reconstruction thread is then initiated. This reconstruction thread runs independently in the background, parallel to the foreground query processing, to avoid interfering with online services. Furthermore, during the reconstruction process, a compression algorithm switching and reorganization operation is performed on the target evolvable compressed block. That is, a better compression strategy is selected based on the current data characteristics and access mode, and the data block is reconstructed in memory to generate a new evolvable compressed block. Further, the new evolvable compressed block replaces the old block atomically, ensuring data consistency and operational integrity.

[0061] In one embodiment of this application, the method of monitoring the changes in semantic fingerprints in an evolvable compressed block and, when the changes meet preset conditions, adaptively switching the compression algorithm on the evolvable compressed block to achieve dynamic reconstruction may include the following steps: Step 1041: Monitor the rate of change of semantic fingerprints in the evolvable compressed block.

[0062] In one embodiment of this application, changes in the semantic fingerprint of an evolvable compressed block can be detected in real time by analyzing access logs. Specifically, when the access pattern changes, for example, when an access pattern that was originally dominated by range scanning suddenly changes to a high-frequency random query, the rate of change between the current semantic fingerprint and the historical semantic fingerprint is calculated.

[0063] Step 1042: When the change rate of the semantic fingerprint is detected to exceed the preset threshold, it is determined that the change rate meets the preset condition. The target evolvable compressed block corresponding to the change rate is asynchronously read from the disk into the memory buffer through the background thread. The asynchronous reading process does not block the foreground query request.

[0064] In one embodiment of this application, when the change rate of the semantic fingerprint exceeds a preset threshold, it is determined that the preset condition is met, and a background asynchronous reconstruction thread is started. Specifically, in one embodiment of this application, the background thread asynchronously loads the affected target evolvable compressed block from the disk into a memory buffer. This process is completed through asynchronous I / O operations and does not block the foreground query response.

[0065] In one embodiment of this application, the aforementioned preset threshold can be determined as needed or based on statistical data, such as 30%.

[0066] In one embodiment of this application, when the query frequency exceeds a preset number of times per second (e.g., more than 1,000 queries per second), and it is determined that the preset condition is met, the target evolvable compressed block corresponding to the rate of change is asynchronously read from the disk into the memory buffer through a background thread. The asynchronous reading process does not block the foreground query request.

[0067] Step 1043: Select an appropriate recombination strategy based on the current encoding type and data distribution characteristics in the target evolvable compressed block, and recompress the data blocks contained in the loaded target evolvable compressed block using the recombination strategy, and update the corresponding metadata and status control bits to obtain a new evolvable compressed block.

[0068] In one embodiment of this application, an appropriate reassembly strategy can be selected based on the current encoding type and data distribution characteristics in the target evolvable compressed block. Specifically, if the original compression algorithm is run-length encoding (RLE), and the current access mode is mainly high-frequency discrete value lookup, the reassembly strategy is to reassemble it into a hybrid structure of dictionary encoding and bitmap indexing. The dictionary encoding stores a list of unique values, and the bitmap index records the set of row numbers corresponding to each value, thereby supporting fast lookup. If the data blocks in the target evolvable compressed block are sparse data (i.e., a large number of null values ​​or few duplicate values), the reassembly strategy is to merge them into dense blocks to improve scanning efficiency. In one embodiment of this application, during the reconstruction process, the status control bit can be marked as "reconstruction in progress."

[0069] In one embodiment of this application, the data blocks contained in the loaded target evolvable compressed block are recompressed using a reorganization strategy, and the corresponding metadata and status control bits are updated to obtain a new evolvable compressed block. At this time, the status control bits can be marked as not being reconstructed.

[0070] Step 1044: The new evolvable compressed block is written to a new storage location on the disk using copy-on-write technology, and the pointer pointing to the target evolvable compressed block is atomically updated to point to the new storage location.

[0071] In one embodiment of this application, after obtaining a new evolvable compressed block through the above-mentioned reconstruction, the new evolvable compressed block is written to a new location on the disk using copy-on-write technology, and the pointer pointing to the new evolvable compressed block is atomically updated to ensure that the upper-layer application can still access the old block normally during the reconstruction period, and that data consistency is guaranteed.

[0072] In one embodiment of this application, by introducing a background asynchronous reconstruction mechanism triggered by semantic fingerprint change rate, the ability of the compressed structure to dynamically evolve with the access mode is realized, overcoming the rigidity defect of "one-time compression, lifelong unchanged" in the traditional static compression scheme. Thus, the storage format can be adaptively adjusted without interrupting the service, thereby continuously optimizing the balance between query performance and storage efficiency and reducing performance jitter caused by sudden changes in access mode.

[0073] In one embodiment of this application, a deep reinforcement learning model can be used to model and predict the collected system state information, business characteristic information, and data characteristic information to infer the distribution of query load types and the trend of access popularity within a future time window. Furthermore, based on the prediction results, scheduling decisions are dynamically generated within a predefined action space. These scheduling decisions cover the selection of query execution paths, the migration direction of data between different storage levels, and the type of compression algorithm used for newly written data. Based on this, through a prediction-based proactive scheduling mechanism, a global optimal balance between storage resource consumption and computational performance overhead can be achieved while ensuring that query response latency meets service quality requirements.

[0074] Specifically, in one embodiment of this application, the above method may further include the following steps: Step 105: Real-time collection of system status information, business characteristic information, and data characteristic information. System status information includes CPU utilization, GPU (Graphics Processing Unit) utilization, memory level, and I / O queue length; business characteristic information includes the current query type distribution; and data characteristic information includes the semantic fingerprint distribution of each level of evolvable compressed blocks. Step 106: Model the collected system status information, business characteristic information and data characteristic information using an LSTM neural network to predict the distribution of query load types and the trend of access popularity within a preset time window in the future. Step 107: Based on the distribution of query load types, the trend of access popularity and the current system status, select the action to be executed in the predefined action space and generate a scheduling decision. The action space includes the selection of the execution path of the current query, the selection of the migration direction of data blocks between HDD and SSD, and the selection of the compression algorithm for newly written data. Step 108: Execute scheduling decisions, including query path switching, data migration, and compression algorithm adjustments.

[0075] In one embodiment of this application, the type (such as range scan, point search, aggregation operation), frequency and response latency of each query within a preset time period are recorded to obtain business characteristic information; the current CPU / GPU utilization, memory level and I / O queue length status indicators are obtained to obtain system status information; and data characteristic information is obtained through the semantic fingerprint distribution of each level of evolvable compressed blocks, that is, the access popularity and data distribution characteristics reflected by the multidimensional feature hash digest of each evolvable compressed block.

[0076] In one embodiment of this application, steps 106-108 can be processed by a deployed LSTM-RL agent. Specifically, an LSTM neural network performs time-series modeling on system state information, business characteristic information, and data characteristic information to predict traffic trends in future time windows (e.g., the next hour), including query load type distribution and access popularity trends. Based on this prediction, the reinforcement learning module can make decisions according to a preset reward function. The goal of this reward function is to minimize storage costs and energy consumption while ensuring that the query latency is below a preset query latency threshold (e.g., Latency < 30 milliseconds). For example, the reward function could be... , The current actual query delay, The delay threshold, For current storage costs, C max To allow for maximum storage cost, E pow E represents the current operating energy consumption. max To allow maximum energy consumption, P penalty The penalty value for exceeding the delay limit is a negative number with a very large absolute value. α and β are weighting coefficients that balance the importance of storage and energy consumption, with α>0 and β>0.

[0077] In one embodiment of this application, the dynamic decisions output by the LSTM-RL agent may specifically include: the selection of the query execution path, i.e., deciding whether the current query should be executed in pure compression, partial decompression, or full decompression; the data migration direction, i.e., determining whether data blocks need to be migrated between HDD and SSD to match the predicted access pattern; and the selection of the compression algorithm for newly written data, i.e., selecting the most suitable compression algorithm (such as RLE, dictionary encoding, or hybrid structure) for the data blocks to be written based on data characteristics and future access expectations. Furthermore, the agent performs scheduling decisions, switching query paths, migrating data, and adjusting compression algorithms to achieve a global balance between storage cost and query performance.

[0078] In one embodiment of this application, by introducing a predictive scheduling strategy based on deep reinforcement learning, the limitations of traditional static rules or threshold-driven resource management are broken. This allows the system to proactively adapt to dynamically changing access patterns and load fluctuations, reducing storage costs while ensuring the stability of query performance, thus achieving an adaptive global balance between cost and efficiency in the storage system.

[0079] This application presents a dynamic compression, reconstruction, and query method based on semantic fingerprint awareness. By introducing a semantic fingerprint that integrates content features and access behavior features during data compression and encapsulation, and adaptively switching compression algorithms based on the dynamic changes of the semantic fingerprint to achieve reconstruction, this method solves the problem of query performance degradation caused by the fixed compression structure failing to adapt to changes in access patterns in existing technologies, thus improving query performance under dynamic loads. At the same time, by utilizing metadata pruning and embedded indexes in compressed data to perform decompression or computation in the compressed state, this method avoids the problem of high resource consumption caused by full decompression, reduces computational resource overhead, and achieves a dynamic balance between storage costs and query efficiency.

[0080] Figure 2 This application provides an example of an interactive flow diagram of a dynamic compression reconstruction and query method based on semantic fingerprint awareness. Figure 2As shown, the method may include: Stage 1: Normal operation and monitoring, including users sending query requests and returning query results using the above query method with hard disk storage; Stage 2: Triggering a reconstruction strategy, continuously collecting access logs through a monitoring agent to obtain the semantic fingerprint change rate or query frequency, and determining whether the semantic fingerprint change rate or query frequency exceeds a preset threshold through the reconstruction module. If it is determined that it exceeds the preset threshold, a reconstruction instruction is sent to the background reconstruction thread; the background reconstruction thread asynchronously reads the old ECB block (i.e., the target evolvable compressed block in the above embodiment) into the memory buffer, reassembles the data in memory to obtain a new ECB block (i.e., the new evolved compressed block in the above embodiment), writes the new ECB block into the disk storage, and switches the atomic pointer; subsequent queries automatically point to the new ECB block, and the old ECB block is marked as reclaimed; Stage 3: Continued operation based on the new ECB block, including users sending query requests and returning query results using the above query method with hard disk storage.

[0081] Figure 3 A schematic diagram of a dynamic compression reconstruction and query system based on semantic fingerprint awareness, provided in an embodiment of this application, is shown below. Figure 3 As shown, the system may include: Compression module 301 is used to initially compress the target data block and encapsulate it into an evolvable compressed block. The evolvable compressed block contains compressed data and metadata. The metadata includes semantic fingerprints and statistical fingerprints generated based on the content and access characteristics of the target data block. The filtering module 302 is used to respond to the received query request by performing pruning based on the query request using statistical fingerprints to obtain candidate evolvable compressed blocks; The query module 303 is used to decompress or calculate compressed data on demand based on the query request and candidate evolvable compressed blocks, and obtain the target query result corresponding to the query request. The reconstruction module 304 is used to monitor the changes in semantic fingerprints in the evolvable compressed block, and when the changes meet preset conditions, to perform adaptive switching of compression algorithms on the evolvable compressed block to achieve dynamic reconstruction.

[0082] In one embodiment of this application, the compression module 301 is specifically used for: The compressed data corresponding to the compressed bitstream is obtained by compressing the target data block using a preset algorithm; Determine the version information and algorithm identifier field of the current preset algorithm; Obtain statistical feature information of the target data block and generate a statistical fingerprint based on the statistical feature information. The statistical fingerprint includes minimum / maximum extreme values, histogram binning information, and Bloom filter bitmap. By performing hash operations on the multidimensional features of the target data block, a semantic fingerprint of the target data block is generated, and the algorithm identifier field, statistical fingerprint, and semantic fingerprint are identified as the metadata of the target data block. Set status control bits to mark the lifecycle state and refactoring state of the evolvable compressed block, and combine compressed data, metadata, and status control bits into an evolvable compressed block.

[0083] In one embodiment of this application, the compression module 301 is further configured to: Extract the distribution entropy of field values, the correlation matrix between fields, and the access popularity vector from the target data block; The distribution entropy, correlation matrix, and access popularity vector are used as multi-dimensional features input into the hash algorithm to generate the corresponding hash digest. The hash digest is used to determine the semantic fingerprint of the target data block.

[0084] In one embodiment of this application, the screening module 302 is specifically used for: Extract the data constraints contained in the query request. The data constraints include at least one of the following: time range, data source, or data size. Iterate through the statistical fingerprints of each evolvable compressed block, identify evolvable compressed blocks whose statistical fingerprints satisfy the data constraints as candidate evolvable compressed blocks, and filter out evolvable compressed blocks that do not meet the conditions.

[0085] In one embodiment of this application, the query module 303 is specifically used for: When a query request contains aggregation operations and uses run-length encoded RLE, the numerical and count items of the RLE bitstream in the compressed data of the candidate evolvable compressed block are parsed, and the count items are multiplied by the corresponding numerical items as scalars and accumulated. The resulting aggregation result is determined as the target query result corresponding to the query request. When a query request contains regular expression matching or fuzzy query, the pre-built Local Sensitive Hash (LSH) index is used to locate the matching target data range in the candidate evolvable compressed block, and the data after decompression of the target data range is determined as the target query result corresponding to the query request. When a query request involves multi-table join operations, a multi-hash bloom filter is used to remove mismatched key-value pairs from the candidate evolvable compressed blocks, and the data after decompression of the remaining candidate evolvable compressed blocks is determined as the target query result corresponding to the query request.

[0086] In one embodiment of this application, the reconstruction module 304 is specifically used for: Monitor the rate of change of semantic fingerprints in evolvable compressed blocks; When the rate of change of the semantic fingerprint is detected to exceed the preset threshold, it is determined that the rate of change meets the preset condition. The target evolvable compressed block corresponding to the rate of change is asynchronously read from the disk into the memory buffer through the background thread. The asynchronous reading process does not block the foreground query request. Based on the current encoding type and data distribution characteristics in the target evolvable compressed block, an appropriate reassembly strategy is selected, and the data blocks contained in the loaded target evolvable compressed block are recompressed through the reassembly strategy, and the corresponding metadata and status control bits are updated to obtain a new evolvable compressed block. The new evolvable compressed block is written to a new storage location on the disk using copy-on-write technology, and the pointer to the target evolvable compressed block is atomically updated to point to the new storage location.

[0087] In one embodiment of this application, the system is further used for: The system collects system status information, business characteristic information, and data characteristic information in real time. System status information includes CPU utilization, GPU utilization, memory level, and I / O queue length; business characteristic information includes the current query type distribution; and data characteristic information includes the semantic fingerprint distribution of each level of evolvable compressed blocks. By using an LSTM neural network to model the collected system status information, business characteristic information, and data characteristic information, the system can predict the distribution of query load types and the trend of access popularity within a future preset time window. Based on the distribution of query load types, the trend of access popularity, and the current system status, select actions to be executed from the predefined action space and generate scheduling decisions. The action space includes the selection of the execution path for the current query, the selection of the migration direction of data blocks between HDD and SSD, and the selection of compression algorithms for newly written data. Execute scheduling decisions, including switching query paths, migrating data, and adjusting compression algorithms.

[0088] This application presents a dynamic compression reconstruction and query system based on semantic fingerprint awareness. By introducing a semantic fingerprint that integrates content features and access behavior features during data compression and encapsulation, and adaptively switching compression algorithms based on the dynamic changes of the semantic fingerprint to achieve reconstruction, this solves the problem of query performance degradation caused by the fixed compression structure failing to adapt to changes in access patterns in existing technologies, thus improving query performance under dynamic loads. At the same time, by utilizing metadata pruning and embedded indexes in compressed data to perform decompression or computation in the compressed state, this avoids the problem of high resource consumption caused by full decompression, reduces computational resource overhead, and achieves a dynamic balance between storage costs and query efficiency.

[0089] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0090] Although terms such as first, second, third, etc., may be used in this document to describe multiple elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or segment from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and other numerical terms used herein do not imply order or sequence. Therefore, the first element, component, region, layer, or segment discussed below may be referred to as the second element, component, region, layer, or segment without departing from the teachings of the exemplary embodiments.

[0091] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A dynamic compression, reconstruction, and query method based on semantic fingerprint awareness, characterized in that, The method includes: The target data block is initially compressed and encapsulated into an evolvable compressed block. The evolvable compressed block contains compressed data and metadata. The metadata includes a semantic fingerprint and a statistical fingerprint generated based on the content and access characteristics of the target data block. In response to receiving a query request, the statistical fingerprint is used to prune based on the query request to obtain candidate evolvable compressed blocks; Based on the query request and the candidate evolvable compression blocks, the compressed data is decompressed or calculated on demand to obtain the target query result corresponding to the query request; Monitor the changes in the semantic fingerprint in the evolvable compressed block, and when the changes meet preset conditions, perform adaptive switching of the compression algorithm on the evolvable compressed block to achieve dynamic reconstruction.

2. The method according to claim 1, characterized in that, The initial compression and encapsulation of the target data block into an evolvable compressed block includes: The target data block is compressed using a preset algorithm to obtain compressed data corresponding to the compressed bitstream; Determine the version information and algorithm identifier field of the current preset algorithm; Obtain the statistical feature information of the target data block, and generate the statistical fingerprint based on the statistical feature information, wherein the statistical fingerprint includes minimum / maximum extreme values, histogram binning information, and Bloom filter bitmap; By performing hash operations on the multidimensional features of the target data block, a semantic fingerprint of the target data block is generated, and the algorithm identifier field, the statistical fingerprint, and the semantic fingerprint are determined as the metadata of the target data block. Set a status control bit to mark the lifecycle state and reconfiguration state of the evolvable compressed block, and combine the compressed data, the metadata, and the status control bit into an evolvable compressed block.

3. The method according to claim 2, characterized in that, The step of generating a semantic fingerprint of the target data block by performing a hash operation on the multidimensional features of the target data block includes: Extract the distribution entropy of field values, the correlation matrix between fields, and the access popularity vector from the target data block; The distribution entropy, the correlation matrix, and the access popularity vector are used as multi-dimensional features input into the hash algorithm to generate the corresponding hash digest. The hash digest is determined as the semantic fingerprint of the target data block.

4. The method according to claim 1, characterized in that, The step of pruning based on the statistical fingerprint to obtain candidate evolvable compressed blocks based on the query request includes: Extract the data constraints contained in the query request, wherein the data constraints include at least one of time range, data source, or data size; Traverse the statistical fingerprints of each of the evolvable compression blocks, determine the evolvable compression blocks whose statistical fingerprints satisfy the data constraints as candidate evolvable compression blocks, and filter out the evolvable compression blocks that do not meet the conditions.

5. The method according to claim 2, characterized in that, The step of performing on-demand decompression or computation based on the query request and the candidate evolvable compressed blocks to obtain the target query result corresponding to the query request includes: When the query request contains an aggregation operation and uses run-length encoding (RLE), the numerical and count items of the RLE bitstream in the compressed data of the candidate evolvable compressed block are parsed, and the count item is multiplied by the corresponding numerical item as a scalar and accumulated. The resulting aggregation result is determined as the target query result corresponding to the query request. When the query request contains regular expression matching or fuzzy query, the pre-built Local Sensitive Hash (LSH) index is used to locate the matching target data range in the candidate evolvable compressed block, and the data after decompression of the target data range is determined as the target query result corresponding to the query request. When the query request includes multi-table join operations, a multi-hash bloom filter is used to remove mismatched key-value pairs in the candidate evolvable compressed blocks, and the data after decompression of the remaining candidate evolvable compressed blocks is determined as the target query result corresponding to the query request.

6. The method according to claim 1, characterized in that, The step of monitoring the changes in the semantic fingerprint within the evolvable compressed block, and adaptively switching the compression algorithm for the evolvable compressed block to achieve dynamic reconstruction when the changes meet preset conditions, includes: Monitor the rate of change of the semantic fingerprint in the evolvable compressed block; When the change rate of the semantic fingerprint is detected to exceed a preset threshold, it is determined that the change rate meets the preset condition. The target evolvable compressed block corresponding to the change rate is asynchronously read from the disk into the memory buffer through a background thread. The asynchronous reading process does not block the foreground query request. Based on the current encoding type and data distribution characteristics in the target evolvable compressed block, an appropriate reassembly strategy is selected, and the data blocks contained in the loaded target evolvable compressed block are recompressed using the reassembly strategy, and the corresponding metadata and status control bits are updated to obtain a new evolvable compressed block. The new evolvable compressed block is written to a new storage location on the disk using copy-on-write technology, and the pointer pointing to the target evolvable compressed block is atomically updated to point to the new storage location.

7. The method according to claim 1, characterized in that, The method further includes: The system collects system status information, business characteristic information, and data characteristic information in real time. The system status information includes CPU utilization, GPU utilization, memory level, and I / O queue length. The business characteristic information includes the current query type distribution. The data characteristic information includes the semantic fingerprint distribution of each level of evolvable compressed blocks. The system status information, business feature information and data feature information collected are modeled using an LSTM neural network to predict the distribution of query load types and the trend of access popularity within a future preset time window. Based on the query load type distribution, the access popularity change trend and the current system status, select the action to be executed in the predefined action space and generate a scheduling decision. The action space includes the execution path selection for the current query, the migration direction selection of data blocks between hard disk drives (HDD) and solid disk drives (SSD), and the compression algorithm selection for newly written data. The scheduling decision is executed, including switching query paths, migrating data, and adjusting compression algorithms.

8. A dynamic compression reconstruction and query system based on semantic fingerprint awareness, characterized in that, The system includes: A compression module is used to initially compress the target data block and encapsulate it into an evolvable compressed block. The evolvable compressed block contains compressed data and metadata. The metadata includes a semantic fingerprint and a statistical fingerprint generated based on the content and access characteristics of the target data block. A filtering module is used to respond to a received query request by pruning the statistical fingerprint based on the query request to obtain candidate evolvable compressed blocks. The query module is used to decompress or calculate the compressed data on demand based on the query request and the candidate evolvable compressed blocks, and obtain the target query result corresponding to the query request; The reconstruction module is used to monitor the changes in the semantic fingerprint in the evolvable compressed block, and when the changes meet preset conditions, to perform adaptive switching of the compression algorithm on the evolvable compressed block to achieve dynamic reconstruction.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, A storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method as described in any one of claims 1-7.

Images