A high-performance and high-compression label calculation method

Abstract

The present invention relates to the field of data processing technology, and specifically to a high-performance, high-compression label operation method, including defining labels and operation rules through a UI end; parsing the labels and operation rules into three-segment execution plans through an application end; storing the metadata of the labels in the form of a closure table through a metadata end; and performing operations and compression storage on the labels based on the three-segment execution plans and operation rules using a bitmap through a data end. The present invention optimizes the logical results of the labels by using a bitmap. The bitmap is an expanded roaring bitmap. The roaring bitmap itself has an ultra-high compression rate and operation efficiency. Based on the underlying implementation of the roaring bitmap, the present invention also has the advantages of the roaring bitmap, and can directly perform operations and compressed storage on the labels, solving the problem of wasting memory when the labels are sparse.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a high-performance and high-compression-ratio label calculation method. Background Art

[0002] Traditional methods for compressed tag storage involve either compressed storage of details or compressed storage of bitmaps. The problem with compressed storage of details is that it consumes space, requires a large amount of I / O, and consumes a large amount of memory after loading. Computation of tags stored in compressed detail relies on database statements such as JOIN, IN, UNION, ANTI JOIN, EXCEPT, and NOT IN. Complex multi-tag operations require lengthy structured query language (SQL) and are complex to write. Furthermore, computational efficiency is low when the number of tags is large.

[0003] The compressed storage bitmap method can effectively reduce the complexity of writing label operation rules, and the operation efficiency after loading into memory is also considerable. However, the problem with compressed storage bitmap is that when the labels are sparse, there are a large number of vacancies, which occupy a lot of invalid space, and extracting them into memory will also waste memory. Summary of the Invention

[0004] The purpose of the present invention is to provide a high-performance and high-compression label calculation method, aiming to solve the problem of memory waste when labels are sparse.

[0005] To achieve the above object, the present invention provides a high-performance and high-compression label calculation method, comprising the following steps:

[0006] Define tags and operation rules through the UI;

[0007] Parsing the label and the operation rule into a three-stage execution plan through the application end;

[0008] Storing metadata of the tag in the form of a closure table through a metadata end;

[0009] The data end uses a bitmap method to calculate and compress the tags based on the three-stage execution plan and the operation rules.

[0010] The metadata includes the name, identifier and tree structure of the tag;

[0011] The labels include basic labels, overlay labels and rejection labels.

[0012] The three execution plans include a label screening execution plan, an AND / OR operation execution plan, and a warehousing and compression storage execution plan.

[0013] The performing of operations and compression storage on the tags based on the three-stage execution plan and the operation rules by the data end includes:

[0014] Based on the filtering tag execution plan, the tags are filtered in a bitmap manner to obtain an object set;

[0015] Splitting the object set based on the AND / OR operation execution plan to obtain a plurality of object bitmaps, and performing AND / OR operations on the plurality of object bitmaps to obtain operation results;

[0016] Based on the warehousing compression storage execution plan, each object bitmap and the label will be compressed and stored.

[0017] The execution plan based on the filtering tag uses a bitmap method to filter the tag to obtain an object set, including:

[0018] Searching for a target tag from the tags using the metadata based on the screening tag execution plan;

[0019] Compressing and storing the target tag as a bitmap;

[0020] The bitmap is operated based on the conditions met to obtain an object set.

[0021] The bitmap is an expanded roar bitmap.

[0022] The unsigned integer range of the bitmap is 64 bits, 128 bits and 256 bits.

[0023] The present invention provides a high-performance, high-compression tag calculation method. Labels and calculation rules are defined through a user interface (UI); the labels and calculation rules are parsed into a three-stage execution plan through an application; metadata is stored in the form of a closure table through a metadata end; and the labels are individually calculated and compressed and stored based on the three-stage execution plan and the calculation rules using a bitmap method through a data end. By optimizing the logical results of the labels using a bitmap method, the present invention allows for direct calculation and compressed storage of the labels, resolving the issue of memory waste when labels are sparse. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 It is the organization method of the data side, metadata side, application side and UI side.

[0026] Figure 2 It is a traditional bitmap diagram.

[0027] Figure 3 It is a schematic diagram of the bitmap method of the present invention.

[0028] Figure 4 This is a schematic diagram of how the roaring bitmap is implemented.

[0029] Figure 5 It is a diagram in the form of a closure table.

[0030] Figure 6 This is a flowchart of a high-performance and high-compression-ratio label calculation method provided by the present invention. DETAILED DESCRIPTION

[0031] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0032] See also Figures 1 to 6 The present invention provides a high-performance and high-compression-ratio label calculation method, comprising the following steps:

[0033] S1 defines tags and operation rules through the UI;

[0034] Specifically, the labels include basic labels, overlay labels and removal labels.

[0035] The operation rule is each logical OR and multiple logical ANDs in each unit.

[0036] S2 parses the label and the operation rule into a three-stage execution plan through the application end;

[0037] Specifically, the three execution plans include a label screening execution plan, an AND / OR operation execution plan, and a warehousing and compression storage execution plan.

[0038] The tags and operation rules defined on the UI are parsed into a three-step execution plan: the first step filters tags, the second step performs AND, OR, and NOT operations on the tags, and the third step stores the data in the database. All three steps are executed on the data side. The application-side execution units are expressed in Structured Query Language (SQL), which is parsed and executed on the data side. Using SQL reduces development effort for developers and separates business operations from underlying implementation.

[0039] S3 stores the metadata of the tag in the form of a closure table through the metadata terminal;

[0040] Specifically, the form of closure table is adopted ( Figure 5 ) Compressed storage of tag metadata facilitates quick lookup of corresponding tags and their subtags. The closure table's compressed storage solution is a typical space-for-time solution. Compressed storage of the full path structure of a tree structure allows for clear relationships between any two nodes without redundant queries, and also facilitates cascading deletions and node movement.

[0041] S4 uses a bitmap method on the data end to respectively calculate and compress and store the labels based on the three-stage execution plan and the operation rules.

[0042] The specific method is:

[0043] S41 filters the tags in a bitmap manner based on the tag filtering execution plan to obtain an object set;

[0044] Specifically, based on the filtered tag execution plan, the metadata is used to search for a target tag from the tags; the target tag is compressed and stored as a bitmap; and the bitmap is operated on based on the conditions met to obtain an object set. The bitmap is an expanded roaring bitmap. The unsigned integer range of the bitmap is 64 bits, 128 bits, and 256 bits.

[0045] S42: dividing the object set based on the AND / OR operation execution plan to obtain a plurality of object bitmaps, performing AND / OR operations on the plurality of object bitmaps to obtain operation results;

[0046] S43 compresses and stores each of the object bitmaps and the labels based on the warehousing compression storage execution plan.

[0047] After the step of compressing and storing each of the object bitmaps and the labels based on the warehousing compression storage execution plan, the method further includes:

[0048] Access rights are set for storing the object bitmap and the tag based on access information of the access object.

[0049] Specifically, access to the object needs to be verified using the corresponding access information, and only after the verification is passed can the stored object bitmap and the tag be accessed, thereby increasing data security.

[0050] Specifically, the bitmap method is used for compression storage and calculation. The present invention optimizes the logical structure of the label. The traditional bitmap method uses Figure 2 The structure is compressed and stored, and the present invention uses Figure 3When filtering objects that meet the conditions, Figure 2 The method still needs to perform label calculations on all users separately, and Figure 3 The tag can be directly operated to obtain the set of objects that meet the conditions. The implementation of the bitmap is based on the roaring bitmap and is extended to support unsigned integer ranges of 64 bits, 128 bits, and 256 bits. The implementation method is as follows Figure 4 , the bitmap is divided into multiple bitmaps for calculation and compression storage. The roaring bitmap itself has an ultra-high compression rate and calculation efficiency. Based on the underlying implementation of the roaring bitmap, the present invention also has the advantages of the roaring bitmap.

[0051] The present invention provides a high-performance and high-compression-ratio label operation method. First, the basic label, superimposed label, and eliminated label and operation logical OR and logical AND rules are defined through the UI end; then, the label and the operation rules are parsed into a filter label execution plan, an AND / NOT operation execution plan, and a storage compression and storage execution plan through the application end; then, the metadata of the label is stored in the form of a closure table through the metadata end; finally, the target label is searched from the label using the metadata based on the filter label execution plan through the data end; the target label is compressed and stored as a bitmap; the bitmap is operated based on the conditions to obtain an object set. The object set is divided based on the AND / NOT operation execution plan to obtain multiple object bitmaps, and AND / NOT operations are performed on the multiple object bitmaps to obtain operation results. Based on the storage compression and storage execution plan, each object bitmap and the label are compressed and stored. The present invention optimizes the logical results of the label by adopting a bitmap method, and can directly operate and compress the label for storage, thereby solving the problem of wasting memory when the label is sparse.

[0052] What is disclosed above is only a preferred embodiment of a high-performance and high-compression ratio label calculation method of the present invention. Of course, this cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A high-performance and high-compression label calculation method, characterized in that: The following steps are involved: Define tags and operation rules through the UI; Parsing the label and the operation rule into a three-stage execution plan through the application end; Storing metadata of the tag in the form of a closure table through a metadata end; The data end uses a bitmap method to calculate and compress the labels based on the three-stage execution plan and the operation rules; The performing calculations and compression storage on the tags based on the three-stage execution plan and the calculation rules by the data end includes: Based on the filtering tag execution plan, the tags are filtered using a bitmap method to obtain an object set; Splitting the object set based on an AND / OR operation execution plan to obtain a plurality of object bitmaps, and performing AND / OR operations on the plurality of object bitmaps to obtain operation results; Based on the warehousing compression storage execution plan, each object bitmap and the label will be compressed and stored.

2. The high-performance and high-compression-ratio label calculation method according to claim 1, characterized in that: The metadata includes the name, identifier and tree structure of the tag; The labels include basic labels, overlay labels and rejection labels.

3. The high-performance and high-compression label calculation method according to claim 1, characterized in that: The execution plan based on the filtering tag uses a bitmap method to filter the tag to obtain an object set, including: Searching for a target tag from the tags using the metadata based on the screening tag execution plan; Compressing and storing the target tag as a bitmap; The bitmap is operated based on the conditions met to obtain an object set.

4. The high-performance and high-compression-ratio label calculation method according to claim 3, wherein: The bitmap is an expanded roaring bitmap.

5. The high-performance and high-compression-ratio label calculation method according to claim 4, characterized in that: The unsigned integer range of the bitmap is 64 bits, 128 bits, and 256 bits.

Images