In-memory and on-disk compression and tuple update / insertion method for multi-dimensional databases

Abstract

The application discloses a method for internal and external memory compression and tuple update insertion suitable for a multidimensional database, and relates to the technical field of multidimensional databases.The method comprises the following steps: based on an extended compressed row storage xCRS algorithm, all tuples of a multidimensional database are compressed; based on the compression algorithm, the corresponding disk block number, in-block index and offset of the tuples after compression in the underlying block storage of the multidimensional database are calculated, and the query of the tuples after compression is completed; based on the disk block number and in-block index, the update of the tuples is completed; when processing an insertion request, the precursor node of the tuple to be inserted in the compressed structure is first queried, and based on the precursor node of the tuple obtained by the query, a new tuple is inserted in an incremental buffer area; the method designs the xCRS algorithm suitable for the range of compressed row storage from a static multidimensional matrix to a multidimensional database supporting addition, deletion, modification and query, the structure of the incremental buffer area and related algorithms, which are used for processing the problem of incremental data in the multidimensional compressed row storage xCRS table, and can be used for engineering practice.

Description

Technical Field

[0001] This invention relates to the field of multidimensional database technology. Background Technology

[0002] Relational database theory has played a driving role in the development of the database-related information technology field. However, in today's era of rapid technological advancements, the limitations of relational databases are gradually becoming apparent with the large-scale application of information technology. For the vast amounts of multidimensional data existing in the real physical world, it is difficult for relational databases based on relational schemas to establish a direct model. This significantly impacts the design of multidimensional data models and subsequent optimization of multidimensional data queries, leading to considerable limitations when handling relatively complex multidimensional transactions.

[0003] Compared to traditional relational databases, multidimensional databases use data models that can concisely and intuitively describe transactions in the real physical world without having to deconstruct them and abstract them into technically feasible models. Because data in multidimensional databases is stored in array form, it can update data without indexes. Therefore, multidimensional databases are very suitable for applications such as storage, querying, and analysis of large-scale, multidimensional data.

[0004] A multidimensional database can be considered a database specifically adapted for the needs of online analytical processing (OLAP) programs for multidimensional data. It constructs a "data cube" by combining all the two-dimensional tables in a relational database, mapping several rows of data from multiple tables that have relationships to a point in a multidimensional space. Theoretically, because multidimensional databases reduce the redundant data attached to relational tables, they theoretically have a more efficient storage structure than relational databases under the same data storage requirements. They also have better query processing performance for data retrieval. At the same time, data scientists can observe data through a more intuitive multidimensional view and then perform various analysis and mining operations.

[0005] Multidimensional databases outperform relational databases in areas such as data modeling, data analysis, and mining potential information. However, like traditional relational databases, they also face various problems that need to be solved in the database domain. For example, how can database management systems more effectively store and manage massive amounts of data within limited main and secondary storage? Due to the characteristics of real-world business data, multidimensional databases often contain a large number of sparse matrices, which can lead to wasted space and unnecessary I / O costs.

[0006] Traditional methods for storing massive amounts of data involve using parallel environments and distributed processing systems for distributed processing, or loading massive amounts of data into storage space and then performing calculations based on big data processing algorithms or computing frameworks. While this approach can improve efficiency, it consumes a large amount of computing resources. Distributed processing systems inevitably experience communication latency; furthermore, as data volume increases, hardware resources also need to be increased accordingly. Therefore, simply consuming resources to handle massive data is not a sustainable solution. Traditional compressed row storage (CRS) for compressing sparse data in multidimensional data is a good approach, but it has poor support for the large-scale CRUD operations required by multidimensional databases.

[0007] Therefore, how to provide an extended method suitable for multidimensional data compression, thereby improving storage utilization and compressing the space occupied by multidimensional data, while also improving the speed of querying or updating multidimensional data, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a method for compressing internal and external memory and updating and inserting tuples in multidimensional databases. This method designs the xCRS algorithm, the structure of the incremental buffer, and related algorithms to extend the applicability of compressed row storage from static multidimensional matrices to multidimensional databases that support addition, deletion, modification, and querying. It is used to handle the problem of incremental data in multidimensional compressed row storage xCRS tables and can also be used in engineering practice.

[0009] Based on the same inventive concept, this invention has four independent technical solutions: 1. A method for compressing internal and external memory and updating and inserting tuples in a multidimensional database, comprising the following steps: S1. Based on the extended compressed row storage xCRS algorithm, compress all tuples in the multidimensional database; S2. Calculate the disk block number, intra-block index and offset of the compressed tuple in the underlying block storage of the multi-dimensional database, and complete the query of the compressed tuple. S3. Based on the disk block number and the intra-block index, update the tuple; S4. Obtain the predecessor node tuple of the tuple to be inserted, query the predecessor node tuple based on the method in step S2, and then insert the tuple to be inserted into the incremental buffer based on the predecessor node tuple.

[0010] Further, step S1 includes: S11. Create compression control information, including: the total number of data tuples after compression, the offset of rptr in the disk block, the offset of cind in the disk block, and the offset of val in the disk block. S12. Use a loop to traverse the tuples in the iterator and save the compression information. The traversal process includes: Based on the pre-defined column dimensions in the tuple The member ID in the tuple is used as the cind value of the tuple and stored in the corresponding location of the disk block that records the cind information; The value val of the data on the tuple is used as its data and stored in the corresponding position within the corresponding disk block in the disk block range that records the val information; Iterate through and remove the pre-selected column dimensions. The statistical remaining dimension is used to calculate the row number of the row where the tuple is located, and the row number is updated.

[0011] Further, step S2 includes: First, initialize the disk block number and intra-block index to 0; Retrieve the IDs of each member in the queried tuple; according to - The information calculation tuple of the dimension corresponds to the row index in the xCRS table constructed in step S1, based on... The information calculation tuple of the dimension is the column index corresponding to the column in the xCRS table; The starting offset of the corresponding row is obtained from the xCRS table based on the row index; Identify existing tuples and iterate through them until you find a record in the xCRS table where the cind value of the input tuple is equal to the cind value of the cind value in the xCRS table. The value in the val table corresponding to this record is the value of the tuple you are looking for. The disk block number and the intra-block index are determined based on the traversal variables from the previous step. The value of the disk block number is obtained by dividing the traversal variable by the maximum amount of data loaded in each data block, and the intra-block index is obtained by taking the remainder after dividing the traversal variable by the maximum amount of data loaded in each data block. The offset is calculated based on the intra-block index, and the base offset of the corresponding segment within the block.

[0012] Furthermore, the determined tuples include: Get the tuple containing the positions where startOffset is not equal to -1, and use it as the starting position; Traverse the tuples based on the row index until the next startOffset value is not equal to -1 is found as the termination position, and determine the range of existing tuples based on the start position and the termination position.

[0013] Further, step S3 includes: Update the data val value at the corresponding location based on the disk block number and the intra-block index.

[0014] Further, step S4 includes: Upon receiving an insertion request, query the corresponding predecessor node tuple based on the information of the tuple to be inserted. Calculate the disk block number and intra-block index corresponding to the preceding node tuple; Create a new incremental buffer unit, set the row index and column offset according to the preceding node tuple, and initialize the address of the next incremental buffer unit to null; The incremental buffer unit is inserted into the incremental buffer of the multidimensional database.

[0015] Furthermore, the update of the tuple also includes: setting the val value at the corresponding position to null based on the disk block number and the intra-block index, thereby deleting the tuple.

[0016] 2. A device for compressing internal and external memory and updating and inserting tuples in a multidimensional database, comprising: The compression module is used to compress all tuples in a multidimensional database based on the extended compressed row storage (xCRS) algorithm. The query module is used to calculate the disk block number, intra-block index and offset corresponding to the tuple in the underlying block storage of the multi-dimensional database, and to complete the query of the compressed tuple; The update module is used to update the tuple based on the disk block number and the intra-block index; The insertion module is used to insert a new tuple by taking the tuple retrieved from the query module as the preceding node.

[0017] 3. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0018] 4. An electronic device, comprising a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is configured to read the plurality of instructions from the storage device and execute the method described above.

[0019] The method for compressing internal and external memory and updating and inserting tuples in multidimensional databases provided by this invention has at least the following beneficial effects: (1) An improved extended compressed row storage algorithm was proposed, and a compressed row storage algorithm based on linked lists and buffers was proposed to meet the needs of multidimensional databases. An internal and external storage compression module suitable for commercial multidimensional database system GmmCube was designed. (2) This method can save storage space. Compared with the original data storage module based on multidimensional matrix, it can save 90% of physical storage space on the test dataset. At the same time, when accessing data, the module improves the efficiency of querying and retrieving data through locality optimization. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a multidimensional matrix in space; Figure 2 The result of multidimensional matrix compression is shown in the figure. Figure 3 This is a diagram illustrating the addition of a nextptr row to the xCRS table. Figure 4 Schematic diagram of the design for an incremental buffer for expanding compressed line storage; Figure 5 A flowchart of one embodiment of the method for compressing internal and external memory and updating and inserting tuples in a multidimensional database provided by the present invention; Figure 6 This is a schematic diagram illustrating the steps of a method for compressing all tuples in a multidimensional database. Figure 7 A schematic diagram illustrating the steps to complete a query on the compressed tuple; Figure 8 A diagram illustrating the steps for inserting new tuple data; Figure 9 A schematic diagram of the design for incremental caching of multidimensional data compression. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0024] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0027] The embodiments of the present invention are based on existing compression technologies and are extended. The technical basis of the present invention is described below.

[0028] 1. Line compression format and extended compressed line storage method Compressed Row Storage (CRS) is a widely used data compression algorithm that maps points in a two-dimensional space to three one-dimensional spaces (which can also be understood as three one-dimensional arrays). These three one-dimensional spaces are: the value of the element, the column number of the row containing the element, and the offset of each data row. CRS is a global encoding format. The value rows and column number rows record the values ​​of non-zero elements and their column indices, respectively, while the row offset indicates the position of the first element of a given row within the value array.

[0029] The main methods for implementing multidimensional sparse matrix compression on multidimensional databases include the extended compressed row storage method, which improves upon the row compression storage format for multidimensional data. Specifically, it extends the applicable dimensions of the compressed row storage method from two-dimensional rows and columns to a K-dimensional space (k>2). The technical solution is as follows: To use the CRS method in multidimensional space, a mapping is needed to transform the k-dimensional array into the row and column form of a compressed row storage method. Specifically, this involves mapping the original k-dimensional space to a format containing rows and columns. OK( Let be the two-dimensional form of the cardinality of the set (where d is the j-th dimension); under this mapping, each row element of the original multidimensional space can be transformed into a unique key, which is obtained by the row offsets and the dimension. The index values ​​are jointly determined; for row offsets, it uses k-1 index values, for example... To represent; and for each element in the row, further can be used... The calculation is performed in the form of a dimension mapping; therefore, a unique key value can be calculated for each row element through this dimensional mapping.

[0030] 2. Implementation of extended compressed row storage data structure Based on the above, we can deduce the storage method of extended compressed row storage for non-zero elements in multidimensional data: for the data records themselves and their corresponding dimensions in the original multidimensional matrix. The index is stored in two one-dimensional arrays: `val` and `cind`. In addition, a third one-dimensional array, `rptr`, is used to store the starting position of each row in `cind`. If a row in xCRS has no valid element, its starting position is marked with the default value "-1". In xCRS, each row has an entry in `rptr`, whose position corresponds to the row offset. Therefore, we can define an xCRS table as a data table composed of three one-dimensional arrays: `val`, `cind`, and `rptr`. Figure 1 and Figure 2 Here is a case study of xCRS storage.

[0031] Extended compressed row storage is simple and efficient on multidimensional arrays, but this storage format is more suitable for storing static multidimensional data. Due to the frequent updates of multidimensional databases, we need to modify it to adapt to the needs of adding, deleting, and modifying data.

[0032] For extended row storage format, if it is necessary to delete or modify the compressed data, it is only necessary to first calculate the position of the value in the value array based on the compressed offset, and then modify it according to the index from the corresponding position of the non-zero data in the compressed data. If it is a deletion, set the record at that position to null; if it is a modification, simply modify the value at that position to the new value to complete the modification and deletion operation. The following introduces the principles and theoretical basis on which the method of the embodiments of the present invention is based.

[0033] 1. Expanded support for additional operations in compressed line storage. Before discussing the addition operation of extended compressed row storage insertion, it is necessary to propose the concept of a predecessor node in multidimensional space, defined as: Points in 3D space It is a point The preceding node of a is determined if and only if point a is a node that satisfies the condition that point a is a node that ... and The largest point.

[0034] in, It is a one-dimensional array that stores the multiplier coefficients of multi-dimensional rows, satisfying... ,and Dimensions representing multidimensional space All elements in the set constitute the cardinality of the set.

[0035] This paper proposes an extended compressed row storage increment support based on linked lists. In addition to the original table consisting of three compressed row arrays, a pointer row is added to store the "overflow" increment, recorded as the nextptr row. Its structure is as follows: Figure 3 As shown.

[0036] The incremental buffer's design is inspired by the hash bucket algorithm, but differs in that each preceding node corresponds to a linked list, on which newly added data is stored. This linked list data structure allows data to be accessed by node. Sort the elements in order, select a suitable position in the incremental buffer, and then perform a linked list insertion operation with a time complexity of O(1). Its structure is as follows: Figure 4 As shown.

[0037] In this way, whenever a request to insert a new record is received, for Let the tuple of the newly inserted record in the 1-dimensional space be . , The first step is to find its predecessor node in the xCRS table. , Then, obtain the insertion buffer address corresponding to this record from the nextptr line of that record. At the head of the linked list corresponding to that address, a check is performed. If it is empty, the inserted data is directly recorded in the record, and the row of that record is set to... cind is set to ,val can be set to the value of the newly inserted record. You can set nextptr to empty; If the head of the linked list is not empty, then begin the comparison. The value is until it can be found in the linked list. , The position satisfies If and only if there exists an inequality Established; At this point, the node in the linked list , Simply insert a new record between them.

[0038] Example 1: See Figure 5 In some embodiments, a method for updating and inserting tuples with compressed internal and external memory for multidimensional databases is provided, including the following steps: S1. Based on the extended compressed row storage xCRS algorithm, compress all tuples in the multidimensional database; S2. Calculate the disk block number, intra-block index and offset of the compressed tuple in the underlying block storage of the multidimensional database, and complete the query of the compressed tuple. S3. Based on the disk block number and the intra-block index, update the tuple; S4. Obtain the predecessor node tuple of the tuple to be inserted, query the predecessor node tuple based on the method in step S2, and then insert the tuple to be inserted into the incremental buffer based on the predecessor node tuple.

[0039] Specifically, in step S1, the input is an iterator containing a multi-dimensional data tuple, and the maximum number of blocks that can be loaded per disk block, maxNumberOfValuePerBlock.

[0040] The output is an xCRS table containing three rows: val, cind, and rptr. The val value, also called the value, stores the data record itself; the cind value stores the corresponding dimension of the data in the original multidimensional matrix. The index; the rptr value is used to store the starting position of each row in the cind value. That is, the numerical value corresponding to the data record itself is stored as the val column in the compressed xCRS table, indicating the location of the data record. The position of a dimension in the compressed xCRS table is stored as a cind column, which is collapsed into the dimension of the row. - The information is stored in the rptr row of the xCRS table.

[0041] Step S1 specifically includes the following steps: S11. Create compression control information, including: the total number of data tuples after compression, the offset of rptr in the disk block, the offset of cind in the disk block, and the offset of val in the disk block. S12. Use a loop to traverse the tuples in the iterator and save the compression information. The traversal process includes: Based on the pre-defined column dimensions in the tuple The member ID in the tuple is used as the cind value of the tuple and stored in the corresponding location of the disk block that records the cind information; The value val of the data on the tuple is used as its data and stored in the corresponding position within the corresponding disk block in the disk block range that records the val information; Iterate through and remove the pre-selected column dimensions. The statistical remaining dimension is used to calculate the row number of the row where the tuple is located, and the row number is updated.

[0042] For detailed steps, please refer to [link / reference]. Figure 6 Pseudocode demonstrating the construction of an xCRS table from multidimensional data: Step 1: Create compression control information CompressCtrlData, which records the total number of compressed data tuples, the position of the rptr table in the disk block (in the form of offset), the offset of the cind table in the disk block, and the offset of the val table in the disk block.

[0043] Step 2: Use a loop to iterate through the tuples in the iterator: a. Based on the pre-defined column dimensions in the tuple. The member ID (no) of the tuple is used as the cind value of the tuple and will be stored in the location corresponding to the disk block that records the cind information.

[0044] b. Based on the value of the data in the tuple, the data will be stored in the corresponding position within the corresponding disk block of the record value.

[0045] c. Iterate through and count the remaining dimensions (excluding the pre-selected column dimensions). ), calculate the row number rowIndex of the row containing the tuple, and update it accordingly.

[0046] Step 3: Save compression-related information (CompressCtrlData) Step 4: Initialize the compressed expansion buffer for subsequent insertion.

[0047] Preferably, the algorithm in step S2 is used to process tuple access in the multidimensional database storage engine. Specifically, it calculates the block number, cell index, offset, and other information corresponding to the multidimensional data tuple in the underlying block storage of the multidimensional database based on the information contained within the tuple. See also... Figure 7 It includes the following steps: S21. First, initialize the disk block number blockNumber and the intra-block index cellIndex to 0; S22. Obtain the ID memnos of each member in the tuple; S23, according to - The information calculation tuple of the dimension is based on the row index tupleRowIndex corresponding to the xCRS table constructed in step S1. The information calculation tuple of the dimension is the column index tupleCind corresponding to the tuple in the xCRS table; S24. Based on the row index tupleRowIndex, retrieve the starting offset startIndex of the corresponding row from the xCRS table; S25. Determine the existing tuples and traverse them until you find a record in the xCRS table where the cind value of the input tuple is equal to the cind value of the tuple. The value in the value table corresponding to this record is the value of the tuple you are looking for. S26. Determine the disk block number and the intra-block index based on the traversal variable from the previous step. The value of the disk block number is obtained by dividing the traversal variable index by the maximum amount of data loaded in each data block. The intra-block index is obtained by taking the remainder after dividing the traversal variable index by the maximum amount of data loaded in each data block. S27. Calculate the offset based on the intra-block index, and the base offset of the corresponding segment within the block. A data block contains multiple data segments.

[0048] In the xCRS table, each row has a unique row number, called the RowIndex. This RowIndex is calculated based on the dimension corresponding to that row. - The information of the member (a concept abstracted by multidimensional databases for business convenience, which can be understood here as the value of a multidimensional data tuple in this dimension) is calculated. Other parameters involved are shown in Table 1.

[0049] Table 1

[0050] Preferably, determining the existence of tuples includes: Get the tuple containing the positions where startOffset is not equal to -1, and use it as the starting position; Traverse the tuples based on the row index until the next startOffset value is not equal to -1 is found as the termination position, and determine the range of existing tuples based on the start position and the termination position.

[0051] It should be noted that the startOffset value is a preset value. If the startOffset value is not equal to -1, it means that the row to which the compressed tuple belongs exists; otherwise, it means that the row to which the tuple belongs does not exist, and the tuple itself does not exist.

[0052] Preferably, step S4 includes: S41. After receiving the insertion request, query the predecessor node tuple of the tuple to be inserted in the compressed structure. S42. Calculate the disk block number and intra-block index corresponding to the preceding node tuple; S43. Create a new incremental buffer unit, and set the row index and column offset according to the preceding node tuple, and initialize the address of the next incremental buffer unit to null. S45. Insert the incremental buffer unit into the incremental buffer of the multidimensional database.

[0053] The multidimensional data compression incremental cache module primarily implements incremental support for extended compressed row storage. Its main structure is a dynamically expandable buffer, `CompressExtendBuffer`. Whenever the data block system storage engine receives an insertion request, it first queries the position of the preceding node and its address in the insertion buffer based on the information of the tuple to be inserted. Then, it inserts the record at the corresponding position in the buffer. Whenever the data block query engine needs to retrieve the value of a newly inserted data record, it accesses the corresponding value by traversing the linked list in the buffer. The pseudocode flow of the insertion process is roughly as follows: Figure 8 The provided algorithm is shown. See also... Figure 9This method adds a pointer row to the existing compressed table consisting of three row arrays to store the "overflow" increment, recorded as the nextptr row. New data is inserted into the increment buffer. Whenever the database system storage engine receives an insert request, the algorithm performs the following operations: 1. First, based on the information of the inserted tuple, query its corresponding predecessor node tuple, Tupleforward; 2. Then calculate the disk block number (blockNumber) and intra-block index (cellIndex) of the preceding node tuple (Tupleforward); 3. Then create a new xCRSUnit incremental buffer unit and set its properties: rowIndex (the row index mentioned earlier), cind ( The column offset in the dimension), the address of the next incremental buffer unit nextptr (initialized to null); 4. Finally, insert it into the incremental buffer of the multidimensional database, waiting for the right time to merge it into the main data area.

[0054] If too much new data is inserted, the multidimensional data block data engine may experience a decrease in query performance due to excessively long linked lists. Therefore, it is necessary to periodically merge the incremental buffer with the main data area. The merging can be initiated by the user or detected by the management module when specific conditions are met. When the incremental buffer is merged into the main data table, the storage engine will re-read the multidimensional data compressed xCSR table and the incremental buffer into memory, and then recalculate the new xCSR table.

[0055] As a preferred implementation method, step S3 includes: Update the data val value at the corresponding location based on the disk block number and the intra-block index.

[0056] As a preferred implementation, updating the tuple further includes: setting the corresponding `val` value to null based on the disk block number and intra-block index, thereby deleting the tuple. If it is necessary to delete or modify the compressed data, it is only necessary to first calculate the position of this value in the `value` array based on the compressed offset, and then, referring to step S2, modify the value from the corresponding position of the non-zero data in the compressed array based on the index. If it is a deletion, the record at that location is set to null; if it is a modification, the value at that location is simply modified to the new value to complete the modification or deletion operation.

[0057] Specifically, based on calcIndex(tuple) in Algorithm 5-7 corresponding to step S2, calculate the position (block number blockNumber and intra-block index cellIndex) of the tuple to be updated / deleted in the value table in physical storage; update the data val corresponding to this position (block number blockNumber and intra-block index cellIndex).

[0058] Example 2: In some embodiments, a tuple update and insertion apparatus for multidimensional databases with internal and external memory compression is provided, comprising: The compression module is used to compress all tuples in a multidimensional database based on the extended compressed row storage (xCRS) algorithm. The query module is used to calculate the disk block number, intra-block index and offset corresponding to the tuple in the underlying block storage of the multi-dimensional database, and to complete the query of the compressed tuple; The update module is used to update the tuple based on the disk block number and the intra-block index; The insertion module is used to insert a new tuple by taking the tuple retrieved from the query module as the preceding node.

[0059] Example 3: In some embodiments, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0060] Example 4: In some embodiments, an electronic device is provided, including a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is configured to read the plurality of instructions from the storage device and execute the method described above.

[0061] It should be understood that, in the embodiments of this application, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0062] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0063] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0064] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0065] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0066] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of the modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0067] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for compressing internal and external memory and updating and inserting tuples in a multidimensional database, characterized in that, Includes the following steps: S1. Based on the extended compressed row storage xCRS algorithm, compress all tuples in the multidimensional database; S2. Calculate the disk block number, intra-block index and offset of the compressed tuple in the underlying block storage of the multidimensional database, and complete the query of the compressed tuple. S3. Based on the disk block number and the intra-block index, update the tuple; S4. Obtain the predecessor node tuple of the tuple to be inserted, query the predecessor node tuple based on the method in step S2, and then insert the tuple to be inserted into the incremental buffer based on the predecessor node tuple. Step S2 includes: First, initialize the disk block number and intra-block index to 0; Retrieve the IDs of each member in the queried tuple; according to - The information calculation tuple of the dimension corresponds to the row index in the xCRS table constructed in step S1, based on... The information calculation tuple of the dimension is the column index corresponding to the column in the xCRS table; The starting offset of the corresponding row is obtained from the xCRS table based on the row index; Identify existing tuples and iterate through them until you find a record in the xCRS table where the cind value of the input tuple is equal to the cind value of the cind value in the xCRS table. The value in the val table corresponding to this record is the value of the tuple you are looking for. The disk block number and intra-block index are determined based on the traversal variables from the previous step. The value of the disk block number is obtained by dividing the traversal variable by the maximum amount of data loaded in each data block, and the intra-block index is obtained by taking the remainder after dividing the traversal variable by the maximum amount of data loaded in each data block. The offset is calculated based on the intra-block index, and the base offset of the corresponding segment in the block is also calculated based on the offset. Step S4 includes: Upon receiving an insertion request, query the corresponding predecessor node tuple based on the information of the tuple to be inserted. Calculate the disk block number and intra-block index corresponding to the preceding node tuple; Create a new incremental buffer unit, set the row index and column offset according to the preceding node tuple, and initialize the address of the next incremental buffer unit to null; Insert the incremental buffer unit into the incremental buffer of the multidimensional database; The predecessor node is defined as: Points in 3D space It is a point The preceding node of a is determined if and only if point a is a node that satisfies the condition that point a is a node that ... and The largest point; in, It is a one-dimensional array that stores the multiplier coefficients of multi-dimensional rows, satisfying... ,and Dimensions representing multidimensional space All elements in the set constitute the cardinality of the set.

2. The method according to claim 1, characterized in that, Step S1 includes: S11. Create compression control information, including: the total number of data tuples after compression, the offset of rptr in the disk block, the offset of cind in the disk block, and the offset of val in the disk block. S12. Use a loop to traverse the tuples in the iterator and save the compression information. The traversal process includes: Based on the pre-defined column dimensions in the tuple The member ID in the tuple is used as the cind value of the tuple and stored in the corresponding location of the disk block that records the cind information; The value val of the data on the tuple is used as its data and stored in the corresponding position within the corresponding disk block in the disk block range that records the val information; Iterate through and remove the pre-selected column dimensions. The statistical remaining dimension is used to calculate the row number of the row where the tuple is located, and the row number is updated.

3. The method according to claim 1, characterized in that, The tuples that are determined to exist include: Get the tuple containing the positions where startOffset is not equal to -1, and use it as the starting position; Traverse the tuples based on the row index until the next startOffset value is not equal to -1 is found as the termination position, and determine the range of existing tuples based on the start position and the termination position.

4. The method according to claim 1, characterized in that, Step S3 includes: Update the data val value at the corresponding location based on the disk block number and the intra-block index.

5. The method according to claim 1, characterized in that, The update of the tuple further includes: setting the val value at the corresponding position to null based on the disk block number and the intra-block index, thereby deleting the tuple.

6. A device for compressing internal and external memory and updating and inserting tuples in a multidimensional database, characterized in that, The apparatus uses a method for compressing internal and external memory and updating and inserting tuples in a multidimensional database as described in any one of claims 1-5. The apparatus includes: The compression module is used to compress all tuples in a multidimensional database based on the extended compressed row storage (xCRS) algorithm. The query module is used to calculate the disk block number, intra-block index and offset corresponding to the tuple in the underlying block storage of the multidimensional database, and to complete the query of the compressed tuple; The update module is used to update the tuple based on the disk block number and the intra-block index; The insertion module is used to insert a new tuple by taking the tuple retrieved from the query module as the preceding node.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

8. An electronic device comprising a processor and a storage device, characterized in that, The storage device contains a plurality of instructions, and the processor is used to read the plurality of instructions in the storage device and execute the method as described in any one of claims 1-5.

Images