Data retrieval method and system and related products

Abstract

The embodiment of the invention discloses a data retrieval method and system and a related product, compared with an index tree structure in the prior art, each group of bit vectors constructed in the embodiment of the invention can store and correspondingly represent cell data belonging to the same sub-dimension in a current data table in a compact mode, and the construction cost of a retrieval path is reduced. Besides, for complex cell data (namely, complex dimension combination) simultaneously belonging to a plurality of sub-dimensions, the positions of the cell data stored in the corresponding bit vectors are designed to be the same, and intersection calculation is performed on the bit vectors, so that whether the cell data exist in a current data table or not can be quickly determined, and the accuracy of the cell data stored in the current data table is improved. The problem that the linear retrieval efficiency is low due to the fact that the depth of a traditional index tree is large is solved, and therefore the retrieval efficiency of needed data is improved, and tabulation time is shortened.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular, to data retrieval methods, systems and related products. Background Art

[0002] As a common form of data arrangement, tables are widely used in various fields, including financial analysis, scientific research, and corporate management.

[0003] Taking financial statements as an example, in order to retrieve target data from fund plan statements, an index tree (or retrieval model) is constructed for the entire statement through the dimension combination to which the statement data belongs. Each dimension can be used as a node in the index tree, and the corresponding statement data is mounted under the leaf node of the corresponding dimension combination (such as Figure 1-1 As shown in the figure, when you want to retrieve report data of a certain dimension combination (such as retrieving account B with dimensions of organization B and currency B), you can traverse each node of the index tree according to its dimension combination, find the branch or path that matches its dimension combination, and then find the mounted report data to complete the retrieval.

[0004] It can be seen that when the dimension combination becomes more complex, the depth of the traditional index tree will deepen, and the following problems are likely to occur: in order to retrieve or locate the last-level leaf node and then find the mounted target data, a long search path and search time are required, and the search efficiency is not ideal. Summary of the invention

[0005] The embodiments of the present application provide a data retrieval method, system and related products for improving the retrieval efficiency and tabulation progress of table data.

[0006] A first aspect of an embodiment of the present application provides a data retrieval method, comprising:

[0007] Acquire multiple groups of bit vectors corresponding to and representing the current data table; wherein each group of the bit vectors corresponds to a sub-dimension as a table header in the current data table, and stores cell data belonging to the corresponding sub-dimension in the current data table; when the cell data belongs to multiple sub-dimensions, the positions where the cell data are stored in the corresponding bit vectors are the same;

[0008] For each cell to be filled in the target data table, a target bit vector is determined from the plurality of groups of bit vectors; the sub-dimension pointed to by the target bit vector is the same as the sub-dimension pointed to by the cell to be filled;

[0009] If the determined target bit vectors are multiple groups, the intersection calculation is performed on the cell data stored in the same position in the multiple groups of the target bit vectors. If there are intersection positions that store the same cell data, the cell data at the intersection position is taken from any of the target bit vectors and filled into the cell to be filled, so as to draw the target data table filled with specific data.

[0010] Optionally, the acquiring of a plurality of groups of bit vectors corresponding to and representing the current data table includes:

[0011] Based on the total amount of cell data to be stored in the current data table, determine the size and type of the bit vector pre-corresponding to each sub-dimension in the current data table to construct a bit vector of the data to be stored;

[0012] Each of the cell data is sequentially stored in the bit vector corresponding to the corresponding sub-dimension to which it belongs, so as to fill the bit vector corresponding to each of the sub-dimensions, and the position of the cell data in the bit vector is marked to indicate the presence of the cell data.

[0013] Optionally, the cell data belonging to the sub-dimension in the current data table is stored in the position ranked in the order in the bit vector according to the order in which it is stored in the corresponding bit vector.

[0014] Optionally, when the total amount is N, the bit vector is a long type array with a size of N / 64+1, or a byte array with a size of N / 8+1.

[0015] Optionally, the acquiring of a plurality of groups of bit vectors corresponding to and representing the current data table includes:

[0016] For each dimension and sub-dimension corresponding to each cell data in the current data table, the distribution position of the dimension in all dimensions is calculated and recorded, and the distribution position of the bit vector corresponding to the sub-dimension in the corresponding dimension is calculated and recorded; the distribution positions of the dimension and the bit vector are used to index the target bit vector.

[0017] Optionally, the distribution position of each dimension and each bit vector is recorded in a hash table.

[0018] Optionally, after performing intersection calculation on the cell data respectively stored at the same position in the plurality of groups of the target bit vectors, the method further comprises:

[0019] If the intersection position does not exist, it is determined that the required cell data corresponding to the to-be-filled cell does not exist in the current data table, and a null value reminder and / or a reminder to fill in the required cell data in the to-be-filled cell is initiated.

[0020] When the method described in the first aspect of the present application is implemented, the content described in the second aspect of the present application can be adopted to achieve it.

[0021] A second aspect of the embodiment of the present application provides a data retrieval system, including: an acquisition unit, a processing unit;

[0022] The acquisition unit is used to acquire multiple groups of bit vectors corresponding to the current data table; each group of the bit vectors corresponds to a sub-dimension as a table header in the current data table, and stores the cell data belonging to the corresponding sub-dimension in the current data table; when the cell data belongs to multiple sub-dimensions, the cell data is stored in the same position in the corresponding bit vector;

[0023] The processing unit is used to determine, for each cell to be filled in the target data table, a target bit vector from the plurality of groups of bit vectors; the sub-dimension pointed to by the target bit vector is the same as the sub-dimension pointed to by the cell to be filled;

[0024] The processing unit is also used to, if the determined target bit vectors are multiple groups, perform intersection calculation on the cell data stored in the same position in the multiple groups of target bit vectors; if there are intersection positions that store the same cell data, take out the cell data at the intersection position from any of the target bit vectors and fill it into the cell to be filled, so as to draw the target data table filled with specific data.

[0025] A third aspect of an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0026] The processor is configured to communicate with the memory and execute instructions in the memory to implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0027] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, they implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0028] A fifth aspect of an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, they implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0029] It can be seen from the above technical solutions that the embodiments of the present application have at least the following advantages:

[0030] Compared with the index tree structure in the prior art, each group of bit vectors constructed by the embodiment of the present application can store and correspond to the cell data belonging to the same sub-dimension in the current data table in a compact manner, simplifying the construction cost of the search path. In addition, for complex cell data belonging to multiple sub-dimensions at the same time (i.e., complex dimension combinations), the location where the cell data is stored in the corresponding bit vector is designed to be the same, and the intersection calculation of these bit vectors is performed, which helps to quickly determine whether the cell data exists in the current data table, avoiding the problem of low linear retrieval efficiency (i.e., the need to traverse the tree branches from beginning to end) caused by the deep depth of the traditional index tree, thereby speeding up the retrieval efficiency and tabulation time of the required data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0032] It should be noted that, although the steps in the process diagrams (if any) involved in the embodiments are drawn in sequence as indicated by the arrows, unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0033] Figure 1-1 This is a schematic diagram of a traditional index tree;

[0034] Figure 1-2 A schematic diagram of a system architecture of a data retrieval method according to an embodiment of the present application;

[0035] Figure 2 A schematic diagram of a flow chart of a data retrieval method according to an embodiment of the present application;

[0036] Figure 3 This is an index model diagram of the data retrieval method of the embodiment of the present application;

[0037] Figure 4 A schematic diagram of a target data table of a data retrieval method according to an embodiment of the present application;

[0038] Figure 5 A structural diagram of a data retrieval system according to an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0041] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] In the following description, similar expressions such as "a specific implementation" or "a specific example" are involved, which describe a subset of all possible embodiments, but it can be understood that "a specific implementation" or "a specific example" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. In the following description, the term multiple refers to at least two. In some specific examples, the numerical value mentioned in this application reaches the threshold value (if any), which may include the case where the former is greater than the latter of the threshold value; if similar expressions such as "any" or "at least one" are mentioned, it can specifically refer to any one of the listed examples or any combination of these examples.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] The method provided in the embodiment of the present application can be applied to Figure 1-2In the application environment shown, the terminal 102 communicates with the server 101 through the network, and the data storage system 100 can store the data that the server 101 needs to process or needs, such as the multiple sets of bit vectors corresponding to the current data table in the embodiment of the present application. The data storage system 100 can be integrated on the server 101, or it can be placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 101 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0045] The terminal 102 can obtain the target data table drawn by the user and send the target data table to the server 101. At this time, the target data can be understood as a table with only a header (the sub-dimension value and its corresponding dimension can be used as a header), but the cells under the header are empty (i.e., to be filled with specific data); of course, the target data table can also be drawn by the server 101. For each cell to be filled in the target data table, the server 101 can determine a target bit vector from multiple groups of bit vectors, and the sub-dimension pointed to by this target bit vector is the same as the sub-dimension pointed to by the cell to be filled; if the determined target bit vectors are multiple groups, the server 101 can perform intersection calculations on the cell data stored in the same position in the multiple groups of target bit vectors. If there are intersection positions that store the same cell data, the server 101 can take out the cell data at the intersection position from any target bit vector and fill it into the cell to be filled, so as to draw a target data table filled with specific data, that is, gradually export a complete target data table.

[0046] It should be noted that the method provided in the embodiment of the present application can be implemented jointly by the terminal device and the server as described above, or it can be implemented entirely on the server side, or it can be implemented entirely on the terminal device side. The specific method can be determined according to the actual application scenario and is not limited here.

[0047] The method of the present application will be further described in detail below, and some specific possible implementation examples will be provided. In actual applications, the implementation contents between these examples can be combined or implemented separately as needed according to the corresponding functional principles and application logic. If implemented in combination, the execution order between the combined examples can be determined according to their respective processing logics, which may be determined by the actual scenario.

[0048] See also Figure 2 In a first aspect, the present application provides a specific embodiment of a data retrieval method, which includes the following steps:

[0049] Step S21, obtaining multiple groups of bit vectors corresponding to and representing the current data table;

[0050] Each group of bit vectors corresponds to a sub-dimension as a header in the current data table, and stores the cell data belonging to the corresponding sub-dimension in the current data table; when the cell data belongs to multiple sub-dimensions, the cell data is stored in the same position in the corresponding bit vector.

[0051] For example, the sub-dimensions under the period dimension can be January 2024, February 2024, etc., or the first quarter of 2024, the second quarter of 2024, etc.; the sub-dimensions under the currency dimension can be RMB, USD, etc., the sub-dimensions under the company dimension can be Company A, Company B, etc., and the sub-dimensions under the account dimension can be income items of different themes such as income 1-4 and income 1-2. The cell data pointed to by the sub-dimension members composed of four sub-dimensions can be the specific amount values ​​that conform to these four sub-dimensions.

[0052] For ease of explanation and understanding, the current data table of the embodiment of the present application is exemplified by a capital plan report (or simply a report). The amount entered in the capital plan report (which can be understood as an amount such as a balance) is the cell data, and each amount corresponds to a unique set of dimension member combinations (or simply referred to as sub-dimension combinations) in a report, that is, each amount can correspond to multiple sub-dimensions, and this sub-dimension can be regarded as the corresponding associated header of this amount. In other words, it can also be understood that the sub-dimension combination is used to identify the uniqueness of the capital plan amount. When this amount corresponds to multiple sub-dimensions, this amount can essentially be considered to correspond to multiple dimensions. For example, the sub-dimension combination corresponding to the amount 1 (such as a value of 2000) recorded in the current report is "Company A + January 2024 + RMB + Income 1-2", which means: Company A's amount value for income 1-2 in January 2024 is 2000 yuan.

[0053] The above bit vector can be understood as a data structure such as a vector or array composed of multiple binary bits (i.e., bits). The bit vector can be regarded as a higher-level concept of a bit array; due to its compact storage method and efficient bit operations, the bit vector can be widely and flexibly used in scenarios where data needs to be stored and indexed. In some examples, the above bit vector can be called a bitmap index structure, which is used to record cell data and its corresponding sub-dimension, so that in subsequent retrieval scenarios, the cell data corresponding to this sub-dimension can be found through the sub-dimension recorded by it, thereby completing the retrieval of the required data.

[0054] Exemplarily, when the cell data belongs to multiple sub-dimensions, the cell data may be stored in the xth binary bit in the corresponding bit vector.

[0055] Step S22: for each cell to be filled in the target data table, determine a target bit vector from the multiple groups of bit vectors;

[0056] The sub-dimension pointed to by the target bit vector is the same as the sub-dimension pointed to by the cell to be filled in. The target data in step S22 can be understood as a null value table with only a table header (i.e., a row header or a column header), but the cell content under the table header is empty (i.e., specific data to be filled in).

[0057] Specifically, the target data table can use at least one dimension in the current data table and at least one sub-dimension under the dimension as a header. In other words, there may be an intersection in the header contents between the target data and the current data table, that is, the header contents of the two may be partially or completely the same, or, this intersection may be 0, that is, the header contents of the two may be completely different; of course, the filling positions of the header contents of the two in the table may be the same or different, which is not limited in the embodiments of the present application.

[0058] In order to draw a complete target data table filled with specific data, it is necessary to find the corresponding target data and fill it into the corresponding cells to be filled in the target data table: for each cell to be filled in the target data table (which can be regarded as an empty space at this time), check whether there is a specific corresponding target data pre-stored in the current data table. For the embodiment of the present application, it is to check whether the target data is stored in multiple groups of bit vectors. If so, take it out and put it in this cell to be filled. Repeat this process to draw a complete report filled with specific data one by one. Of course, the data search and filling operations can be completed simultaneously or one by one between multiple cells to be filled, that is, the order of executing steps S22 and S23 for multiple cells to be filled can be unlimited, and can also be executed simultaneously, which can be determined according to the specific situation.

[0059] The aforementioned target data means that the sub-dimension combination to which this data belongs must be consistent with the sub-dimension combination pointed to by the cell to be filled, because more or less any sub-dimension will most likely affect the corresponding data such as changes in numerical values; the cell data belonging to the same sub-dimension in the current data table is simplified and stored in the corresponding bit vector, so searching for the aforementioned target data can be understood as searching for the target bit vector that is the same as the sub-dimension pointed to by the cell to be filled.

[0060] Step S23: If the determined target bit vectors are multiple groups, perform intersection calculation on the cell data stored at the same position in the multiple groups of target bit vectors. If there are intersection positions that store the same cell data, take out the cell data at the intersection position from any target bit vector and fill it into the cell to be filled, so as to draw a target data table filled with specific data.

[0061] For example, if the sub-dimension combination 1 pointed to by the cell to be filled is "Company A + January 2024 + RMB + Income 1-2" (i.e., the number of corresponding sub-dimensions is 4), then the number of groups of the corresponding target bit vectors should also be consistent with the number of sub-dimensions, i.e., 4 groups of target bit vectors corresponding to the four sub-dimensions of Company A, January 2024, RMB, and Income 1-2 should be found, i.e., the sub-dimension pointed to by the target bit vector should be the same as the sub-dimension pointed to by the cell to be filled. Of course, the number of sub-dimensions pointed to by the cell to be filled may also be 1. For example, if it only corresponds to the sub-dimension of Company A, then the corresponding target bit vector found at this time is single, i.e., the group of bit vectors corresponding to the expression of Company A is found; specifically, the number of groups of the target bit vectors found can be determined by the drawing characteristics of the header of the data table, etc., but in general, the headers pointed to by the cells in the data table are two or more, so the number of groups of the target bit vectors found is generally multiple.

[0062] If the sub-dimension combination 1 pointed to by the cell to be filled is "Company A + January 2024 + RMB + Revenue 1-2", the intersection positions (such as the xth position) that all store the same cell data can be similarly understood as the cell data (such as the value 2000) stored in the xth binary bit in each target bit vector is the target data required for the cell to be filled, that is, the data stored here fully corresponds to each dimension value member of the sub-dimension combination 1, no more and no less. In other words, the data at the xth position in each target bit vector is completely associated with each sub-dimension pointed to by the cell to be filled, while the cell data stored in other positions is not the case. For example, the cell data stored in the x+nth position (n≥1) may only correspond to 1 to 3 sub-dimensions in the sub-dimension combination 1, that is, the corresponding sub-dimensions are incomplete, and thus are not the target data required for the cell to be filled, and should be ignored.

[0063] As described above, compared to the index tree structure in the prior art, each set of bit vectors constructed by the embodiment of the present application can store and correspond to the cell data belonging to the same sub-dimension in the current data table in a compact manner, simplifying the construction cost of the search path. In addition, for complex cell data that belongs to multiple sub-dimensions at the same time (i.e., complex dimension combinations), the location where the cell data is stored in the corresponding bit vector is designed to be the same, and the intersection calculation of these bit vectors is performed, which helps to quickly determine whether the cell data exists in the current data table, avoiding the problem of low linear retrieval efficiency (i.e., the need to traverse the tree branches from beginning to end) caused by the deep depth of the traditional index tree, thereby speeding up the retrieval efficiency and tabulation time of the required data.

[0064] Based on the above example descriptions, some specific possible implementation examples will be provided below. These added or refined example contents or possible implementation methods do not necessarily have to be executed during the specific implementation. If more than two examples or possible implementation methods are added, these examples or possible implementation methods can be implemented in combination or separately. If they are implemented in combination, the execution order between the combined examples can be determined according to their respective processing logic, which may depend on the actual scenario.

[0065] based on Figure 2 In some specific examples, the specific operations of step S21 may include: based on the total amount of cell data to be stored in the current data table, determining the size and type of the bit vector corresponding to each sub-dimension in the current data table to construct a bit vector for the data to be stored; storing each cell data in turn in the bit vector corresponding to the sub-dimension to which it belongs, so as to fill in the bit vector corresponding to each sub-dimension, and marking the location of the cell data in the bit vector to indicate the presence of the cell data; wherein, when the cell data in the current data table belongs to multiple sub-dimensions, the locations where the cell data are stored in the corresponding bit vectors are the same.

[0066] Specifically, based on the total amount N of all cell data to be retrieved in the current report (or the total amount of data), the size and type of the bit vector used to store the N items of data can be set first to construct a bit vector of the data to be stored. The bit vector at this time can be regarded as an empty shell or empty set of the data to be entered. Figure 3 As shown, each dimension (node) and its sub-dimensions can be considered as an index model as a whole, and each dimension (node) can be considered as a dimension index map or a large directory. There are multiple sub-dimensions under each node, and the bit vector (or bitmap index) corresponding to each sub-dimension is equivalent to the thumbnail or small directory of the node to which it belongs. These N items of data can be added to the index model in sequence. Specifically, each cell data of the current report is stored in the bit vector corresponding to the sub-dimension to which it belongs in sequence, so as to fill in the bit vector corresponding to each sub-dimension. At this time, the bit vector is a non-empty vector after the data is entered.

[0067] For example, the sub-dimension combinations corresponding to the three cell data items "Amount 1, Amount 2, Amount 3" in the current data table can be as follows: Figure 3 As shown, these three items of cell data can be put into the dimension index map respectively; specifically, they can be put into the corresponding bitmap index (ie, bit vector) under their respective dimensions to complete the classification and storage of massive data.

[0068] In summary, the above dimension index graph: utilizes the similarity of the sub-dimension combination contained in the same report data, decomposes or analyzes each dimension member value (i.e., sub-dimension), and uses independent dimension member values ​​as word roots to construct an index structure for such dimension data. Each cell data (such as a specific amount value) in the same report is marked only with those known sub-dimensions defined by the table, which is the aforementioned similarity. The above bitmap index: uses bit vectors to store index data, which is very memory-efficient in terms of space, and can efficiently process the intersection of massive index data.

[0069] As explained above, it can be understood that the embodiment of the present application is to store or integrate the cell data belonging to the same sub-dimension in the same data table into a set of bit vectors, and the multiple sets of bit vectors of the current data table are divided according to the corresponding sub-dimensions; the multiple sets of bit vectors of the current data table can be combined and regarded as a dictionary structure (or index model) for correspondingly representing the current data table, and this dictionary structure is used to streamline the storage and quickly access each sub-dimension and its corresponding cell data (or report data), thereby simplifying the time and memory cost of constructing the search path, increasing the retrieval ability of a single dimension or a combined dimension of the data, and improving the retrieval efficiency of the report data. Or it can be simply understood that the embodiment of the present application is to write the massive data in the same data table into different pages of the dictionary in sequence (that is, each data is added to a page separately), and to number each data in the order of addition (that is, to mark the page number), and this number is the page number of the page, so that the subsequent search can know which page of the dictionary each data is in, which is convenient for subsequent data indexing.

[0070] In some specific examples, the cell data belonging to the sub-dimension in the current data table may be stored in the bit vector at positions ranked in the order in which they are stored in the corresponding bit vector.

[0071] Exemplarily, when the cell data belongs to n (n≥2) sub-dimensions, the cell data can be stored at one time in each bit vector corresponding to the n sub-dimensions, specifically in the binary bit at the xth position (x is the order number of the data being uniformly stored in the n bit vectors) in the n bit vectors, to indicate that the cell data exists at this position, so as to perform intersection calculation and fast indexing according to the position later. In short, each data can be added to the same position in each bitmap index at one time, and the position number is the order number of the data being uniformly added to the bitmap index, or it can be simply considered that each data is added to the same position of different bit vectors at the same time; that is, the cell data can be added to the same position in each bit vector corresponding to the n sub-dimensions at the same time. Of course, the value of n can also be 1, that is, the cell data can belong to 1 sub-dimension, which depends on the specific situation.

[0072] In other words, each data can be numbered in sequence according to the order x in which the data is added to the bit vector (eg, ID number = x), and the position sequence number x is recorded at the xth position in the bit vector to mark that the xth cell data exists at this position.

[0073] based on Figure 2 In some specific examples, when the above total amount is N, the bit vector is a long type array of size N / 64+1, or a byte array of size N / 8+1, that is, the bit vector can be specifically called a bit array. Of course, this bit vector itself can also be a vector of integer type (such as int or long type) or binary type. Furthermore, the size and type of the bit vector can be comprehensively determined based on multiple factors such as the total amount N and the space occupied by each data, and the embodiments of the present application are not limited. Taking the N / 64+1 long type array as an example, the bit vector is constructed or represented as an array of long type and size N / 64+1 to realize data storage, taking into account that each long type integer can store or occupy 64 binary bits, so in the scenario of massive data, the number of array expansions can be effectively reduced, reducing storage space. In addition, the "+1" in the expression N / 64+1 takes into account that if N is not a multiple of 64, there will be an extra bit or several bits left unstored, so 1 needs to be added (i.e. rounded up) to ensure that there is enough space to store all the bits, that is, to ensure that N data can be stored. Similarly, a byte integer in the above byte array can store 8 binary bits. Then, correspondingly, the position sequence number x can be used to record the existence of a cell data at this position on the xth binary bit of the bit vector.

[0074] In short, the embodiment of the present application adopts bit vectors because binary bits are the smallest unit in computer memory, and the computer's operations on binary bits are also relatively efficient. This ensures that the bitmap index can efficiently record massive data and save memory space.

[0075] based on Figure 2 In some specific examples, the specific operations of step S21 may include: for each dimension and sub-dimension corresponding to each cell data in the current data table, calculating and recording the distribution position of the dimension in all dimensions, and calculating and recording the distribution position of the bit vector corresponding to the sub-dimension under the corresponding dimension; the distribution positions of the dimensions and bit vectors are used to index the target bit vector.

[0076] For example, each time a cell data item is added to the index model, its sub-dimension combination can be decomposed. When processing each dimension, the hash function can be used to calculate the distribution position of the dimension Node to which each data belongs in the dimension index graph, so that the dimension Node corresponding to the data can be quickly located or directly obtained without traversal, thereby speeding up the efficiency of subsequent positioning and obtaining the bit vector corresponding to the data; similarly, the hash function can also be used to calculate the distribution position of the bit vector corresponding to each sub-dimension in the dimension Node to which it belongs.

[0077] For another example, a search algorithm other than the hash algorithm, such as a binary search algorithm or a block search algorithm, may be used to calculate the distribution position of each dimension and each bit vector mentioned above, which will not be described in detail.

[0078] In some specific examples, the distribution position of each dimension and each bit vector can be recorded through a hash table. Specifically, the above distribution position can be recorded in the hash table with the corresponding calculated hash value, so that the dimension Node to which the required data belongs can be quickly located without traversal, thereby speeding up the positioning and acquisition of the required target bit vector, completing the search efficiency of the required data, and using a hash table to represent the distribution position can reduce the occupation of resources and facilitate fast indexing.

[0079] As described above and Figure 3 The actual search and deduction process of the embodiment of the present application may be as follows:

[0080] (1) For a cell to be filled in with sub-dimension combination 1 being [A0, B1, C1, D1], to find out whether there is cell data (or target data) in the current data table that can be filled in the cell to be filled in, you can first analyze whether the sub-dimension combination 1 corresponds to the four dimensions A, B, C, and D (such as account, period, currency, and company), and determine the distribution positions of these related dimension nodes in the dimension index graph, and then obtain these four dimension nodes. This process is equivalent to checking which large directory in the dimension index graph the target data belongs to.

[0081] (2) Similarly, the distribution position of the bitmap index (i.e., bit vector) corresponding to each sub-dimension (such as A0, B1, C1, D1) in the corresponding dimension Node can be determined, and then the bitmap indexes corresponding to the four sub-dimensions can be obtained.

[0082] (3) Perform intersection processing on the four obtained bitmap indexes. If it is known that the same position x in the four bitmap indexes all records a certain data (this data can be the value 1000 which can be called the planned amount), then this value 1000 can be used as the target data; this position x is the position sequence number x of the value 1000. Afterwards, through this position sequence number x, the value 1000 can be indexed and taken out from the index model to fill in the above-mentioned cell to be filled, so as to obtain the following: Figure 4 The target data table is shown filled with corresponding data.

[0083] based on Figure 2 In some specific examples, after "intersection calculation of cell data stored in the same position of multiple groups of target bit vectors" in step 23, the method of the embodiment of the present application may also include: if there is no intersection position, determining that the required cell data corresponding to the cell to be filled does not exist in the current data table, and initiating a null value reminder and / or a reminder to fill in the required cell data in the cell to be filled.

[0084] In actual situations, there may indeed be no required cell data (i.e., target data) corresponding to the cell to be filled in the current data table. For example, for a cell to be filled whose sub-dimension combination 2 is [A0, B1, C1], the data stored in each binary bit in the multiple groups of target bit vectors in the current data table only corresponds to a part of the sub-dimensions pointed to by the cell to be filled, and is not the target data that "corresponds to sub-dimension A0, sub-dimension B1, sub-dimension C1, and all sub-dimensions". Therefore, a null value reminder can be initiated to prompt that there is no target data in the current data table, and / or to remind you to fill in the required cell data in the cell to be filled. If you choose to fill in the data, the corresponding bit vector will also fill in this data to form a new bit vector, and repeat the above steps S22 and S23 to try to complete the data search and filling of the target data table again.

[0085] In summary, in order to solve the problem of efficient retrieval of massive report data, the embodiments of the present application redesign the retrieval model, introduce the concept of dimension index graph, and establish a dictionary structure (including multiple sets of bit vectors) for the dimension member values ​​(i.e., sub-dimensions) contained in each dimension and their corresponding cell data, thereby simplifying the time and memory cost of building the retrieval model, increasing the single-dimensional retrieval capability of data, and improving the retrieval efficiency of report data.

[0086] Specifically, 1) by introducing the concept of dimension index graph, a dictionary structure is established for the dimension member values ​​contained in each dimension and their corresponding cell data, which can ensure that for any dimension member value, a large amount of data corresponding to this dimension member value can be obtained by searching once.

[0087] 2) For massive report data containing complex dimensional combinations, the search can be completed by intersecting the dictionary indexes (i.e., bit vectors) corresponding to a single sub-dimension through binary bit operations and other positional knowledge, thus avoiding the problem of low efficiency of linear search (needing to traverse each branch from beginning to end) caused by the high depth of the traditional index tree. In addition, the multiple sets of bit vectors constructed for each data table in the embodiment of the present application can avoid the problem that the traditional index tree will repeatedly create index nodes under different sub-dimensional combination paths, resulting in more time and memory consumption for constructing the model. That is, compared with the traditional index tree, the embodiment of the present application can improve the search efficiency of complex cell data and reduce cost investment.

[0088] 3) Hash tables and bitmap indexes are selected as the index data structure to achieve compressed storage of index data, thereby greatly reducing the performance consumption of index construction.

[0089] See also Figure 5 ,The second aspect of the present application provides a specific embodiment of a data retrieval system, the system comprising: an acquisition unit, a processing unit;

[0090] The acquisition unit is used to acquire multiple groups of bit vectors corresponding to the current data table; each group of bit vectors corresponds to a sub-dimension as a table header in the current data table, and stores cell data belonging to the corresponding sub-dimension in the current data table; when the cell data belongs to multiple sub-dimensions, the cell data is stored in the same position in the corresponding bit vector;

[0091] The processing unit is used to determine, for each cell to be filled in the target data table, a target bit vector from the plurality of sets of bit vectors; the sub-dimension pointed to by the target bit vector is the same as the sub-dimension pointed to by the cell to be filled;

[0092] The processing unit is also used to, if the determined target bit vectors are multiple groups, perform intersection calculation on the cell data stored in the same position in the multiple groups of target bit vectors. If there are intersection positions that store the same cell data, take out the cell data at the intersection position from any target bit vector and fill it into the cell to be filled, so as to draw a target data table filled with specific data.

[0093] In some examples, the acquisition unit is specifically used to:

[0094] Based on the total amount of cell data to be stored in the current data table, determine the size and type of the bit vector pre-corresponding to each sub-dimension in the current data table to construct the bit vector of the data to be stored;

[0095] Each cell data is sequentially stored in the bit vector corresponding to the corresponding sub-dimension to which it belongs, so as to fill the bit vector corresponding to each sub-dimension, and the location of the cell data is marked in the bit vector to indicate the existence of the cell data.

[0096] In some examples, the cell data belonging to the sub-dimension in the current data table is stored in the bit vector at the positions ranked in order according to the order in which it is stored in the corresponding bit vector.

[0097] In some examples, when the total amount is N, the bit vector is an array of long type with size N / 64+1, or a byte array with size N / 8+1.

[0098] In some examples, the acquisition unit is specifically used to:

[0099] For each dimension and sub-dimension corresponding to the data in each cell in the current data table, the distribution position of the dimension in all dimensions is calculated and recorded, and the distribution position of the bit vector corresponding to the sub-dimension in the corresponding dimension is calculated and recorded; the distribution positions of the dimension and the bit vector are used to index the target bit vector.

[0100] In some examples, the processing unit is also used to:

[0101] If there is no intersection position, it is determined that the required cell data corresponding to the to-be-filled cell does not exist in the current data table, and a null value reminder and / or a reminder to fill in the required cell data in the to-be-filled cell is initiated.

[0102] In the embodiment of the present application, the operations performed by each unit of the data retrieval system are similar to the operations described in the first aspect or any specific method embodiment of the first aspect, and will not be described in detail here. Of course, the specific implementation process of each operation in the first aspect of the present application can also refer to the relevant description of the second aspect.

[0103] See also Figure 6 The electronic device of the embodiment of the present application may include one or more processors (such as central processing units (CPU)) and a memory, in which one or more applications or data are stored.

[0104] The memory may be a volatile memory or a persistent memory. The program stored in the memory may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the processor may be configured to communicate with the memory and execute a series of instruction operations in the memory on the electronic device.

[0105] The electronic device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0106] The processor can execute the operations performed by the aforementioned first aspect or any specific method embodiment of the first aspect, and the details will not be repeated here.

[0107] The present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect or any specific implementation of the first aspect.

[0108] The present application provides a computer program product comprising instructions or a computer program. When the computer program product is run on a computer, the computer is enabled to execute the method described in the first aspect or any specific implementation of the first aspect.

[0109] It is understood that in various embodiments of the present application, the sequence number of each step does not mean the order of execution, and the execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The operation content added or refined in each example scheme of the above method, system or device (if any) does not necessarily have to be executed in the specific implementation. If more than two operations are added, these operations can be implemented in combination or separately, depending on the actual scenario.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system (if any) and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system or device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product (or computer program product) is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a business server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data retrieval method, characterized in that: include: Acquire multiple groups of bit vectors corresponding to and representing the current data table; wherein each group of the bit vectors corresponds to a sub-dimension as a table header in the current data table, and stores cell data belonging to the corresponding sub-dimension in the current data table; when the cell data belongs to multiple sub-dimensions, the positions where the cell data are stored in the corresponding bit vectors are the same; For each cell to be filled in the target data table, a target bit vector is determined from the plurality of groups of bit vectors; the sub-dimension pointed to by the target bit vector is the same as the sub-dimension pointed to by the cell to be filled; If the determined target bit vectors are multiple groups, the intersection calculation is performed on the cell data stored in the same position in the multiple groups of the target bit vectors. If there are intersection positions that store the same cell data, the cell data at the intersection position is taken from any of the target bit vectors and filled into the cell to be filled, so as to draw the target data table filled with specific data.

2. The data retrieval method according to claim 1, characterized in that: The obtaining of a plurality of sets of bit vectors corresponding to and representing the current data table includes: Based on the total amount of cell data to be stored in the current data table, determine the size and type of the bit vector pre-corresponding to each sub-dimension in the current data table to construct a bit vector of the data to be stored; Each of the cell data is sequentially stored in the bit vector corresponding to the corresponding sub-dimension to which it belongs, so as to fill the bit vector corresponding to each of the sub-dimensions, and the position of the cell data in the bit vector is marked to indicate the presence of the cell data.

3. The data retrieval method according to claim 1 or 2, characterized in that: The cell data belonging to the sub-dimension in the current data table are stored in the positions ranked in the order in the bit vector according to the order in which they are stored in the corresponding bit vector.

4. The data retrieval method according to claim 2, characterized in that: When the total amount is N, the bit vector is a long type array with a size of N / 64+1, or a byte array with a size of N / 8+1.

5. The data retrieval method according to claim 1 or 2, characterized in that: The obtaining of a plurality of sets of bit vectors corresponding to and representing the current data table includes: For each dimension and sub-dimension corresponding to each cell data in the current data table, the distribution position of the dimension in all dimensions is calculated and recorded, and the distribution position of the bit vector corresponding to the sub-dimension in the corresponding dimension is calculated and recorded; the distribution positions of the dimension and the bit vector are used to index the target bit vector.

6. The data retrieval method according to claim 1, characterized in that: After performing intersection calculation on the cell data respectively stored at the same position in the plurality of groups of the target bit vectors, the method further comprises: If the intersection position does not exist, it is determined that the required cell data corresponding to the to-be-filled cell does not exist in the current data table, and a null value reminder and / or a reminder to fill in the required cell data in the to-be-filled cell is initiated.

7. A data retrieval system, characterized in that: include: Acquisition unit, processing unit; The acquisition unit is used to acquire multiple groups of bit vectors corresponding to the current data table; each group of the bit vectors corresponds to a sub-dimension as a table header in the current data table, and stores the cell data belonging to the corresponding sub-dimension in the current data table; when the cell data belongs to multiple sub-dimensions, the cell data is stored in the same position in the corresponding bit vector; The processing unit is used to determine, for each cell to be filled in the target data table, a target bit vector from the plurality of groups of bit vectors; the sub-dimension pointed to by the target bit vector is the same as the sub-dimension pointed to by the cell to be filled; The processing unit is also used to, if the determined target bit vectors are multiple groups, perform intersection calculation on the cell data stored in the same position in the multiple groups of target bit vectors; if there are intersection positions that store the same cell data, take out the cell data at the intersection position from any of the target bit vectors and fill it into the cell to be filled, so as to draw the target data table filled with specific data.

8. An electronic device, characterized in that: include: Processor and memory; The processor is configured to communicate with the memory and execute instructions in the memory to implement the method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images