Data processing device, data processing method, and program

Abstract

A data processing device according to one aspect of the present disclosure comprises a data operation unit that performs a sort operation, a search operation, or an aggregation operation on virtual tabular data composed of virtual columns having a second data structure obtained by converting a first data structure of one or more columns included in one or more pieces of tabular data by using predetermined mapping, by using the second data structure as an index.

Description

Data processing device, data processing method, and program

[0001] The present disclosure relates to a data processing device, a data processing method, and a program.

[0002] In recent years, advances in various sensor devices and observation equipment have made it possible to obtain tabular data containing large amounts of data (so-called big data) representing sensing results, observation results, and the like. As a result, there is an increasing need to select multiple columns from one or more tabular data sets to create virtual tabular data (hereinafter referred to as virtual tabular data) tailored to one's intended use. One method for achieving this need is to use a technology known as a virtual database or data virtualization (see, for example, Non-Patent Document 1). This technology accepts a query from a user and executes a subquery on a distributed database behind the query.

[0003] "Data Hub vs. Data Lake vs. Data Virtualization - MarkLogic," Internet <URL: https: / / jp.marklogic.com / product / comparisons / data-hub-vs-data-lake / >

[0004] However, with the technology known as virtual database or data virtualization, it is not possible to have the virtual database inherit the indexes that realize sorting, searching, and aggregation of the real database.

[0005] The present disclosure has been made in consideration of the above points, and provides a technique that enables indexes that realize sorting, searching, and aggregation of real tabular data to be inherited by virtual tabular data.

[0006] A data processing device according to one aspect of the present disclosure has a data manipulation unit that performs a sort operation, a search operation, or an aggregation operation on virtual tabular data composed of virtual columns having a second data structure obtained by converting a first data structure of one or more columns included in one or more tabular data using a predetermined mapping, using the second data structure as an index.

[0007] A technology is provided that allows indexes that enable sorting, searching, and aggregation of real tabular data to be inherited by virtual tabular data.

[0008] 1 is a diagram illustrating an example of virtual tabular data constructed hierarchically starting from D5A that holds values. FIG. 1 is a diagram illustrating an example (part 1) of A mapping. FIG. 2 is a diagram illustrating an example (part 2) of A mapping. FIG. 3 is a diagram illustrating an example of D5A. FIG. 4 is a diagram illustrating an example of allocation. FIG. 5 is a diagram illustrating an example of an allocation method using enumeration mapping. FIG. 6 is a diagram illustrating an example of an allocation method using linear function mapping. FIG. 7 is a diagram illustrating an example of a source column and a virtual column. FIG. 8 is a diagram illustrating an example of a source column, a transposed structure, and a virtual transposed structure. FIG. 9 is a diagram illustrating an example of allocation mapping for two-level virtual tabular data. FIG. 10 is a diagram illustrating an example of a virtual transposed structure for two-level virtual tabular data. FIG. 11 is a diagram illustrating an example of the overall configuration of a system including a data processing device according to an embodiment. FIG. 12 is a diagram illustrating an example of the hardware configuration of a data processing device according to an embodiment. FIG. 13 is a flowchart illustrating an example of the flow of creation processing of virtual tabular data. FIG. 14 is a flowchart illustrating an example of the flow of sorting processing in virtual tabular data. FIG. 15 is a flowchart illustrating an example of the flow of search processing in virtual tabular data. FIG. 16 is a flowchart illustrating an example of the flow of aggregation processing in virtual tabular data.

[0009] An embodiment of the present invention will be described below. In the following embodiment, first, necessary explanations and definitions are provided, followed by a definition of data called D5A that represents tabular data. Next, a method for creating new virtual tabular data from D5A or other virtual tabular data, and a method for inheriting indexes from (real) tabular data to sort, search, and aggregate the virtual tabular data will be described. It will also be explained that, in this case, indexes can be inherited even when the virtual tabular data is hierarchically structured. Finally, a data processing device 10 capable of creating, sorting, searching, and aggregate such virtual tabular data will be described.

[0010] <Introduction> Archive data such as IoT (Internet of Things) data, various observation data, and log data is being accumulated daily. In many cases, tabular data is compiled into a single data set at regular intervals, such as daily or monthly, and added to the archive. Tabular data added to the archive can be considered read-only. Tabular data in such archives can become huge, and is often distributed across LANs (Local Area Networks) and the Internet.

[0011] When trying to utilize the tabular data in the archive, the following two steps are generally required, and there is a problem in that both steps often take a long time and consume a large amount of storage space.

[0012] The first step is to generate new tabular data. This new tabular data is created by unioning or joining multiple tabular data sets, or by extracting only the necessary columns. This can take a long time if the original tabular data is huge or distributed over a wide area network such as the Internet. Furthermore, if the new tabular data is large, a large amount of storage space is required to store it.

[0013] The second step involves sorting, searching, and tabulating. The newly created tabular data is not indexed, so sorting, searching, and tabulating takes time. In addition, the sorting results when the new tabular data is large, the search results when the number of hits is large, and the tabulation results all require large storage areas.

[0014] The problems in the above two steps are becoming more serious with the increasing size of archive data and the growing need to utilize distributed archive data.

[0015] Therefore, we propose a technology that shortens the time required for the above two steps to the time required for interactive operation and requires only a small amount of storage space.The tabular data to be operated on may be, for example, one trillion records or 100,000 columns.Furthermore, the tabular data to be operated on may be hierarchically stacked, using combinations such as unions and joins.Furthermore, the tabular data to be operated on may be distributed on a LAN or multiple HTTP (Hypertext Transfer Protocol) servers on the Internet.

[0016] This is possible because a network system for mapping archive data can be constructed using D5A, a file format for tabular data with indexes for every column that speeds up sorting, searching, and aggregation; virtual tabular data that inherits values ​​directly or indirectly from the D5A file; and virtual indexes on the virtual tabular data that are automatically established by directly or indirectly inheriting the data structure used by the D5A file's index. The virtual tabular data and virtual indexes are immediately available by simply connecting them directly or indirectly to D5A, and they consume very little storage space. Additionally, sorting, search, and aggregation results obtained using virtual indexes require very little storage space, no matter how large they are.

[0017] Such a mapping network of archive data using D5A enables new ways of using archive data, in which users create virtual tabular data according to their purpose and use archive data distributed on the network interactively. For example, it enables each organization to use archive data distributed among its various departments for various purposes, and on the Internet, it enables archive data such as IoT data from around the world to be combined and extracted and used in a variety of linked ways.

[0018] Here, we clarify the terminology used in this specification. First, D5A refers to a file format for tabular data in which all columns have indexes that speed up sorting, searching, and aggregation. Next, virtual tabular data refers to tabular data that inherits values ​​from a D5A file (a file in D5A format) or other virtual tabular data. The tabular data from which it inherits values ​​is called source tabular data. Furthermore, columns on D5A are called D5A columns, and indexes that speed up sorting, searching, and aggregation of D5A columns are called D5A indexes. Similarly, virtual tabular data is called virtual columns and virtual indexes. Similarly, source tabular data is called source columns and source indexes. Furthermore, the data structure used by a D5A index is called an inverted structure, the data structure used by a virtual index is called a virtual inverted structure, and the data structure used by a source index is called a source inverted structure. Note that virtual tabular data may also be called "mapped tabular data," and similarly, virtual columns may also be called "mapped columns."

[0019] <Solution to the problem> As mentioned above, when trying to utilize tabular data in an archive, two steps are generally required: Step 1 and Step 2, both of which take a long time and often consume a large amount of storage space. Note that Step 1 is a step for generating new tabular data by performing UNION, JOIN, or extracting only the necessary columns, and Step 2 is a step for sorting, searching, and aggregating the data.

[0020] The first step takes time because it takes time to read, compare, and store values. Instead, a possible approach is to create new tabular data using a mapping defined by a correspondence table and rules that maps cells in the source tabular data to cells in the newly created tabular data. In the case of archived data, the mapping definition is often possible and compact. In this case, the new tabular data does not need to contain values, which is a significant advantage, especially when the new tabular data is large. The new tabular data can be displayed in just the short time it takes to load the mapping definition, and it requires only a small amount of storage space to hold the mapping definition. This new tabular data does not retain values ​​but inherits them from the original tabular data, making it virtual tabular data. The problem of the first step can be solved using such virtual tabular data.

[0021] The second step takes time because the columns of the newly created tabular data are not indexed, and because it takes time to write out the often-huge results of sorting, searching, and aggregation. However, the time-consuming problem is solved by two reasons: every column in the virtual tabular data automatically has a virtual index that speeds up sorting, searching, and aggregation for that column, and because this virtual index can store even huge results of sorting, searching, and aggregation with only a small amount of memory, thereby reducing the write time. Furthermore, the storage space problem is also solved because the virtual index requires only a small amount of memory to store these results. In this way, the problem in the second step can be solved by virtual indexes.

[0022] The D5A file, which is expressed in a storage format for tabular data called D5A, holds the values ​​that are the basis of such virtual tabular data and the data structure that is the basis of the data structure used by the virtual index.

[0023] <<D5A: Storage Format of Tabular Data That Provides Values ​​and Indexes>> Virtual tabular data inherits values ​​from one or more source tabular data. The source tabular data is either another virtual tabular data or tabular data called a D5A file. In the former case, the virtual tabular data inherits values ​​from yet another source tabular data, ultimately arriving at a D5A file. Therefore, it can be said that virtual tabular data is structured hierarchically, directly or indirectly inheriting values ​​from D5A files.

[0024] A virtual index uses a virtual inverted structure to speed up sorting, searching, and aggregation. The virtual inverted structure is automatically established by inheriting one or more source inverted structures. A source inverted structure is either a virtual inverted structure on another virtual tabular data, or an inverted structure on a D5A file. In the former case, the virtual inverted structure inherits yet another source inverted structure, ultimately arriving at the inverted structure on the D5A file. Therefore, it can be said that the virtual inverted structure is established hierarchically by directly or indirectly inheriting the inverted structure on the D5A file.

[0025] D5A is a storage format for tabular data to provide values ​​to virtual tabular data and inverted structures to virtual inverted structures, with a D5A index that holds values ​​and uses the inverted structures for all its columns.

[0026] <<Implementation of a Network System for Mapping Archive Data>> A simple hypothetical example of a network system for mapping archive data is shown below, and the use of the D5A, virtual tabular data, and virtual indexes described above is explained with reference to Figure 1. Figure 1 illustrates the process of combining meteorological observation data from Sunday to Saturday for Tokyo, Osaka, and Nagoya. This process is performed in two stages: first, seven daily data sets for each region are combined into weekly data, and then the weekly data sets are juxtaposed to form a single tabular data set. Here, the 21 daily data sets for one week for three regions on the left side of Figure 1 are stored in a 55A file. The three tabular data sets for each region in the center of Figure 1 are virtual tabular data sets. Each of these virtual tabular data sets uses the seven D5A files for one week on the left side of Figure 1 as source tabular data, extracts the necessary columns from them, and then unions them. The single virtual tabular data set on the right side of Figure 1 juxtaposes the three virtual tabular data sets for each region in the center of Figure 1 as source tabular data.

[0027] As shown in Figure 1, the mapping network for archive data starts with the D5A file, with virtual tabular data built hierarchically on top of it. As mentioned above, virtual tabular data is automatically equipped with virtual indexes, allowing any column to be sorted, searched, and aggregated in a short amount of time, and only a small amount of storage space is required to store the sorting, search, and aggregation results. This type of virtual tabular data can be created and used by users themselves, which is expected to promote the widespread use of archive data.

[0028] <A-mapping> D5A, virtual tabular data, and virtual indexes are all described as a combination of mappings from a continuous interval of natural numbers starting from 0 to some value (which is also often a natural number). This mapping can be represented as a one-dimensional array (Array) with subscripts (indexes) starting from 0, so it is called the A-mapping. Using the A-mapping makes it possible to view the correspondence relationships from an overall perspective, and as a result, it becomes possible to derive algorithms that utilize the properties of sets and groups. The advantages of the A-mapping are explained below.

[0029] The first advantage of A-mapping is that it can be combined in various ways to create new A-mappings. For example, let's take A-mapping S, which represents the search result column, and A-mapping C, which represents column A. Ａ By combining these, we obtain a mapping A that represents the search results of column A, and a mapping C that represents column B. Ｂ By combining with this, we can obtain the A mapping that represents the search results of column B. In this case, S can be any A mapping that represents the result column, and C Ａ and C Ｂ The A-mapping can be any A-mapping that represents a column. They can exist on a local storage device or on a network. Such a combination of A-mappings can be said to be an algebraic combination.

[0030] Furthermore, the result of combining A maps and A maps does not necessarily need to be written to a storage area; it can be a virtual A map. A virtual A map is an A map that uses one or more A maps as the combination source, has a known overall size, and allows for the extraction of any i-th part from the combination source. Because a virtual A map is also an A map, it is possible to combine virtual A maps to create further virtual A maps hierarchically. A column in virtual tabular data is a type of virtual A map and can be created hierarchically. An index is a mechanism realized by a data structure for the index and an algorithm that uses that data structure. A virtual index is an index that uses one or more virtual A maps as the data structure for the index and can be created hierarchically.

[0031] The second advantage of A-mapping is that it can be decomposed to create a new A-mapping. By decomposing an A-mapping, it is possible to create and use a new A-mapping with functions not present in the A-mapping before decomposition. One particularly effective decomposition is the LP decomposition (decomposition into A-mapping L and A-mapping P) of A-mapping M, which defines the correspondence (mapping) between cells in tabular data. Decomposing M into L and P enables efficient element search using bisection search in L, and makes it possible to obtain the inverse mapping in P. In this case, M can be any A-mapping that defines the mapping, and L automatically becomes an A-mapping that can be bisection searched, and P automatically becomes an A-mapping with an inverse. Because the decomposition of an A-mapping produces a new A-mapping, it can be considered an algebraic decomposition.

[0032] On the other hand, since the A map is a one-dimensional array, it takes time to insert or delete elements into a large A map. However, since archive data is rarely updated, this disadvantage is not a problem.

[0033] Using such an A mapping, it is possible to design D5A, a storage format for tabular data that equips every column with an index that allows for high-speed sorting, searching, and aggregation. Virtual tabular data can then be defined using D5A and the A mapping. This virtual tabular data automatically has a virtual index that uses one or more A mappings as the data structure for the index. This makes it possible to realize a network system of archive data mappings that can be used in various combinations of archive data distributed across a network.

[0034] Therefore, in the following, we first define the A-map and establish its notation. Next, we introduce the index operator, which is an operator for combining A-maps. Next, we discuss the decomposition of the A-map, explaining the particularly important SN decomposition, LP decomposition, and spectral decomposition. Finally, we classify the A-map according to four aspects.

[0035] <<Definition of A-mapping>> Consider a one-dimensional array of size N, with an index starting from 0. This one-dimensional array can be seen as a mapping whose domain is a continuous interval of natural numbers from 0 to N-1, and whose range is a maximum of N types of discrete intervals. This is called an A-mapping. If records are assigned consecutive record numbers starting from 0, the columns of tabular data can also be seen as an A-mapping with the record numbers as the domain.

[0036] For example, consider a one-dimensional array in which "4" is stored as the 0th element, "0" as the 1st element, "6" as the 2nd element, and "3" as the 3rd element, as shown in Figure 2. This one-dimensional array maps 0 to "4", 1 to "0", 2 to "6", and 3 to "3", so it can be regarded as an A-mapping with a domain of {0, 1, 2, 3} and a range of {0, 3, 4, 6}.

[0037] 3, consider a one-dimensional array in which "Bob" is stored as the 0th element, "Alice" as the 1st element, "Cathy" as the 2nd element, and "Bob" as the 3rd element. This one-dimensional array associates 0 with "Bob", 1 with "Alice", 2 with "Cathy", and 3 with "Bob", so it can be regarded as an A-mapping with a domain of {0, 1, 2, 3} and a range of {Alice, Bob, Cathy}.

[0038] Although the A mapping can be expressed as a one-dimensional array, it is called a mapping when discussing its mapping aspect, and an array when discussing its operations. Accordingly, when treated as a mapping, it is called a domain and a range, and when it is an array containing columns or tabular data, it is called a record number and a value. However, the notation for the A mapping will be used as an array whenever possible.

[0039] <Notation of A-mapping> Below is a notation of A-mapping that incorporates general array notation.

[0040] Notation 1. When defining a mapping by enumeration of elements, (a ０ , a １ , ..., a ｎ－１ ) should be written.

[0041] Notation 2. When specifying the domain n of A-mapping A, use A （ｎ） According to this notation, the columns are written as follows: R is the total number of records, and C is the total number of records. （Ｒ） etc.

[0042] Notation 3. When we know that the range of an A-mapping A is over the natural numbers and that the maximum value of the range does not exceed n-1, we can use A to explicitly state the range. （ｎ） It is written as follows.

[0043] Notation 4. A mapping A （ｎ） The i-th element of A （ｎ） [i]. Therefore, A （ｎ） ≡(A （ｎ） [0], A （ｎ） [1], ..., A （ｎ） [n-1]).

[0044] Notation 5. (i ０ , i １ , ...), (j ０ , j １ , ...), ... is connected as (i ０ , i １ , ...) + (j ０ , j １ , ...) + ...

[0045] Notation 6. A mapping λ on the range is a function of i on the domain. ０ , i １ , ... is accepted by λ:(i ０ , i １ , ...) For example, the example shown in FIG. 3 can be written as Alice: (1), Bob: (0, 3), Cathy: (2).

[0046] Notation 7. λ ０ : (i ０ , i １ , ...), λ １ : (j ０ , j １ , ...), ... are joined together using the symbol +, ０ : (i ０ , i １ , ...) + λ １ : (j ０ , j１ , ...) + .... However, in this case, λ ０ <λ １ For example, the example shown in FIG. 3 can be written as Alice: (1) + Bob: (0, 3) + Cathy: (2).

[0047] <<Index Operator for Combining A Mappings>> An operator called an index operator, which combines A mappings to create a new A mapping, is defined below.

[0048] A （ｎ） ・B （ｍ） （ｎ） = (A （ｎ） [B （ｍ） （ｎ） [0]],...,A （ｎ） [B （ｍ） （ｎ） [m-1]) Expression (1) The above index operator has the following properties 1 to 3.

[0049] Property 1: The associative law holds. That is, (A （ｎ） ・B （ｍ） )・C （ｋ） = A （ｎ） ・(B （ｍ） ・C （ｋ） ) holds.

[0050] Property 2: The size of the array representing the combined result is the size of the array on the right.

[0051] Property 3: The data type of the array representing the join result is the data type of the left-side array.

[0052] Here, we will verify the associative law using an example. （３） =(Alice, Bob, Cathy), B （４） =(1,0,2,1),C （３） = (0, 2, 1). In this case, (A （３） ・B （４） )・C （３） = (Bob, Alice, Cathy, Bob) * (0, 2, 1) = (Bob, Cathy, Alice). On the other hand, A （３） ・(B （４） ・C （３）) = (Alice, Bob, Cathy)·(1, 2, 0) = (Bob, Cathy, Alice), and the two match.

[0053] Due to the above properties 1 and 2, the number of types of values ​​that can appear in the A mapping obtained as a result of combining A mappings with the index operator is equal to or less than the size of the smallest A mapping among the A mappings combined with the index operator. This is an index similar to the rank in a matrix.

[0054] <<Decomposition of A-map>> Combining A-maps using the index operator results in a single A-map. On the other hand, a single A-map can be decomposed into various A-maps. For example, (Bob, Alice, Cathy, Bob) can be decomposed as (Bob, Alice, Cathy, Bob) = (Alice, Bob, Cathy) · (1, 0, 2, 1), or as (Bob, Alice, Cathy, Bob) = (Bob, Alice, Cathy) · (0, 1, 2, 0).

[0055] Among these, unique decomposition methods are particularly valuable, as they enable faster sorting and searching, and enable inverse mapping. Therefore, the following describes the unique decomposition methods: SN decomposition, LP decomposition, and spectral decomposition.

[0056] SN decomposition is a method of decomposing one A mapping into two, and there is only one way in which the left A mapping of the two resulting A mappings holds only the values ​​that appear in the left A mapping, and holds those values ​​uniquely and in ascending order. This decomposition is called SN decomposition.

[0057] For example, (Bob, Alice, Cathy, Bob) = (Alice, Bob, Cathy) · (1, 0, 2, 1) is an example of SN decomposition.

[0058] The first item on the right side, which is unique and in ascending order, is called the SVL (Sorted Value List). In the above example, SVL = (Alice, Bob, Cathy). The second item on the right side can be thought of as each element on the left side replaced with its storage position in the SVL. This is called the NNC (Natural Numbered Column). In the above example, NNC = (1, 0, 2, 1).

[0059] Such SN decomposition can be generally defined as follows:

[0060] C （Ｒ） =SVL （Ｋ） ・NNC （Ｒ） （Ｋ） i, j∈0,...,K-1 and i<j⇒SVL （Ｋ） [i] <SVL （Ｋ） [j] Equation (2) In equation (2), the upper column is column C （Ｒ） SVL （Ｋ） and NNC （Ｒ） （Ｋ） The bottom row shows that the （Ｋ） The elements of are arranged in ascending order. （Ｒ） （Ｋ） It can be seen from the notation that it is an A-map of size R with natural numbers 0, . . . , K-1 as elements.

[0061] Using SN decomposition, it is possible to extract an A-mapping NNC whose elements are natural numbers from an A-mapping whose elements are not necessarily natural numbers. Using the extracted NNC, efficient algorithms that use the properties of natural numbers, such as counting sort, become available.

[0062] (SN Decomposition Algorithm) Here, an example of the SN decomposition algorithm will be described using (Bob, Alice, Cathy, Bob) as an example. In the SN decomposition algorithm, the following steps 1-1 to 1-5 are executed.

[0063] Step 1-1: Add the position in the A mapping to each value of (Bob, Alice, Cathy, Bob) to create ((Bob, 0), (Alice, 1), (Cathy, 2), (Bob, 3)).

[0064] Step 1-2: Sorting is performed while evaluating the magnitude relationship between pairs of value and position to obtain ((Alice, 1), (Bob, 0), (Bob, 3), (Cathy, 2)).

[0065] Step 1-3: Assign a value number to each value in ascending order starting from 0, and obtain the sorting result ((Alice=0,1), (Bob=1,0), (Bob=1,3), (Cathy=2,2)).

[0066] Note that in the above example, the same value Bob appears twice, but both Bobs are assigned the value number 1. The above sorting result has a structure of ((value = value number, position in A mapping), ...). For example, the first element (Alice = 0, 1) in the sorting result of the above example has the value Alice, the value number 0, and the position in A mapping 1.

[0067] Step 1-4: Using the above sorting results, secure a storage area for SVL (area size 3) and write "SVL [value number] = value". That is, write SVL[0] = Alice, SVL[1] = Bob, SVL[2] = Cathy. This completes SVL = (Alice, Bob, Cathy).

[0068] Step 1-5: Using the above sorting results, secure a storage area for NNC (area size 4) and write it out as "NNC [position in A mapping] = value number". That is, write as NNC[1] = 0, NNC[0] = 1, NNC[3] = 1, NNC[2] = 2. This completes NNC = (1, 0, 2, 1).

[0069] From the above, (Bob, Alice, Cathy, Bob)=(Alice, Bob, Cathy)·(1, 0, 2, 1) is obtained.

[0070] LP decomposition is a special case of SN decomposition that decomposes an A-mapping M (Mapping projection) with unique natural numbers as elements into two. For example, (5, 2, 7, 3) = (2, 3, 5, 7) * (2, 0, 3, 1) is an example of LP decomposition.

[0071] The first term on the right-hand side obtained by LP decomposition has natural numbers as elements and is unique and in ascending order. This is called L (seLection). In the above example, L = (2, 3, 5, 7). On the other hand, the second term on the right-hand side can be thought of as each element on the left-hand side being replaced with its storage position in L. This is called P (Permutation). In the above example, P = (2, 0, 3, 1). In LP decomposition, M, L, and P are the same size.

[0072] When LP decomposition is performed, it is possible to determine the presence or absence of an element in M ​​and the order of the elements using L, and to determine the inverse element using P. Note that LP decomposition can be performed using the same algorithm as SN decomposition.

[0073] Spectral decomposition The above notation 7 expresses the position (index) on the A-map where each value appears as a new A-map. This is called spectral decomposition of the A-map. Spectral decomposition makes it possible to view the A-map from the range to the domain, enabling a variety of algorithms. Below is an example showing how spectral decomposition of a column, which is an A-map, makes it easy to sort, search, and aggregate columns.

[0074] (Sorting) In the example shown in FIG. 3, the sorting is performed by removing values ​​from the spectral decomposition of the column Alice: (1) + Bob: (0, 3) + Cathy: (2) to obtain (1) + (0, 3) + (2), and then performing the concatenation defined in Notation 5 to obtain the sorted result sequence (1, 0, 3, 2). Note that the sorting result is shown in column C. （Ｒ） = (Bob, Alice, Cathy, Bob) and the sorted result string can be combined as follows:

[0075] Sorting result = C （Ｒ） Sorting result string = (Bob, Alice, Cathy, Bob) (1, 0, 3, 2) = (Alice, Bob, Bob, Cathy)

[0076] (Search execution) In the example shown in Figure 3, Bob's search is performed by identifying Bob's part using a bisection search from the spectral decomposition of the column Alice: (1) + Bob: (0, 3) + Cathy: (2), extracting it to obtain Bob: (0, 3), and obtaining the search result string (0, 3). （Ｒ） =(Bob, Alice, Cathy, Bob) and the search result string can be combined as follows:

[0077] Search result = C （Ｒ） Search result string = (Bob, Alice, Cathy, Bob) (0, 3) = (Bob, Bob)

[0078] (Performing Counting) In the example shown in FIG. 3, counting is performed by extracting values ​​and their occurrence counts from the spectral decomposition of the column Alice: (1) + Bob: (0, 3) + Cathy: (2), and obtaining the count result = Alice: 1 time, Bob: 2 times, Cathy: 1 time.

[0079] (General Formula of Spectral Decomposition) Here, spectral decomposition can be generally rewritten as follows.

[0080] In formula (3), R is the total number of records, λ is the key value, w is the number of transposed record numbers belonging to the key value λ, and X is the array of transposed record numbers.

[0081] For example, when the sorting result of the one-dimensional array shown in FIG. 3 is written in the form of equation (3), it is as follows.

[0082] R = 4, λ ０ = Alice, λ １ = Bob, λ ２ =Cathy w ０ = 1, w １ = 2, w ２ = 1 X ０（１） （４） = (1), X １（２） （４） = (0, 3), X ２（１） （４）= (2) (Spectral Decomposition Algorithm) Here, an example of an algorithm for spectrally decomposing the A-map will be shown, taking (Bob, Alice, Cathy, Bob) as an example. The algorithm consists of the following steps 2-1 to 2-3.

[0083] Step 2-1: Add the position in the A mapping to each value of (Bob, Alice, Cathy, Bob) to create ((Bob, 0), (Alice, 1), (Cathy, 2), (Bob, 3)).

[0084] Step 2-2: Sorting is performed while evaluating the magnitude relationship between pairs of value and position to obtain ((Alice, 1), (Bob, 0), (Bob, 3), (Cathy, 2)).

[0085] Step 2-3: Grouping the positions in the A map for each same value completes the spectral decomposition Alice: (1) + Bob: (0, 3) + Cathy: (2).

[0086] <Classification of A-maps> A-maps can be classified based on four criteria: associativity, searchability, whether they represent a set, and whether their inverse can be easily found. Keeping these classifications in mind will help you understand the explanations that follow, so we will explain them here.

[0087] Associativity: A-maps with natural numbers as elements can be placed on both sides of the index operator. A-maps with other elements can only be placed on the left side of the index operator.

[0088] - Searchability An A-map whose elements are arranged in ascending order can be used to efficiently search for whether a specific element exists, and if so, where it is, using a bisection search. Such an A-map is called an ascending array. An A-map whose elements are unique and ascending can be searched for a certain value, and if a match exists, it will be the only one. Such an A-map is called a unique ascending array.

[0089] Whether it represents a set If there are no duplicates in the elements of the A mapping, it can be considered to represent a set. However, in this specification, it is also assumed that the elements are natural numbers. This is because in this specification, a set is a result sequence, which is an A mapping with natural numbers (record numbers) as elements.

[0090] - Can the inverse be easily found? When the size of an A-mapping whose elements are natural numbers is N, if the A-mapping contains all elements with values ​​between 0 and N-1, the inverse with respect to the index operator can be easily found. This is called a symmetric array. The definition of a symmetric array P is shown below.

[0091] i, j∈0, 1, ..., N-1; i≠j⇒P （Ｎ） （Ｎ） [i]≠P （Ｎ） （Ｎ） [j] Symmetric arrays form a group with respect to the index operator. Therefore, there exists an identity element and an inverse element. The inverse element of P is used to identify a cell in the source tabular data from a cell in the virtual tabular data.

[0092] Identity element: E = (0, 1, ..., N-1) Inverse element: P[i] = j ⇒ P －１ [j] = i <D5A> R refers to the number of records in the tabular data, and K refers to the type of values ​​contained in the column. Tabular data has R records identified by numbers 0 to R-1 from top to bottom, and one to multiple columns identified by numbers or names from left to right, where one record and one column define one value, and each column has a unique data type (e.g., integer, floating point, string, etc.). An index is established when the size W of the sort / search / aggregation results for the target column is known, and when i∈0, 1, ..., W-1 is specified, the i-th row of the sort / search / aggregation results can be read in approximately O(log(R)) or less.

[0093] Below, we will first explain the internal structure of D5A, then how to extract values, and then explain D5A indexes along with related topics.

[0094] <<Internal Structure of D5A>> The internal structure of D5A will be described with reference to Figure 4. Figure 4 shows the internal structure of column C （４） 10 shows tabular data in CSV (Comma Separated Values) format including the following data: = (Bob, Alice, Cathy, Bob) and its D5A format.

[0095] D5A has a structure consisting of a collection of D5A columns, each of which has the same structure. Each D5A column consists of two structures, described below, arranged side by side.

[0096] The first structure is a structure for obtaining column values, and is the part surrounded by a dashed line in Figure 4. This structure is configured to either (a) store the column, which is the A mapping, as is, (b) store the SVL and NNC obtained by SN decomposing the column, or (c) store both. In the case of (a) storing the columns as is, the performance of reading values ​​improves, but algorithms using NNC become unusable. In the case of (b) storing SVL and NNC, algorithms using NNC can be used, but the performance of reading values ​​decreases. In the case of (c) storing both, the size of the D5A file increases. In this embodiment, the case of (b) will be explained.

[0097] The second structure is for the D5A index, and is the part enclosed by the dashed line in Figure 4. This structure consists of SVL, ACM, and INV, and is called a transposed structure. The transposed structure is a rewriting of the spectral decomposition of the D5A column described above into the form of an A map. Details of the transposed structure will be described later.

[0098] Once a transposed structure (i.e., second structure) is obtained, it is combined with the first structure to form a D5A column, and the D5A columns are collected to form D5A.

[0099] <<Structure for getting column values>> The structure for getting column values ​​is Column C. （Ｒ） For example, in the example shown in FIG. （４）= (Bob, Alice, Cathy, Bob) = (Alice, Bob, Cathy) * (1, 0, 2, 1). Therefore, SVL （３） =(Alice, Bob, Cathy), NNC （４） （３） = (1, 0, 2, 1).

[0100] On the other hand, using SVL and NNC, the column is obtained by the following formula:

[0101] C （Ｒ） =SVL （Ｋ） ・NNC （Ｒ） （Ｋ） Equation (4) Therefore, the i-th row of the column is obtained by the following equation.

[0102] value = C （Ｒ） [i] = SVL （Ｋ） ・NNC （Ｒ） （Ｋ） [i] Equation (5) For example, in the example shown in FIG. 4, the first row of the column is (Alice, Bob, Cathy)·(1,0,2,1)[1]=Alice.

[0103] The following information 1 to 4 can also be obtained from the SVL.

[0104] Information 1: The number of types of values ​​in the column, K, is SVL （Ｋ） For example, in the example shown in FIG.

[0105] Information 2: The ith smallest value can be read as SVL[i].

[0106] Information 3: Whether a value v exists in a column can be determined by a bisection search of SVL. If the value v exists, it is also possible to determine the ordinal position of the value v.

[0107] Information 4: SVL is the dimension of the aggregation. Note that in the past, sorting was required to create the dimension of the aggregation.

[0108] <<Transposed Structure>> Next, we will explain how to create a transposed structure, which is a structure for a D5A index. We have mentioned that spectral decomposition of a column allows for high-speed sorting, searching, and aggregation, but spectral decomposition has two difficult points. The first is that it is not based on the A mapping, so it cannot be accessed using an index operator (problem number 1). The second is that when trying to obtain the i-th item in the sorted result column, w ０ , w １ The second problem is that it is necessary to search for the i-th element while adding , .... The transposed structure is obtained by converting the spectral decomposition into three A-maps, and it solves these two problems. The definition and creation method of the transposed structure are explained below.

[0109] Definition of transposed structure and method of creating it from spectral decomposition The definition of the transposed structure is shown below.

[0110] Here, SVL （Ｋ） , A.C.M. （Ｋ） （Ｒ＋１） , INV （Ｒ） （Ｒ） are obtained by the following steps 3-1 to 3-3. Step 3-1: SVL （Ｋ） is the value part of the spectral decomposition in Eq. (3), λ ０ , λ １ , ..., λ Ｋ－１ The SVL obtained by equation (4) is （Ｋ） is the same as

[0111] Step 3-2: ACM （Ｋ） （Ｒ＋１） teeth,

[0112] and the value λ in equation (3) ｉ The number of occurrences of ｉ is accumulated and converted into mapping A. This step takes O(K) time.

[0113] Step 3-3: INV （Ｒ） （Ｒ） teeth,

[0114] and in equation (3)

[0115] Here, x ｉ，ｊ is λ ｉ is the j-th record number from the top among the record numbers in which

[0116] is λ ０ <λ １ <...<λ Ｋ－１ Since the record numbers correspond to the inverted record numbers, INV is an array of inverted record numbers.

[0117] For example, in the example shown in FIG. 4, the transposed structure is <(Alice, Bob, Cathy): (1, 3, 4), (1, 0, 3, 2)>.

[0118] Spectral decomposition is not an A-map, but when trying to get the i-th result of the sorted sequence, ｉ The transposed structure derived from the spectral decomposition is a structure consisting of three A maps, which can solve the first problem, and the second problem can also be solved because the i-th item in the sorted result sequence can be obtained with INV[i].

[0119] Method for obtaining spectral decomposition from transposed structure The spectral decomposition can be obtained from the transposed structure in D5A in the following manner.

[0120] First, in the spectral decomposition of Equation (3), λ ｉ is λ ｉ =SVL （Ｋ） [i]. Also, if you decide in advance that ACM[-1]≡0, then

[0121] is obtained as follows:

[0122] From the above two, the i-th term of the spectral decomposition can be obtained as follows:

[0123] For example, in the example shown in FIG. 4, the first term of the spectral decomposition is calculated as follows:

[0124] Information Obtained from a Transposed Structure When a transposed structure is used, in addition to the information obtained from SVL (information 1 to information 4 above), the following information 5 to information 8 can also be obtained.

[0125] Information 5: The number of times the ith value appears from the smallest can be found as follows:

[0126] count = ACM （Ｋ） [i]-ACM （Ｋ） [i-1] Equation (8) For example, in the example shown in FIG. 4, the number of occurrences of the smallest value is ACM （Ｋ） [1] -ACM （Ｋ） [0] = 3 - 1 = 2.

[0127] Information 6: The i-th record number of the sorted result can be read as follows:

[0128] i-th record number of sorted result=INV[i] Equation (9) For example, in the example shown in FIG. 4, the first record number of the sorted result can be read as INV[1]=0 (Bob).

[0129] Information 7: The record numbers having the ith smallest value can be read out as the following array.

[0130] (INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i-1]],...,INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i]-1]) For example, in the example shown in FIG. 4, the record number having the smallest value (Bob) can be read as (INV[1], . . . , INV[2]) = (0, 3).

[0131] Information 8: The record numbers with values ​​v0 to v1 can be read as follows:

[0132] i ０ v0≦SVL[i ０ ] is the smallest i ０ , i １ SVL[i １ ]≦i １ The largest i that satisfies １ As, (INV （Ｒ） （Ｒ）[A.C.M. （Ｋ） [i ０ -1]],...,INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i １ ]-1]) Equation (10) For example, in the example shown in FIG. 4, the record numbers having values ​​Bob to Cathy are i ０ = 1, i １ = 2, so they can be read as (INV[ACM[1-1]], ..., INV[ACM[2]-1]) = (INV[1], ..., INV[4-1]) = (0, 3, 2).

[0133] - An index can be created using a transposed structure As mentioned above, the requirement for an index to be valid is that the size W of the sorted, searched, and aggregated results is known, and when i∈0, 1, ..., W-1 is specified, the i-th row of the sorted, searched, and aggregated results can be read in approximately O(log(R)) or less. Below, we will verify that an index can be created using a transposed structure.

[0134] (Sorting) It can be seen that the size of the sorted result is R. The i-th element of the sorted result can be read in O(1) using equation (9). Therefore, the transposed structure satisfies the requirements for creating an index for sorting.

[0135] (Search) The case of formula (10) will be explained. The size of the search result is ACM （Ｋ） [i １ ]-ACM （Ｋ） [i ０ For example, in the example shown in FIG. 4, the record numbers having values ​​Bob to Cathy are i ０ = 1, i １ = 2, so ACM （Ｋ） [2] -ACM （Ｋ） We can see that [1-1] = 4-1 = 3.

[0136] The i-th element of the search result is (INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i ０ -1]],...,INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i１ ]-1])[i], it can be read in O(1). For example, in the example shown in FIG. 4, when searching for values ​​Bob to Cathy, i ０ = 1, i １ = 2, so (INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [1-1]],...,INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [2]-1])[i] = (0,3,2)[i].

[0137] Therefore, the inverted structure meets the requirements for creating an index for searching.

[0138] (Aggregation) The size of the aggregation result is known to be K from information 1 above. The i-th value of the aggregation result is given by information 2 above and can be obtained in O(1). The number of occurrences of the i-th value of the aggregation result is given by equation (8) and can be obtained in O(1). Therefore, the transposed structure satisfies the requirements for creating an index for aggregation.

[0139] For example, in the example shown in Figure 4, when i = 1, the size of the aggregation result is 3, the first value of the aggregation result is Bob, and the number of times the first value of the aggregation result appears is ACM[1] - ACM[1-1] = 3 - 1 = 2.

[0140] From the above, it can be said that a D5A index can be created that uses an inverted structure to speed up sorting, searching, and aggregation.

[0141] - Only a small amount of storage space is required to store the results of sorting, searching, and aggregation using the D5A index (sort results). The sort results are already stored in INV, so no new storage space is required.

[0142] (Search Results) The case of formula (10) will be explained. The search results are the array (INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i ０ -1]],...,INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i １]-1]). Of these, INV and ACM have already been created, and the memory area required to hold the search results is i ０ , i １ It is just a storage area.

[0143] (Counting Result) The i-th value of the counting result is given by the above information 2, and does not require a new storage area. The number of occurrences of the i-th value of the counting result is given by equation (8), and does not require a new storage area.

[0144] As described above, D5A has the role of supplying values ​​to virtual table data and indexes to virtual indexes. By using the SVL and NNC obtained by SN decomposition of a column, it is possible to not only supply values ​​but also obtain the four types of information, information 1 to information 4, listed above. We also demonstrated that a D5A index can be created using a transposed structure (that is, a transposed structure can be used as a data structure for indexes in sorting, searching, and aggregation). A D5A index not only speeds up sorting, searching, and aggregation, but can also store the results of sorting, searching, and aggregation in a small amount of memory space.

[0145] <Virtual tabular data> For example, when a JOIN or UNION is performed in an RDB (relational database), the source tabular data is read, compared, the data to be written is created, its storage location is determined, and the data is written to a newly allocated storage area. This requires time and storage space. This is the first step in trying to utilize archived tabular data. The larger the newly generated tabular data, the more serious this problem becomes. This is because it takes a long time to generate and requires a large storage area. Moreover, often only a small portion of the generated tabular data is used.

[0146] Virtual tabular data solves this problem by creating only the necessary parts when they are needed. In other words, when a cell needs to be displayed, the corresponding cell in the source tabular data is referenced at that time. However, this requires a predefined mapping that defines the correspondence between the cells in the virtual tabular data and the cells in one or more source tabular data. This mapping can be from virtual tabular data to multiple source tabular data, or conversely, from multiple source tabular data to virtual tabular data. Both are essentially the same. However, while the former requires a pair of pieces of information—a cell in a given source tabular data—the destination of the mapping, the latter requires only a single piece of information—a cell in the virtual tabular data—and is therefore easier to define. Therefore, the following explanation focuses on the latter. Note that both require an inverse mapping. The former requires a virtual index, while the latter is required when the virtual tabular data references the source tabular data to obtain a value.

[0147] The mapping that determines which cell in each source tabular data corresponds to which cell in the virtual tabular data is called an allocation mapping. However, an actual allocation mapping defines a mapping from each source column, which is a column in the source tabular data, to a virtual column, which is a column in the virtual tabular data. Each source column is divided into one or more intervals, and for each interval, cells in that interval are mapped to cells in the virtual column using a mapping table or rules. The mapping for each interval is called an interval mapping. An allocation mapping is the sum of all interval mappings.

[0148] <<Procedure for Creating an Allocation Mapping>> The procedure for creating an allocation mapping will be described with reference to Fig. 5. Note that hereinafter, rs indicates the record number in the source tabular data, and rv indicates the record number in the virtual tabular data.

[0149] A virtual column, which is a column in virtual tabular data, is composed of allocation mappings from one or more source columns. In the example shown in Figure 5, the virtual column is allocated from source column 0 and source column 1.

[0150] The blank D5A with source column 1 in the lower left of Figure 5 is a D5A that is automatically generated programmatically when a virtual column is generated. Blank D5A supplies blank values ​​to unassigned cells in the virtual column, ensuring that the virtual column satisfies the bijection condition described below. If the bijection condition is satisfied, the virtual tabular data is automatically provided with a virtual index. However, there are cases where sorting, searching, or aggregation of virtual tabular data is not necessary. For example, this may be the case when you want to use the entire result of a UNION or JOIN as is. In such cases, you can of course define and use virtual tabular data without considering the bijection condition.

[0151] Each source column is divided into one or more source intervals. In the example shown in FIG. 5, source column 0 is divided into interval 1 and interval 2. There are two types of mappings from each source interval to a virtual column (i.e., interval mappings). The first is an enumeration mapping defined by an A-mapping M whose elements are natural numbers and have no duplicates. The second is a linear function mapping defined by a linear function. The former enumeration mapping can define any allocation mapping but consumes a large amount of memory space. The latter linear function mapping is a mapping in which M in the enumeration mapping is replaced by a linear function. It consumes little memory space and can define even large interval mappings, but it can only support regular allocations. Once interval mappings have been defined for all source intervals, the allocation mapping is complete.

[0152] Once any of the above interval mappings is defined, its inverse mapping and the conditions for its existence are also determined. When displaying virtual tabular data, first identify the source interval where the inverse mapping exists from the record number of the virtual column, then identify the record number on the source column using the inverse mapping of the interval mapping, and then obtain and display the value from that record number.

[0153] Below, interval mapping using enumeration type mapping and interval mapping using linear function type mapping will be explained, and then a method for displaying virtual tabular data will be explained with reference to FIG. 5 again.

[0154] <Interval Mapping by Enumeration Mapping and Its Inverse Mapping> Fig. 6 shows an example of allocation by enumeration mapping. In Fig. 6, Rs represents the size of the source column, rs represents the record number of the source column, Rv represents the size of the virtual column, rv represents the record number of the virtual column, Q represents the interval on the source column, q represents the start position of interval Q, u represents the length of interval Q, V represents the interval on the virtual column corresponding to interval Q, v represents the start position of interval V, and w represents the length of interval V.

[0155] In this case, the interval mapping F:Q→V by the enumeration mapping is expressed as follows:

[0156] rv=M （ｕ） （Ｒｖ） [rs-q] Equation (11) Also, its inverse mapping F －１ :V→Q is expressed as follows.

[0157] M （ｕ） （Ｒｖ） =L （ｕ） （Ｒｖ） ・P （ｕ） （ｕ） ; L （ｕ） （Ｒｖ） [j]=rv; rs=P －１ （ｕ） （ｕ） [j] + q Equation (12) Enumeration Mapping First, we will explain equation (11), which defines interval mapping using enumeration mapping. In Figure 6, the numbers written next to the source column and virtual column are record numbers. M can be defined by writing out the record numbers on the corresponding virtual columns in order from the beginning of the source interval. In the example shown in Figure 6, it is (7, 5, 9, 1). Here, the starting position q of source interval Q is 2.

[0158] The source section (section Q) starts from rs = 2 and ends at rs = 5. Applying each rs to equation (11) gives the following:

[0159] When rs = 2, rv = (7, 5, 9, 1) [2 - 2] = 7 When rs = 3, rv = (7, 5, 9, 1) [3 - 2] = 5 When rs = 4, rv = (7, 5, 9, 1) [4 - 2] = 9 When rs = 5, rv = (7, 5, 9, 1) [5 - 2] = 1 Inverse mapping of enumeration mapping Next, we will explain equation (12), which defines the inverse mapping of enumeration mapping.

[0160] M is an A-map whose elements are natural numbers with no duplicates, and can be subjected to LP decomposition. The top row of equation (12) shows the LP decomposition. By LP decomposition, we obtain L = (1, 5, 7, 9) and P = (2, 1, 3, 0). L is a unique ascending-order array, and the presence or absence of an element, and if there is an element, its position can be found by bisection search. P is a symmetric array, and the inverse element P －１ = (3, 1, 0, 2).

[0161] The middle part of equation (12) shows how to find the position of rv in L, where j is its occurrence position. If j cannot be found, it means that rv has not accepted the mapping from the source interval Q, and there is no inverse mapping.

[0162] The lower part of equation (12) is P －１ This means that rs can be found using = (3, 1, 0, 2), q = 2, and the above j.

[0163] Based on the above, rs is calculated for the cases where rv=1, 5, 7, and 9.

[0164] When rv = 1, (1, 5, 7, 9) [j] = 1 to j = 0; rs = (3, 1, 0, 2) [j] + 2 = 5 When rv = 5, (1, 5, 7, 9) [j] = 5 to j = 1; rs = (3, 1, 0, 2) [j] + 2 = 3 When rv = 7, (1, 5, 7, 9) [j] = 7 to j = 2; rs = (3, 1, 0, 2) [j] + 2 = 2 When rv = 9, (1, 5, 7, 9) [j] = 9 to j = 3; rs = (3, 1, 0, 2) [j] + 2 = 4 <Interval Mapping Using Linear Function Mapping and Its Inverse Mapping> Figure 7 shows an example of allocation using linear function mapping. In Figure 7, Rs represents the size of the source column, rs represents the record number of the source column, Rv represents the size of the virtual column, rv represents the record number of the virtual column, Q represents an interval on the source column, q represents the start position of interval Q, u represents the length of interval Q, V represents an interval on the virtual column corresponding to interval Q, v represents the start position of interval V, and w represents the length of interval V.

[0165] In this case, the interval mapping F:Q→V by the linear function mapping is expressed as follows:

[0166] rv = a × rs + b; a and b are integers, a ≠ 0 Equation (13) Also, its inverse mapping F －１ :V→Q is expressed as follows.

[0167] rs = (rv - b) / a Equation (14) However, when v ≤ rv < v + w is not satisfied, and when rs is not an integer, rv is not the range of F.

[0168] Linear Function Mapping First, we explain equation (13), which defines the linear function mapping. In Figure 7, the numbers written next to the source column and virtual column are record numbers. a is the interval on the virtual column; in the example shown in Figure 7, a = 3 and b = -5.

[0169] The source section (section Q) starts from rs = 2 and ends at rs = 5. Applying each rs to equation (13) gives the following:

[0170] When rs = 2, rv = 3 × rs - 5 = 1 When rs = 3, rv = 3 × rs - 5 = 4 When rs = 4, rv = 3 × rs - 5 = 7 When rs = 5, rv = 3 × rs - 5 = 10 Inverse mapping of a linear functional mapping Next, we will explain equation (14), which defines the inverse mapping of a linear functional mapping. As mentioned above, a = 3 and b = -5.

[0171] First, we need to confirm that v≦rv<v+w. Since v=1 and w=10, we must have 1≦rv<1+10. If this condition is met, we use the upper part of equation (14) to find rs. In this case, as in the lower part of equation (14), if rs is not an integer, it means that rv does not accept a mapping from the source interval Q, and no inverse mapping exists.

[0172] When rv = 1, rs = (1 - (-5)) / 3 = 2 When rv = 4, rs = (4 - (-5)) / 3 = 3 When rv = 7, rs = (7 - (-5)) / 3 = 4 When rv = 10, rs = (10 - (-5)) / 3 = 5 <Displaying Virtual Tabular Data> A method for displaying virtual tabular data will be explained with reference to Figure 5 again. Source interval 1 of source column 0 is interval mapped using a linear function type mapping, source interval 2 of source column 0 is interval mapped using an enumeration type mapping, and source interval 1 of source column 1 is interval mapped using a linear function type mapping. The definitions of each interval mapping are as follows:

[0173] The interval mapping of source column 0 in source interval 1 is defined as F:rv=rs×2+1 by equation (13).

[0174] The interval mapping for source interval 2 of source column 0 is defined as F: rv = (2, 7, 0) [rs - 3] by equation (11). This can be confirmed by substituting rs = 3, 4, 5 as follows:

[0175] When rs = 3, rv = (2, 7, 0) [3 - 3] = 2 When rs = 4, rv = (2, 7, 0) [4 - 3] = 7 When rs = 5, rv = (2, 7, 0) [5 - 3] = 0

[0176] The interval mapping in source interval 1 of source column 1 of blank D5A is defined as F:rv=rs×2+4 by equation (13).

[0177] Once each interval mapping F is defined, its inverse mapping F －１ is automatically determined as follows:

[0178] The inverse mapping of source column 0 to source section 1 is given by equation (14): －１ : rs = (rv-1) / 2.

[0179] The inverse mapping of source column 0 in source section 2 is given by F －１ : rs = (2,0,1) [j] + 3; rv = (0,2,7) [j]. First, when rv = 0, a bisection search is performed on (0,2,7) to determine the storage position j = 0 of rv, and rs = (2,0,1) [0] + 3 = 5. Similarly, when rv = 2, a bisection search is performed on (0,2,7) to determine the storage position j = 1 of rv, and rs = (2,0,1) [1] + 3 = 3. Similarly, when rv = 7, a bisection search is performed on (0,2,7) to determine the storage position j = 2 of rv, and rs = (2,0,1) [2] + 3 = 4.

[0180] The inverse mapping of source column 1 of blank D5A in source section 1 is F －１ :rs = (rv-4) / 2.

[0181] To summarize the above, when rv=0,1,2,3,5,7, we get rs=5,0,3,1,2,4 in source column 0, and when rv=4,6, we get rs=0,1 in source column 1.

[0182] <<Method of Configuring Virtual Tabular Data>> To realize the functions described above, the virtual tabular data is configured with the following information.

[0183] 1. Number of records 2. Name and data type of virtual column 3. URL or path of one or more source tabular data 4. Definition of source column and source interval in the above source tabular data 5. Definition of interval mapping and its inverse mapping for each source interval Items other than 5 above do not require large storage areas. Also, the need for large storage areas for 5 above is limited to when the source interval is large and the mapping is of an enumeration type. For this reason, virtual tabular data can often be configured compactly.

[0184] <Detection of collisions between allocations from different sections> Here, we will explain how to check whether there is a collision between allocation destinations by allocation mapping. ０ :Q ０ →V ０ , F １ :Q １ →V １ Let's say.

[0185] First, as a trivial case, when any of the following conditions (1-1) to (1-3) is satisfied, two allocation mappings F ０ , F １ do not collide.

[0186] (Condition 1-1) V ０ and V １ are not on the same virtual column (condition 1-2) v ０ +w ０ -1<v １ If (Condition 1-3) v １ +w １ -1<v ０ In the case where v ０ is V ０ The starting position of w ０ is V ０ The length of v １ is V １ The starting position of w １ is V １ is the length of

[0187] If the above trivial cases do not apply, two allocation mappings F ０ , F １ It is possible to check whether there is a collision.

[0188] (Case 1) F ０ , F １ If both are enumerated maps, then L ０ = (v ０ , ..., v ０ +w ０ -1), L １ = (v １ , ..., v １ +w １ -1), as L ０ and L １ The common range of L ０ , L １ In addition, w ０ is V ０ The length of w １ is V １ is the length of

[0189] Then, for i=0, j=0, the following steps 4-1 to 4-3 are executed.

[0190] Step 4-1: L ０ [i] = L １ If it is [j], it is determined that there is a collision and the process ends.

[0191] Step 4-2: L ０ [i] < L １ If it is [j], return to step 4-1 by setting i←i+1. However, if i is L ０ If the maximum value of the index representing the element of is exceeded, the process ends as "no collision".

[0192] Step 4-3: L ０ [i]>L １ If it is [j], then j←j+1 is set and the procedure returns to step 4-1. However, if j is L as a result of setting j←j+1, １ If the maximum value of the index representing the element of is exceeded, the process ends as "no collision".

[0193] (Case 2) F ０ , F １ If both are linear function maps, then V ０ and V １ Select the common part of F ０ , F １are rewritten as the following expressions with the common part as the range. ０ , a １ Adjust so that is positive.

[0194] y ０ = a ０ ×x ０ +b ０ ;x ０ =0,1,2,...,N ０ -1 y １ = a １ ×x １ +b １ ;x １ =0,1,2,...,N １ -1 where a ０ , a １ is an integer of 1 or more, b ０ , b １ is an integer.

[0195] And x ０ = 0, x １ = 0, and then execute the following steps 5-1 to 5-3.

[0196] Step 5-1: y ０ = y １ If so, the process ends with "a collision has occurred."

[0197] Step 5-2: y ０ <y １ If so, then x ０ ←x ０ +MaxInt(1, (y １ -y ０ ) div a ０ ) and return to step 5-1. ０ ←x ０ +MaxInt(1, (y １ -y ０ ) div a ０ ) and as a result, x ０ >MinInt(N ０ -1, LCM (a ０ , a １ ) div a ０ ) then it is considered "no collision" and the process ends.

[0198] Step 5-3: y ０ >y １If so, then x １ ←x １ +MaxInt(1, (y ０ -y １ ) div a １ ) and return to step 5-1. １ ←x １ +MaxInt(1, (y ０ -y １ ) div a １ ) and as a result, x １ >MinInt(N １ -1, LCM (a ０ , a １ ) div a １ ) then it is considered "no collision" and the process ends.

[0199] Here, MaxInt is a function for selecting the maximum integer, MinInt is a function for selecting the minimum integer, div is a function for dividing integers (the remainder is discarded), and LCM is a function for returning the least common multiple.

[0200] (Case 3) F ０ , F １ If one of the two is an enumeration type map and the other is a linear function type map, then F ０ is a linear function type map, F １ Let V be an enumerated mapping. ０ and V １ Select the common part of F ０ , F １ are rewritten as the following expressions with the common part as the range. ０ Adjust so that is positive.

[0201] y ０ (n) = a ０ ×n+b ０ ;x ０ =0,1,2,...,N-1 y １ (m) = (v １ , ..., v １ +w １ -1) [m] where a ０ >0. Also, (v １ , ..., v １ +w １ −1) is an ascending array of size M.

[0202] Then, n=0 and m=0 are set, and the following steps 6-1 to 6-3 are executed.

[0203] Step 6-1: y ０ (n) = y １ If it is (m), it is determined that there is a collision and the process ends.

[0204] Step 6-2: y ０ (n)>y １ If it is (m), then m←m+1 is set and the process returns to step 6-1. However, if m=M as a result of setting m←m+1, then "no collision" is determined and the process ends.

[0205] Step 6-3: y ０ (n) < y １ If (m), then n←(y １ (m)-y ０ (n)) div a ０ Then return to step 6-1. However, n←(y １ (m)-y ０ (n)) div a ０ If n≧N, the process ends with "no collision".

[0206] <Virtual Index> So far, we have explained how to create virtual tabular data that directly or indirectly inherits values ​​from D5A. Below, we will explain why a virtual inverted structure is automatically established on virtual tabular data by directly or indirectly inheriting the inverted structure on D5A. Once a virtual inverted structure is established, a virtual index that uses it as a data structure for the index is also automatically established. A virtual index simply uses a virtual inverted structure instead of an inverted structure, and uses the same algorithm as a D5A index, so it has the same functions and characteristics as a D5A index. In other words, it can speed up column sorting, searching, and aggregation, and can store even large sort results, search results, and aggregation results in a small amount of memory space.

[0207] Similar to the transposed structure, the virtual transposed structure is a structure that has a virtual SVL, virtual ACM, and virtual INV. Here, the virtual SVL, virtual ACM, and virtual INV are virtual arrays. A virtual array is a mechanism with functionality equivalent to an array, in that it does not hold values ​​like virtual tabular data, but its size W is known in advance, and when i∈0, 1, ..., W-1 is specified, the i-th element can be extracted in approximately O(log(R)) or less. The virtual SVL, virtual ACM, and virtual INV are automatically established if the allocation mapping that defines the virtual tabular data satisfies the following bijective conditions:

[0208] ・Conditions for bijection 1. (Mapping is valid) All cells in the source column are mapped to cells in the virtual column.

[0209] 2. (Injective) No cell on any virtual column is mirrored from cells in more than one source column.

[0210] 3. (Surjection) All cells on the virtual column are mapped from cells in the source column.

[0211] 4. (The source column is bijective) If the source column is D5A, it can be said that the source column is bijective. If the source column is a virtual column, it can be said that the source column is bijective if it satisfies 1 to 3 above.

[0212] In the following, we first explain that a virtual inverted structure can be constructed from multiple inverted structures, and then verify that a virtual index using this virtual inverted structure can perform sorting, searching, and aggregation quickly, and that sorting results, search results, and aggregation results can be stored by allocating only a small amount of new memory space. Next, we verify that further virtual inverted structures can be created hierarchically using virtual inverted structures and inverted structures, and that a virtual index using this can also perform sorting, searching, and aggregation quickly, and that sorting results, search results, and aggregation results can be stored by allocating only a small amount of new memory space. Finally, we verify that the use of spectral decomposition makes it easy to understand the operation of a hierarchically created virtual inverted structure.

[0213] <<Method of constructing a virtual transposed structure>> As an example, source column #0: C ０（４） and Source Column #1: C １（４） A virtual column C defined from Ｖ（８） is shown in Figure 8. ０（４） and C １（４） For the sake of distinction, the elements of A ０ , B ０ , C ０ , B １ , C １ , D １ The source column number is displayed as a subscript, but the value is B ０ =B １ , C ０ =C １ In this case, the allocation destinations from source column #0 are the 5th, 7th, 2nd, and 3rd rows of the virtual column, so the allocation mapping F ０ Similarly, the allocation destination from source column #1 is rows 4, 0, 6, and 1 of the virtual column, so the allocation mapping F １ = (4, 0, 6, 1).

[0214] The transposed structure of source column #0, the transposed structure of source column #1, and the virtual transposed structure are shown in FIG. 9. Here, SVL, ACM, and INV in the transposed structure of source column #0 are respectively ０ , A.C.M. ０ , INV ０ and SVL, ACM, and INV in the transposed structure of source column #1 are denoted as SVL, ACM, and INV, respectively. １ , A.C.M. １ , INV １ In addition, SVL, ACM, and INV in the virtual inverted structure of the virtual column are denoted as SVL, ACM, and INV, respectively. Ｖ , A.C.M. Ｖ , INV Ｖ It is written as follows.

[0215] INV ０ and INV １ are both arrays that store transposed record numbers, and the record numbers in the source column are mapped to the record numbers in the virtual column by allocation mapping. ０ and INV １ When reading out, INV０ ' and INV １ It needs to be read as 'INV'. ０ ', INV １ ' can be calculated using the following formula (15).

[0216] INV ０ '=F ０ ・INV ０ =(5,7,2,3)・(3,0,2,1)=(3,5,2,7) Formula (15)-1 INV １ '=F １ ・INV １ = (4, 0, 6, 1) · (1, 2, 0, 3) = (0, 6, 4, 1) Equation (15)-2 Hereinafter, using FIG. 9 as an example, the method for constructing the virtual transposed structure will be explained using the virtual array SVL. Ｖ , A.C.M. Ｖ , INV Ｖ The following will be explained in order.

[0217] Virtual array SVL Ｖ First, the SVL Ｖ How to calculate the size of SVL ０ and SVL １ The values ​​B and C that appear in common in the SVL are determined to be larger or smaller by taking into account the order of the source columns. ０ B ０ is SVL １ B １ Then, there will be no values ​​that are not in a relationship of magnitude between each SVL, and Ｖ The size is SVL ０ Size = 3 and SVL １ The sum of the sizes of 3 and 6 is 6, which is uniquely determined.

[0218] Next, SVL Ｖ The i-th element of SVL Ｖ The method for finding [i] is explained below. ０ or SVL １ An appropriate value v is selected from the above. However, this v follows the above rule that determines the magnitude relationship taking into account the order of the source columns. Next, SVL ０ and SVL １Find the sum j of the number of values ​​smaller than v. If j<i, select v' larger than v and repeat the above. If j>i, do the same (but select v' smaller than v). If j=i, then v is the SVL Ｖ [i].

[0219] The above SVL Ｖ The method for finding [i] can be done in approximately O(log(K)). Ｖ Since the size of is known and the i-th element can be extracted in O(log(R)) or less, the virtual array SVL Ｖ It is reasonable to assume that exists.

[0220] Here is an example when i = 0. First, let v be the SVL １ B １ Try selecting SVL. ０ There are two values ​​smaller than v, １ There is 0 in it. Therefore, j = 2 + 0 = 2, which is greater than i = 0. Next, let v be the SVL ０ A ０ Then, j = 0 and j = i. Therefore, SVL Ｖ [0] = A ０ This becomes:

[0221] Here is an example when i=4. First, let v be the SVL ０ B ０ Try selecting SVL. ０ There is one value smaller than v in the SVL １ There is 0 in it. Therefore, j = 1 + 0 = 1, which is smaller than i = 4. Next, let v be SVL. １ C １ Then, j = 3 + 1 = 4, and j = i. Therefore, SVL Ｖ [4] = C １ This becomes:

[0222] From the above, the virtual array SVL Ｖ = (A ０ , B ０ , B １ , C ０ , C １ , D １) can be created.

[0223] ・Virtual array ACM Ｖ First, ACM Ｖ The size is SVL Ｖ It is calculated as the size of

[0224] Next, ACM Ｖ The i-th element of ACM Ｖ The method for finding [i] will be explained. First, Ｖ By the method explained in "Construction method of v = SVL Ｖ Find v where [i]. Next, SVL ０ [i ０ ]≦v ０ Find i ０ If not found, ０ = -1. Note that ACM[-1]≡0.

[0225] Similarly, SVL １ [i １ ]≦v １ Then, ACM Ｖ [i] is ACM Ｖ [i] = ACM ０ [i ０ ]+ACM １ [i １ ] can be calculated as follows.

[0226] The above ACM Ｖ The method for finding [i] can be done in approximately O(log(K)). Then, ACM Ｖ The size of is known, and its i-th element can be extracted in O(log(R)) or less. Ｖ It is reasonable to assume that exists.

[0227] Here is an example when i = 3. First, let v be the SVL １ B １ Try selecting SVL. ０ There are two values ​​smaller than v, １ There is 0 in the list. Therefore, j=2, which is smaller than i=3. Next, let v be the SVL. ０ C ０Then, j = 2 + 1 = 3, and SVL Ｖ [3] = C ０ This means that...

[0228] Next, SVL ０ [i ０ ]≦v ０ When we calculate i ０ = 2 is obtained. Next, SVL １ [i １ ]≦v １ When we calculate i １ = 0 is obtained. Therefore, ACM Ｖ [3] is ACM Ｖ [3] = ACM ０ [2] + ACM １ This can be calculated as [0] = 4 + 2 = 6.

[0229] From the above, the virtual array ACM Ｖ =(1, 3, 5, 6, 7, 8) can be created.

[0230] ・Virtual array INV Ｖ First, INV Ｖ The size of is known to be R, where R is the sum of the sizes of the source columns.

[0231] Next, INV Ｖ To find [i], ACM Ｖ [j-1]≦i≦(ACM Ｖ [j]-1) is the ACM. Ｖ INV as defined by Ｖ The number of the upper section.

[0232] ACM Ｖ The size of K Ｖ Let's say. Ｖ ACM ０ and ACM １ First, the sum of the sizes of 0, 1, ..., K Ｖ -1, select an appropriate j, and Ｖ Section ACM Ｖ [j-1] to ACM Ｖ Find [j]-1.

[0233] If i<ACM Ｖ If it is [j-1], then j is made a smaller value and the interval is selected again. On the other hand, if i>ACM Ｖ If [j] -1, set j to a larger value and reselect the interval. Repeat this process to determine j, and once j is determined, set offset = i - ACM Ｖ Find [j-1] in advance.

[0234] Next, v = SVL Ｖ Then, the value v and its source column are found. Using v, find k such that SVL[k] = v in the SVL on that source column. This gives INV Ｖ INV[i]=INV'[ACM[k-1]+offset] Ｖ [i] is obtained, where INV' and ACM are the INV' and ACM on the source column.

[0235] The above INV Ｖ The method for finding [i] can be done in approximately O(log(K)). Ｖ Since the size of is known and the i-th element can be extracted efficiently, the virtual array INV Ｖ It is reasonable to assume that exists.

[0236] Here is an example when i = 4. First, let's select j = 3. The lower limit of the interval is ACM. Ｖ [j-1] = ACM Ｖ [2] = 5, upper limit of the interval is ACM Ｖ [j]-1 = ACM Ｖ [3] - 1 = 5, so we can see that the selected interval is too large. So, let's try selecting j = 2 again. The lower limit of the interval is ACM Ｖ [j-1] = ACM Ｖ [1] = 3, upper limit of the interval is ACM Ｖ [j]-1 = ACM Ｖ [2]-1=4. Therefore, it can be seen that i=4 belongs to the interval j=2. offset=i-ACM Ｖ [j-1]=4-ACM Ｖ [1] = 1. Next, v = SVL Ｖ [j] = SVL Ｖ [2] = B １Next, SVL[k]=B on source column #1 １ It can be seen that k is 0. Ｖ We obtain [4] = INV'[ACM[k-1] + offset] = INV'[1] = 6.

[0237] From the above, the virtual array INV Ｖ =(3, 5, 2, 0, 6, 7, 4, 1) can be created.

[0238] Sorting with a virtual index We will try sorting with a virtual index using a virtual inverted structure. Since NNC is not shown in Figures 8 and 9, we will use C in Figure 8 instead. ｖ（８） The sorting result is as follows:

[0239] C ｖ（８） ・INV Ｖ = (B １ , D １ , B ０ , A ０ , C １ , B ０ , B １ , C ０ )・(3,5,2,0,6,7,4,1)=(A ０ , B ０ , B ０ , B １ , B １ , C ０ , C １ , D １ ) From the above results, we can see that the results are indeed sorted. In addition, even if the same "B" or "C" appears in the sorted results, ０（４） "B" and "C" in １（４） This is because the smaller the column number, the smaller the value when the values ​​are the same. Also, just like the D5A index, no additional storage space is required to store the sorted results.

[0240] Searching with a virtual index Searching with a virtual index using a virtual inverted structure is performed. Since NNC is not shown in Figures 8 and 9, instead, C in Figure 8 is used. ｖ（８）The search will be performed under conditions "B" to "C".

[0241] In equation (10), i ０ = 1, i １ = 4, so (INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i ０ -1]],...,INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i １ ]-1]) = (5, 2, 0, 6, 7, 4). Therefore, the search results are as follows:

[0242] C ｖ（８） ・(5,2,0,6,7,4)=(B １ , D １ , B ０ , A ０ , C １ , B ０ , B １ , C ０ )・(5,2,0,6,7,4)=(B ０ , B ０ , B １ , B １ , C ０ , C １ ) Also, to retain search results, ０ , i １ Since it is only necessary to store the above, it requires only a small storage area, just like the D5A index.

[0243] ・Counting using a virtual index Let's try counting using a virtual index that uses a virtual transposed structure. The i-th value of the counting result, "value," is given by information 2 above, and the number of occurrences of the i-th value in the counting result, "count," is given by equation (8). When counting is performed according to this, the results are as shown in Table 1 below.

[0244] Here, even if the value is the same, if the D5A that provided the value is different, the aggregation results will be separated. For this reason, users who prefer this can leave it as is, while users who do not need to add up the number of occurrences for the same value. Since the summation will not exceed the maximum number of D5As, the processing time required for this is not large. Also, like a D5A index, no additional storage space is required to hold the aggregation results.

[0245] As mentioned above, we can see that there are virtual SVL, virtual ACM, and virtual INV. Therefore, we can say that there is a virtual inverted structure. The inverted structure is a data structure for the D5A index, and for the same reason, we can say that the virtual inverted structure is a data structure for the virtual index.

[0246] Furthermore, the virtual transposed structure is composed of references to the transposed structure of the source column, but if the source column is a virtual column, the transposed structure is also a virtual transposed structure, so in that case, the virtual transposed structure is constructed hierarchically.

[0247] <<Virtual index of virtual tabular data created hierarchically>> It has been mentioned that a virtual inverted structure can be created hierarchically, and this will be verified below using an example. Figure 10 shows a C5A file. ０（３） and C １（２） Virtual tabular data C Ｖ０（５） and create the virtual tabular data C Ｖ０（５） and D5A file C ２（３） Further virtual tabular data C Ｖ１（８） The following shows an example of creating an allocation map.

[0248] F (C ０（３） →C Ｖ０（５） )=(2,3,0) F(C １（２） →C Ｖ０（５） )=(4,1) F(C Ｖ０（５） →C Ｖ１（８） )=(2,3,7,1,5) F(C ２（３） →C Ｖ１（８） ) = (0, 6, 4) At this time, the transposed structure of FIG. 10 is shown in FIG. 11. ０（３） , C１（２） , C Ｖ０（５） , C ２（３） The transposed structure or virtual transposed structure of is as shown in the figure. Each INV can be converted to INV' as follows, similar to equation (15).

[0249] INV ０ '=F(C ０（３） →C Ｖ０（５） ) INV ０ =(2,3,0)・(1,0,2)=(3,2,0) INV １ '=F(C １（２） →C Ｖ０（５） ) INV １ =(4,1)・(0,1)=(4,1) INV Ｖ０ '=F(C Ｖ０（５） →C Ｖ１（８） ) INV Ｖ０ =(2,3,7,1,5)・(3,4,2,0,1)=(1,5,7,2,3) INV ２ '=F(C ２（３） →C Ｖ１（８） ) INV ２ = (0,6,4) · (2,1,0) = (4,6,0) Using the SVL, ACM, and INV' thus obtained, Ｖ０（５） , C Ｖ１（８） A virtual transposed structure can be constructed.

[0250] Here, we will use virtual indexes to sort, search, and aggregate the hierarchically created virtual tabular data.

[0251] - Sorting using a virtual index of hierarchically created virtual table format data We will try sorting using a virtual index that uses a hierarchically created virtual inverted structure. Since NNC is not shown in Figure 10 and Figure 11, we will instead use C in Figure 10. Ｖ１（８） The sorting result is as follows:

[0252] C Ｖ１（８） ・INV Ｖ１ = (D ２ , A ０ , B ０ , C １ , A ２ , A １ , B ２, B ０ )・(1,5,4,7,2,6,3,0)=(A ０ , A １ , A ２ , B ０ , B ０ , B ２ , C １ , D ２ ) From the above results, we can see that the results are indeed sorted. In addition, even if the same "A", "B" or "C" appears in the sorted results, ０（３） "A" and "B" are C １（２） , C ２（３） This is because the smaller the column number, the smaller the value when the values ​​are the same. Also, just like the D5A index, no additional storage space is required to store the sorted results.

[0253] Search using a virtual index of hierarchically created virtual table format data Let's try searching using a virtual index that uses a hierarchically created virtual inverted structure. Since NNC is not shown in Figure 10 and Figure 11, instead, we will use C in Figure 10. Ｖ１（８） The search will be performed under conditions "B" to "C".

[0254] In equation (8), i ０ = 3, i １ = 5, so (INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i ０ -1]],...,INV （Ｒ） （Ｒ） [A.C.M. （Ｋ） [i １ ]-1]) = (7, 2, 6, 3). Therefore, the search results are as follows:

[0255] C Ｖ１（８） ・(7,2,6,3)=(D ２ , A ０ , B ０ , C １ , A ２ , A １ , B ２ , B ０ )・(7,2,6,3)=(B ０ , B０ , B ２ , C １ ) Also, to retain search results, ０ , i １ Since it is only necessary to store the above, it requires only a small storage area, just like the D5A index.

[0256] - Aggregation using a virtual index for hierarchically organized virtual tabular data Let's try aggregation using a virtual index that uses a hierarchically organized virtual inverted structure. The i-th value of the aggregation result, "value," is given by information 2 above, and the number of occurrences of the i-th value in the aggregation result, "count," is given by equation (8). When aggregation is performed according to this, the result is as shown in Table 2 below.

[0257] Here, even if the value is the same, if the D5A that provided the value is different, the aggregation results will be separated. For this reason, users who prefer this will leave it as is, while users who do not will need to add up the number of occurrences for the same value. Even in this case, the maximum number of D5As will not be exceeded, so the processing time required for this is not large. Also, like a D5A index, no additional storage space is required to hold the aggregation results.

[0258] <Understanding virtual indexes through spectral decomposition> Figure 11C ０（３） , C １（２） →C Ｖ０（５） and C Ｖ０（５） , C ２（３） →C Ｖ１（８） It is easy to understand if we interpret it from the viewpoint of spectral decomposition. For ease of understanding, all the following equations are written in a form after INV is converted to INV' using an allocation map.

[0259] C ０（３） and C １（２） is expressed in spectral decomposition as follows:

[0260] C ０（３） = A ０ : (3) + B ０ : (2, 0) C １（２） = A １ :(4)+C １ (1) For this reason, C Ｖ０（５）can be expressed as follows:

[0261] C Ｖ０（５） =C ０（３） +C １（２） = A ０ :(3)+A １ : (4) + B ０ : (2, 0) + C １ (1) Also, C Ｖ０（５） and C ２（３） is expressed in spectral decomposition as follows:

[0262] C Ｖ０（５） = A ０ :(1)+A １ : (5) + B ０ : (7, 2) + C １ (3) C ２（３） = A ２ : (4) + B ２ : (6) + D ２ : (0) Therefore, C Ｖ１（８） can be expressed as follows:

[0263] C Ｖ１（８） =C Ｖ０（５） +C ２（３） = A ０ :(1)+A １ : (5) + A ２ : (4) + B ０ : (7, 2) + B ２ :(6)+C １ (3) +D ２ : (0) Therefore, the sorted result sequence is (1, 5, 4, 7, 2, 6, 3, 0). The search result sequence for "B" to "C" is (7, 2, 6, 3). The results can also be tallied in the same way as in Table 3 above. For this reason, it can be seen that calculations using spectral decomposition are easy to understand and superior when doing calculations by hand.

[0264] <Example of Overall Configuration of a System Including Data Processing Device 10> An example of the overall configuration of a system including the data processing device 10 according to this embodiment is shown in Fig. 12. As shown in Fig. 12, the data processing device 10 according to this embodiment is communicably connected to database servers that are distributed over a network 20 such as the Internet. These database servers store at least one of D5A and virtual tabular data.

[0265] The data processing device 10 according to this embodiment includes a virtual tabular data creation unit 101, a sorting unit 102, a search unit 103, a counting unit 104, and a storage unit 105. The virtual tabular data creation unit 101, the sorting unit 102, the search unit 103, and the counting unit 104 are realized, for example, by processing executed by a processor such as a CPU (Central Processing Unit) of one or more programs installed in the data processing device 10. The storage unit 105 is realized, for example, by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like.

[0266] The virtual tabular data creation unit 101 creates virtual tabular data having information 1 to 5 described above in the "Virtual Tabular Data Configuration Method." When defining the interval mapping 5 described above in the "Virtual Tabular Data Configuration Method," the virtual tabular data creation unit 101 determines whether or not there is a collision in the allocation destinations by the interval mapping (allocation mapping) by the method described above in the "Collision Detection of Allocations from Different Intervals."

[0267] The sort unit 102 executes sorting with a virtual index using the virtual inverted structure of the virtual tabular data described above. For example, the sort unit 102 uses a virtual column to be sorted of the virtual tabular data and a virtual INV included in the virtual inverted structure of the virtual column to calculate a sort result by index operation on those.

[0268] Upon receiving the search conditions, the search unit 103 executes a search using a virtual index that uses the virtual inverted structure of the virtual tabular data described above. For example, the search unit 103 calculates formulas (8) and (10) using a virtual column to be searched in the virtual tabular data, the search conditions for the virtual column, and the virtual INV and virtual ACM included in the virtual inverted structure of the virtual column, and then calculates the search result by index operation between the virtual column and the calculation results of formulas (8) and (10).

[0269] The aggregation unit 104 performs aggregation using a virtual index that uses the virtual inverted structure of the virtual tabular data described above. For example, the aggregation unit 104 performs aggregation using the virtual inverted structure of the virtual column to be aggregated in the virtual tabular data, based on the information 2 and formula (8).

[0270] The storage unit 105 stores various data (for example, virtual table format data, D5A, sorting results, search results, aggregation results, etc.).

[0271] <Example of Hardware Configuration of Data Processing Device 10> An example of the hardware configuration of the data processing device 10 according to this embodiment is shown in Fig. 13. As shown in Fig. 13, the data processing device 10 according to this embodiment includes an input device 201, a display device 202, an external I / F 203, a communication I / F 204, a RAM (Random Access Memory) 205, a ROM (Read Only Memory) 206, an auxiliary storage device 207, and a processor 208. Each of these pieces of hardware is connected to each other via a bus 209 so as to be able to communicate with each other.

[0272] The input device 201 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 202 is, for example, a display, a display panel, etc. Note that the data processing device 10 does not necessarily have to have at least one of the input device 201 and the display device 202, for example.

[0273] The external I / F 203 is an interface with an external device such as a recording medium 203 a. Examples of the recording medium 203 a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0274] The communication I / F 204 is an interface for connecting the data processing device 10 to the network 20. The RAM 205 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 206 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 207 is a non-volatile storage device such as an HDD, SSD, or flash memory. The processor 208 is, for example, a CPU or other type of computing device.

[0275] 13 is an example, and the data processing device 10 may have other hardware configurations. For example, the data processing device 10 may have multiple auxiliary storage devices 207 or multiple processors 208, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0276] <Flow of Virtual Tabular Data Creation Processing> The flow of virtual tabular data creation processing will be described below with reference to FIG.

[0277] Step S101: First, the virtual tabular data creation unit 101 receives the number of records of the virtual tabular data and the names and data types of the virtual columns (information 1 and 2 described in the above "Virtual Tabular Data Configuration Method"). The number of records of the virtual tabular data and the names and data types of the virtual columns are specified, for example, by the user.

[0278] Step S102: Next, the virtual tabular data creation unit 101 accepts the specification of URLs or paths of one or more source tabular data (the information of 3 described in the above "Method for constructing virtual tabular data"). The URLs or paths of the one or more source tabular data are specified, for example, by the user. Alternatively, the URLs or paths of one or more columns included in the one or more source tabular data may be specified.

[0279] Step S103: Next, the virtual tabular data creation unit 101 receives definitions of source columns and source sections in each source tabular data (information 4 described above in "Virtual tabular data configuration method"). The definitions of source columns and source sections in each source tabular data are specified, for example, by the user.

[0280] Step S104: Next, the virtual tabular data creation unit 101 receives the definition of the interval mapping for each source interval (the information in 5 described above in "Virtual Tabular Data Construction Method") and calculates the definition of the inverse mapping. Note that the definition of the interval mapping and its inverse mapping for each source interval is specified, for example, by the user. Also, once the interval mapping is defined, its inverse mapping is automatically determined by equations (12) and (14) and stored in the storage unit 105.

[0281] Step S105: Finally, the virtual tabular data creation unit 101 stores the information received in the above steps S101 to S104 in the storage unit 105 as virtual tabular data.

[0282] <Flow of Sorting Process on Virtual Tabular Data> The flow of sorting process on virtual tabular data will be described below with reference to FIG.

[0283] Step S201: The sorting unit 102 executes sorting with a virtual index using the virtual inverted structure of the virtual column to be sorted out of the virtual columns of the virtual table format data.

[0284] Step S202: Then, the sorting unit 102 stores the sorting result obtained in step S201 in the storage unit 105. However, the sorting unit 102 does not have to store the sorting result.

[0285] <Flow of Search Processing in Virtual Tabular Data> The flow of search processing in virtual tabular data will be described below with reference to FIG.

[0286] Step S301: First, the search unit 103 accepts search conditions.

[0287] Step S302: Next, the search unit 103 executes a search using a virtual index that uses the virtual inverted structure of the virtual column to be searched, among the virtual columns of the virtual table-format data.

[0288] Step S303: Then, the search unit 103 stores the search results obtained in step S302 in the storage unit 105.

[0289] <Flow of tallying process for virtual tabular data> The flow of tallying process for virtual tabular data will be described below with reference to FIG.

[0290] Step S401: The aggregation unit 104 executes aggregation using a virtual index that uses the virtual transposed structure of the virtual column to be aggregated, among the virtual columns of the virtual table format data.

[0291] Step S402: Then, the tallying unit 104 stores the tallying result obtained in the above step S303 in the storage unit 105. However, the tallying unit 104 does not have to store the tallying result.

[0292] As described above, the data processing device 10 according to this embodiment can define new virtual tabular data by allocation mapping using D5A or already defined virtual tabular data as source tabular data. Moreover, this virtual tabular data can be sorted, searched, and tallied at high speed using virtual indexes.

[0293] This makes it possible to create various virtual tabular data hierarchically, starting from the original tabular data D5A, and to quickly sort, search, and aggregate this virtual tabular data. Furthermore, users can create and publish new virtual tabular data from already published D5A or virtual tabular data according to their own purposes.

[0294] <Supplementary Note> The following will provide a supplementary note explaining why index inheritance is possible in this embodiment while existing technologies do not allow index inheritance.

[0295] An index is realized by a data structure for the index and a set of algorithms that use that data structure to speed up processing. An inverted structure can be considered a data structure for indexes that speeds up sorting, searching, and aggregation. A virtual inverted structure can be created for a column by rearranging the elements of the column from an inverted structure. A virtual inverted structure can also be created from multiple inverted structures. Furthermore, multiple virtual inverted structures can be layered to create a further virtual inverted structure. In this way, virtual inverted structures can be created hierarchically or by rearranging the elements of the column. Index algorithms that use inverted structures can also be applied to virtual inverted structures, resulting in a virtual index. Existing indexes do not have a data structure for indexes that can be combined multiple times or hierarchically, as in inverted structures, so inheritance is not possible when combining or layering.

[0296] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0297] This application is based on basic applications No. 2023-010716 and No. 2023-010717 filed in Japan on January 27, 2023, the entire contents of which are incorporated herein by reference.

[0298] 10 Data processing device 20 Network 101 Virtual tabular data creation unit 102 Sorting unit 103 Search unit 104 Aggregation unit 105 Storage unit 201 Input device 202 Display device 203 External I / F 203a Recording medium 204 Communication I / F 205 RAM 206 ROM 207 Auxiliary storage device 208 Processor 209 Bus

Claims

1. A process of constructing an index on a virtual column defined by a predetermined mapping from one or more source columns included in one or more tabular data by referencing a list of values ​​on the source columns and the frequency of occurrence of the values; How a computer runs.

2. 2. The method of claim 1, wherein another virtual column defined by another predetermined mapping is used as a source column, and the index is constructed by referencing a list of values ​​on the source column that is the other virtual column and the frequency of occurrence of the values.

3. The method according to claim 1 or 2, wherein the cumulative total of the values ​​is used as the frequency of occurrence of the values.

4. The method according to claim 1 , wherein the list of values ​​is an array that holds the values ​​in ascending order.

5. The method of claim 1 , further comprising constructing the index by further referencing an inverted index of the source column.

6. The method according to claim 1 , further comprising the step of: executing, by the computer, a process for implementing a sorting operation, a search operation, or an aggregation operation using the index.

7. constructing an index on a virtual column defined by a predetermined mapping from one or more source columns included in one or more tabular data by referencing a list of values ​​on the source columns and the frequency of occurrence of the values; Device.

8. A process of constructing an index on a virtual column defined by a predetermined mapping from one or more source columns included in one or more tabular data by referencing a list of values ​​on the source columns and the frequency of occurrence of the values; A program that causes a computer to execute the following.

9. a process of defining a predetermined mapping that assigns each cell in one or more source columns to a virtual column without overlapping, and obtaining a value of each cell in the virtual column using an inverse mapping of the mapping; How a computer runs.

10. 10. The method of claim 9, further comprising: defining a mapping that uses another virtual column defined by another predetermined mapping as a source column, and allocates each cell on one or more source columns including the other virtual column to the virtual column without overlapping.

11. The method according to claim 9 or 10, wherein the mapping is an enumeration mapping, a linear functional mapping, or both the enumeration mapping and the linear functional mapping.

12. The method according to claim 9 , further comprising the step of displaying the obtained values.

13. the computer further executes a process of determining whether or not a collision occurs between allocation destinations when two different source columns are allocated to virtual columns using different mappings; 13. The method of any one of claims 9 to 12, further comprising defining different non-conflicting mappings.

14. defining a predetermined mapping that allocates each cell in one or more source columns to a virtual column without overlapping, and obtaining a value of each cell in the virtual column using an inverse mapping of the mapping; Device.

15. a process of defining a predetermined mapping that assigns each cell in one or more source columns to a virtual column without overlapping, and obtaining a value of each cell in the virtual column using an inverse mapping of the mapping; A program that causes a computer to execute the following.