Search device, search method, and program

Abstract

A search device according to the present invention comprises: a data set composed of multidimensional data; a first array for storing, for each dimension, a data number of a value appearing in the dimensions of the data set, said data numbers being stored in ascending order of the value; a second array obtained by switching an element and an index of the first array; and a means for obtaining the appearance probability of each dimension and a value in which each dimension appears in the dimensions. When a degree of similarity is included in a dimension search condition, a hit section is set on the first array by using the value and the appearance probability of each dimension, and the data numbers commonly included in the hit section between the dimensions designated by the search condition are identified on the basis of the second array.

Description

Search device, search method, and program

[0001] The present disclosure relates to a search device, a search method, and a program.

[0002] In recent years, advances in various sensor devices and observation equipment have made it possible to obtain large amounts of data, known as big data, which represents sensing results, observation results, etc. This has led to a demand for the ability to extract desired data by performing searches and other operations on big data.

[0003] Patent Document 1 discloses a technique that allows data in a format called D5A to be created on a storage device as internal data used for operations such as searches on tabular data.

[0004] International Publication No. 2023 / 276162

[0005] Columns in tabular data are often called dimensions. Search criteria often specify multiple dimensions, making it difficult to efficiently search for records similar to the desired multidimensional data. Specifically, first, it takes time to determine similarity within each dimension. Second, it takes time to search for highly similar records within each dimension. Third, it takes time to AND the search results across each dimension.

[0006] The present disclosure aims to provide a technology that can efficiently search for data similar to desired data from a data set made up of multidimensional data, and can sort the search results in order of similarity.

[0007] A search device according to one aspect of the present disclosure comprises a dataset consisting of multidimensional data, a first array that stores, for each dimension, the data numbers of values ​​that appear in the dimension of the dataset in ascending order of the values, a second array in which the subscripts and elements of the first array are swapped, and a means for calculating the values ​​that appear in each dimension and the occurrence probability of each dimension, and when similarity is included in the search conditions for a dimension, a hit interval is set on the first array using the values ​​and the occurrence probability of each dimension, and data numbers that are commonly included in the hit interval between the dimensions specified in the search conditions are identified based on the second array.

[0008] A technology is provided that can efficiently search for data similar to desired data from a data set made up of multidimensional data, and can sort the search results in order of similarity.

[0009] FIG. 1 is a diagram showing an example of the overall configuration of a search system according to the present embodiment; FIG. 2 is a diagram showing an example of the hardware configuration of a search device according to the present embodiment; FIG. 3 is a diagram showing an example of the functional configuration of a search device according to the present embodiment; FIG. 4 is a diagram showing an example of a data set made up of multidimensional data; －１ , SVL, ACM, and INV. FIG. 1 is a flowchart showing an example of a search process according to the present embodiment. FIG. 2 is a diagram showing data used in Example 1 and the occurrence probability of each data. FIG. 3 is a diagram for explaining a method for calculating similarity in Example 1. FIG. 4 is a diagram for explaining a method for calculating similarity in Example 1. FIG. 5 is a diagram for explaining a method for calculating similarity in Example 1. FIG. 6 is a diagram for explaining a method for calculating similarity in Example 1. FIG. 7 is a diagram for explaining a method for calculating similarity in Example 1. NNC, INV of each dimension in Example 1. －１20 is a diagram showing examples of GSVL, GACM, LACM, m#, and MACM for each dimension of a dataset made up of multidimensional data according to Example 2. FIG. 21 is a diagram showing an example of module division of the dataset shown in FIG. 20. FIG. 22 is a diagram showing an example of search processing in Example 1. FIG. 23 is a diagram showing an example of search processing in Example 1. FIG. 24 is a diagram showing an example of search processing in Example 1. FIG. 25 is a diagram showing an example of search processing in Example 1. FIG. 26 is a diagram showing an example of search processing in Example 1. FIG. 27 is a diagram showing an example of search results in Example 1. FIG. 28 is a diagram showing an example of a dataset made up of multidimensional data according to Example 2. FIG. 29 is a diagram showing an example of GSVL, GACM, LACM, m#, and MACM for each dimension of a dataset made up of multidimensional data according to Example 2. FIG. 29 is a diagram showing an example of module division of the dataset shown in FIG. 20. FIG. 21 is a diagram showing an example of search processing in Example 2. FIG. 22 is a diagram showing an example of search results in Example 2.

[0010] An embodiment of the present invention will be described in detail below with reference to the drawings. In the following embodiment, a search system 1 will be described that can efficiently search for desired data from a data set composed of multidimensional data when the dimension to be searched is freely specified. Note that the dimension to be searched used in the search system 1 according to this embodiment can be freely selected to speed up searches, and therefore is called a "freely rearrangeable multidimensional index." A feature of a freely rearrangeable multidimensional index is that it can speed up searches for any combination of dimensions (columns) and any subset (record number string).

[0011] Here, multidimensional data refers to data that is composed of multiple dimensions (generally many dimensions). A data set composed of multidimensional data is, for example, the i-th (0≦i≦I−1) multidimensional data set x ｉ In this case, {x ｉ|0≦i≦I−1}. Also, the i-th multidimensional data x ｉ For example, the jth dimension value (0≦j≦J−1) is expressed as x ｉ，ｊ In this case, x ｉ = (x ｉ，０ , ..., x ｉ，Ｊ－１ When this is considered as tabular data, the above x ｉ is the record, i is the record number, and I is the total number of records. ０，ｊ , ..., x Ｉ－１，ｊ ) corresponds to the j-th column. In the following, a data set made up of multidimensional data will be regarded as tabular data where appropriate.

[0012] In the following, data obtained by extracting only values ​​of a certain dimension from a data set composed of multidimensional data will be referred to as column data. The jth (0≦j≦J−1) column data is, for example, y ｊ = (x ０，ｊ , ..., x Ｉ－１，ｊ ) τ (where τ is the symbol for transposition).

[0013] <Example of Overall Configuration of Search System 1> An example of the overall configuration of the search system 1 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the search system 1 according to this embodiment includes a user device 10 and a search device 20. The user device 10 and the search device 20 are connected to each other so as to be able to communicate with each other via an arbitrary communication network 30 such as the Internet.

[0014] The user device 10 is one of various devices used by users of the search system 1. In response to a user's operation, the user device 10 transmits a search request including search criteria specified by the operation to the search device 20, and receives search results for the search request. The user device 10 is realized by various devices, equipment, and terminals, such as a personal computer (PC), a smartphone, a tablet terminal, a wearable device, a general-purpose server, industrial equipment, IoT equipment, etc. Hereinafter, the search criteria are assumed to specify the order of one or more dimensions and conditions related to the values ​​of those dimensions (e.g., a condition expressing an equality relationship, a condition expressing a magnitude relationship, a condition expressed by a logical formula, etc.).

[0015] The search device 20 is a device that, upon receiving a search request from the user device 10, searches for multidimensional data that satisfies the search conditions included in the search request. The search device 20 is realized, for example, by a general-purpose server or a system configured of such servers.

[0016] 1 is an example, and the overall configuration of the search system 1 is not limited to this. For example, the search system 1 may include a plurality of user devices 10.

[0017] <Example of Hardware Configuration of Search Device 20> An example of the hardware configuration of the search device 20 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the search device 20 according to this embodiment includes an input device 201, a display device 202, an external I / F 203, a communication I / F 204, a RAM (Random Access Memory) 205, a ROM (Read Only Memory) 206, an auxiliary storage device 207, and a processor 208. Each of these pieces of hardware is connected to each other via a bus 209 so as to be able to communicate with each other.

[0018] The input device 201 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 202 is, for example, a display, a display panel, etc. Note that the search device 20 does not necessarily have to include at least one of the input device 201 and the display device 202, for example.

[0019] The external I / F 203 is an interface with an external device such as a recording medium 203 a. Examples of the recording medium 203 a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0020] The communication I / F 204 is an interface for connecting to the communication network 30. The RAM 205 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 206 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 207 is a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 208 is, for example, one of various arithmetic devices such as a CPU (Central Processing Unit).

[0021] 2 is an example and is not intended to be limiting. For example, the search device 20 may have multiple auxiliary storage devices 207 or multiple processors 208, may not have some of the hardware shown in the figure, or may have various types of hardware other than the hardware shown in the figure.

[0022] <Example of Functional Configuration of Search Device 20> An example of the functional configuration of the search device 20 according to this embodiment is shown in FIG. 3. As shown in FIG. 3, the search device 20 according to this embodiment includes a search request receiving unit 210, a search unit 211, and a search result response unit 212. These units are realized, for example, by a process in which one or more programs installed in the search device 20 are executed by the processor 208 or the like. The search device 20 according to this embodiment also includes a data storage unit 213. The data storage unit 213 is realized, for example, by a storage area of ​​the auxiliary storage device 207 or the like. However, the data storage unit 213 may be realized, for example, by a storage area of ​​a storage device (e.g., a storage device provided in a database server or the like) connected to the search device 20 so as to be able to communicate with the search device 20, instead of or in addition to the storage area of ​​the auxiliary storage device 207 or the like.

[0023] The search request receiving unit 210 receives a search request transmitted from the user device 10. The search unit 211 searches for multidimensional data that satisfies the search conditions included in the search request received by the search request receiving unit 210. The search result response unit 212 transmits the search results by the search unit 211 to the user device 10. The data storage unit 213 stores a data set composed of multidimensional data. Here, the data set stored in the data storage unit 213 has a structure in which each dimension is an INV (Inverted Record Index) and an INV －１ and a data structure from which a value corresponding to the record number can be read.

[0024] The above-mentioned "data structure from which a value corresponding to a record number can be read" may be, for example, any of the following (1) to (3), but is not limited to these.

[0025] (1) Column data (2) NNC (Natural Numbered Column) and SVL (Sorted Value List) (3) SVL, ACM (Accumulation Array) and INV The column data shown in (1) above refers to the column data for each dimension of a data set consisting of multidimensional data. The column data for each dimension can be obtained simply by referencing the columns of the data set consisting of multidimensional data.

[0026] In the following, the above (3) is mainly assumed as a data structure from which a value corresponding to a record number can be read.

[0027] An NNC is a list-style data structure in which each value contained in the column data of a corresponding dimension is represented by its storage location in the SVL. An SVL is a list-style data structure whose elements are unique values ​​that appear in the column data of a corresponding dimension, arranged in ascending order. An ACM is a list-style data structure whose elements, for each element of the SVL of a corresponding dimension, are values ​​that indicate the position next to the last position on the INV where the record number of that element is stored.

[0028] Each element in the ACM gives the number of occurrences of values ​​equal to or less than the corresponding element on the SVL. In other words, ACM[i] gives the number of occurrences of values ​​equal to or less than SVL[i]. For example, in the dimension "Name" in Figure 4, the number of occurrences of values ​​equal to or less than "Bob" is three, with "Alice" occurring once and "Bob" occurring twice. Looking at this for the dimension "Name" in Figure 5, "Bob" is stored in SVL[1]. Therefore, the number of occurrences of values ​​equal to or less than "Bob" can be found by looking at ACM[1], which is three, which is correct.

[0029] The following formulas are given for ACM. These formulas will be used in the following explanations. Note that it is decided beforehand that ACM[-1] = 0. Formula (1): Number of occurrences of values ​​equal to or less than SVL[i] = ACM[i] Formula (2): Number of occurrences of SVL[i] = ACM[i] - ACM[i-1] Formula (3): The interval on INV corresponding to SVL[i] is the closed interval [ACM[i-1], ACM[i]-1]

[0030] INV is a list-style data structure whose elements are the transposed record numbers of the column data of the corresponding dimension. －１ If you use INV －１ [i] can immediately find the storage position of element i on INV. For example, when INV = (3, 0, 1, 2), INV －１ = (1, 2, 3, 0), but the storage location of element 0 on INV is INV －１ [0] = 1, the storage location of element 3 on INV is INV －１ This can be easily calculated by setting [3] = 0. Since the elements on the INV are record numbers, －１ By using this, it is possible to immediately find out where a certain record number is stored on the INV. For details on NNC, SVL, ACM, and INV and how to create them, please refer to, for example, the above-mentioned Patent Document 1.

[0031] However, the column data for each dimension is INV and INV －１ and a data structure that can read out a value corresponding to the record number, and NNC, SVL, and ACM are not essential.

[0032] In addition, NNC, SVL, ACM, INV and INV of each column data of a data set composed of multidimensional data －１ For example, the data may be created by a functional unit (for example, a functional unit called an "internal data creation unit") provided in the search device 20, or may be created by a device other than the search device 20 and then stored in the data storage unit 213.

[0033] <Data Structure Capable of Reading Values ​​Corresponding to Record Numbers in Each Dimension> Here, a supplementary explanation will be given of the data structure capable of reading values ​​corresponding to record numbers in each dimension.

[0034] The column data for each dimension ((1) above) can read the value of that dimension from the record number.

[0035] The NNC and SVL (above (2)) for each dimension can restore the column data for that dimension. Therefore, the NNC and SVL for each dimension can read the value of that dimension from the record number. For example, let the assignment operator be ":=", the i-th array element of the array that realizes NNC be NNC[i], the i-th array element of the array that realizes SVL be SVL[i], and an empty array be ORG. In this case, the original column data can be restored as ORG by extracting the NNC element NNC[i] sequentially starting from i=0 and storing SVL[NNC[i]] in ORG[i]. In other words, by repeating ORG[i]:=SVL[NNC[i]] sequentially starting from i=0, the array ORG that realizes the original column data can be restored.

[0036] The SVL, ACM, and INV (above (3)) of each dimension can restore the column data of that dimension. Therefore, the SVL, ACM, and INV of each dimension can read the value of that dimension from the record number. For example, let the assignment operator be ":=", the i-th array element of the array that realizes SVL be SVL[i], the i-th array element of the array that realizes ACM be ACM[i], the i-th array element of the array that realizes INV be INV[i], and an empty array be ORG. Also, let ACM[-1] = 0, and the size of SVL (that is, the number of elements of the array that realizes SVL) be N. In this case, i ＳＴＡＲＴ =ACM[i-1],i ＳＴＯＰ =ACM[i]-1, and then execute the following repeatedly for i=0, 1, . . . , N-1.

[0037] ORG[INV[i ＳＴＡＲＴ ]]:=SVL[i] ORG[INV[i ＳＴＡＲＴ +1]]:=SVL[i]...ORG[INV[i ＳＴＯＰ ]]:=SVL[i] This allows the array ORG that realizes the original column data to be restored.

[0038] <NNC, SVL, ACM, INV, INV －１ As an example, NNC, SVL, ACM, INV, and INV of each column data of the data set 1000 shown in FIG. －１is shown in Figure 5. Note that the data set 1000 shown in Figure 4 is composed of two-dimensional multidimensional data, the dimensions "Name" and "Age", and the numbers on the left side represent the record number of each multidimensional data. For example, the record number of (Name, Age) = ("Bob", 30) is "0", the record number of (Name, Age) = ("Alice", 26) is "1", and the record number of (Name, Age) = ("Cat", 26) is "2". The same applies to other multidimensional data.

[0039] As shown in FIG. 5, the column data of the dimension “Name” is NNC1111, SVL1112, ACM1113, INV1114, and INV1115. －１ Similarly, the column data of the dimension "Age" is represented by NNC 1211, SVL 1212, ACM 1213, INV 1214, and INV －１ It is expressed as 1215.

[0040] For example, INV and INV for all column data －１ The data structure that can read out the value corresponding to the record number and the table may be collectively called a "multidimensional index" or the like.

[0041] <Search Processing> An example of search processing according to this embodiment will be described with reference to FIG.

[0042] The search request receiving unit 210 receives a search request transmitted from the user device 10 (step S101). Here, the search request includes search conditions, which include the order of one or more dimensions and conditions related to the values ​​of those dimensions. Hereinafter, the order of one or more dimensions will be referred to as the "dimension order," and the conditions related to the dimension values ​​will be simply referred to as the "dimension search conditions." Note that the dimension search conditions may be set for only some dimensions or for all dimensions.

[0043] A specific example of a search condition is (Name, Age) = ("Cat" ± 2 bits, 26 ± 2 bits). This indicates that the dimension search condition for the dimension "Name" is "Name = a value in the ± 2 bit range centered on "Cat"," and the dimension search condition for the dimension "Age" is "Age = a value in the ± 2 bit range centered on 26."

[0044] Here, the search unit 211 uses the SVL and ACM to calculate the similarity based on the probability that a value appears in the interval from the specified value to the stored value.

[0045] In this case, various definitions of similarity are possible, but it is preferable that the definition be based on the distribution of values ​​in each dimension, and that the sum of the similarities in each dimension be the overall similarity. Similarity based on the amount of information meets this requirement. The following is one such definition of similarity used in the search device 20 according to this embodiment. The definition of similarity is expressed as shown in the following formula (1).

[0046] Since the sum of similarities is not meaningless, the overall similarity can be calculated by summing the similarities of each dimension. Also, for example, a "2-bit" similarity means that the co-occurrence probability is 1 / (2^2) = 1 / 4 or less, and since there are 2 bits above and below, when the target value is near the center of the distribution, it can be said that 1 / 2 of the total falls within that range.

[0047] The similarity s (≧0) is calculated using the following formula (1): ｉ is the occurrence probability of the i-th value, saved is the storage position of the stored value on the SVL, target is the storage position of the specified value on the SVL, Count ｉ indicates the number of occurrences of the i-th value, and TotalCount indicates the total number of records.

[0048]

[0049] The search unit 211 preferably sorts the dimensions included in the search criteria in the order that maximizes the speed of the search. The order that maximizes the speed of the search is simply the order of the shortest hit intervals on the INV. However, in addition to the shortest hit intervals, the elements on the INV are sorted in ascending order, and elements with similar values ​​are more likely to be cached during IO, resulting in faster speeds.

[0050] The search device 20 calculates the hit intervals on the SVL according to the search conditions of each dimension (step S102), and then converts the hit intervals on the SVL of each search dimension into hit intervals on the INV (step S103).

[0051] Then, the elements (=record numbers) i in the Hit interval on INV in the first dimension are sequentially extracted, and if INV-1[i] in the second dimension is within the Hit interval of the second dimension, it is confirmed that INV-1[i] in the third and subsequent dimensions is also within the Hit interval of that dimension. Then, when it is confirmed that INV-1[i] is within the Hit interval of all dimensions from the second onwards, the record number is registered in the Hit column (step S104).

[0052] As a result, the record numbers of the multidimensional data that satisfy all of the dimensional search conditions included in the search conditions are stored in the Hit column. Here, the Hit column is a list-style data structure in which record numbers are stored. The Hit column is stored, for example, in a storage area of ​​the auxiliary storage device 207, a storage area of ​​the RAM 205, etc.

[0053] The search result response unit 212 transmits the record numbers included in the Hit column as search results to the user device 10 (step S105). This allows the user device 10 to obtain the record numbers of the multidimensional data that satisfy the search conditions. Note that the search result response unit 212 may transmit the multidimensional data of those record numbers as search results to the user device 10, instead of the record numbers included in the Hit column.

[0054] <Example 1> Fig. 7 is a diagram showing data used in Example 1 and the occurrence probability of each piece of data. In Fig. 7, (a) shows a data set, (b) shows the number of data and the occurrence probability of each piece of data included in the dimension "Age", (c) shows the dimension "Height", and (d) shows the number of data and the occurrence probability of each piece of data included in the dimension "Weight". In (b) to (d), Count in the diagram indicates the number of data, and p indicates the occurrence probability.

[0055] The occurrence probability is the probability that data belonging to each dimension will appear. Here, the occurrence probability of data whose dimension "Age" is "15" will be calculated as an example. As shown in (a) of FIG. 7, the total number of data items in the dimension "Age" is 16, of which the number of data items whose dimension "Age" is "15" is 2. Therefore, the occurrence probability p of data whose dimension "Age" is "15" is 2 / 16 = 0.125. The results of calculating the occurrence probability of each data item in each dimension using the calculation method of the above example are shown in (b), (c), and (d) of FIG. 7.

[0056] 8A, 8B, 8C, and 8D are diagrams for explaining a method of calculating similarity in Example 1. In FIGS. 8A, 8B, 8C, and 8D, data of the dimension "Height" shown in FIG. 7A is used. In FIGS. 8A, 8B, 8C, and 8D, "saved" in the table indicates the storage position of the stored value on the SVL, ΣCount indicates the number of records from the specified value to the stored value, and Σp i indicates the total occurrence probability from the specified value to the stored value, and s indicates the similarity. In each graph, the horizontal axis indicates the stored data, and the vertical axis indicates the number of data.

[0057] 8A, 8B, 8C, and 8D each have a different specified value: 164 in FIG. 8A, 147 in FIG. 8B, 181 in FIG. 8C, and 165 in FIG. 8D. In other words, FIG. 8A shows the case where the specified value is the stored value, FIG. 8B shows the case where the specified value is equal to or less than the minimum stored value, FIG. 8C shows the case where the specified value is equal to or greater than the maximum stored value, and FIG. 8D shows the case where the specified value is not stored.

[0058] FIG. 9 shows the NNC and INV of each dimension in the first embodiment.－１ 13 is a diagram showing an example of the column data of the dimension "Age" as follows: NNC 1311, SVL 1312, ACM 1313, INV 1314, and INV －１ Similarly, the column data of the dimension "Height" is represented by NNC 1411, SVL 1412, ACM 1413, INV 1414, and INV －１ Similarly, the column data of the dimension "Weight" is expressed as NNC 1511, SVL 1512, ACM 1513, INV 1514, and INV －１ It is expressed as 1515.

[0059] Below, the k-th array element of the dimension “Age” is １ [k], SVL １ [k], ACM １ [k], INV １ [k], INV １ －１ Similarly, the k-th array element of the dimension "Height" is ２ [k], SVL ２ [k], ACM ２ [k], INV ２ [k], INV ２ －１ Similarly, the k-th array element of the dimension "Weight" is ３ [k], SVL ３ [k], ACM ３ [k], INV ３ [k], INV ３ －１ [k]. Note that k starts from 0. Also, ACM １ [-1] = ACM ２ [-1] = ACM ３ It is defined that [-1]=0.

[0060] 10 is a diagram showing an example of calculation of similarity in Example 1. The illustrated example is an example in which the specified value is set to 165 in the data of the dimension “Height” shown in FIG.

[0061] First, find the position on the SVL of the specified value 165. As shown in Figure 10(a), 165 is not the stored value, but a value between the stored values ​​164 and 167. Since the positions on the SVL of 164 and 167 are 5 and 6, respectively, the position on the SVL of 165 is set to 5.5, which is between 5 and 6.

[0062] Figure 10(b) shows the results of calculating the similarity when the record number is 1, with a box drawn on the graph. This figure will be used to explain how to calculate the similarity between record number 1 and target. Here, the similarity of data with record number 1 is calculated. The value corresponding to record number 1 is SVL[NNC[1]] = SVL[4] = 163. Since saved = NNC[1] = 4, this is smaller than the storage location of the specified value, target = 5.5.

[0063] The number of occurrences of values ​​less than or equal to 5.5, which is the storage location of the specified value, is ACM[floor(5.5)] = ACM[5] = 7. On the other hand, the number of occurrences of values ​​less than the value stored in record number 1 is ACM[NNC[1] - 1] = ACM[4 - 1] = 5. Therefore, the number of data that appear between these two is 7 - 5 = 2, so the similarity is -log(2 / 16) = 3 [bit]. Note that (b) in Figure 10 is a graph showing a method for calculating similarity when record number is 1.

[0064] Next, calculate the similarity of the data with record number 4. The value corresponding to record number 4 is SVL[NNC[4]] = SVL[9] = 177. Since saved = NNC[4] = 9, this is greater than the storage location target of the specified value = 5.5.

[0065] The number of occurrences of values ​​equal to or less than 5.5, which is the storage location of the specified value, is ACM[floor(5.5)] = ACM[5] = 7. On the other hand, the number of occurrences of values ​​equal to or less than the value stored in record number 4 is ACM[NNC[4]] = ACM[9] = 15. Therefore, the number of data is 15 - 7 = 8, so the similarity is -log(8 / 16) = 1 [bit]. Note that (c) in Figure 10 is a graph showing a method for calculating similarity when record number is 4.

[0066] Hereinafter, the hit interval is determined using the search conditions (Age, Height, Weight)=(17±2 bits, 165±2 bits, 55±2 bits).

[0067] First, find the hit interval for the dimension "Age". If the similarity is 2 bits, -log(4 / 16) = 2 [bits], so the number of occurrences between the target value and the value on the record is 4. From (b) of FIG. 7, there are 7 occurrences for Age=17 alone, so Age=17 is applicable. Using (a) of FIG. 9, find the elements of SVL1312 that satisfy the dimension search condition "Age="17"" in the dimension "Age", and you get １ Since [2]="17", the Hit interval on INV1314 of the dimension "Age" is calculated by the formula (3) above as [ACM １ [2-1], ACM １ [2] - [1] = [8, 14].

[0068] Similarly, the Hit interval for the dimension "Height" is found. Since the number of occurrences between the target value and the value on the record is four, from (c) of FIG. 7, when Height = 165, the four items below are 155 or greater, and the four items above are 170 or less. When the elements of SVL1412 that satisfy the dimension search condition "Height = "155"" to "Height = "170"" are found for the dimension "Height" using (b) of FIG. 9, the Hit interval on INV1414 for the dimension "Height" is [3, 10].

[0069] Similarly, the Hit interval for the dimension "Weight" is found. Since the number of occurrences between the target value and the value on the record is four, from (d) of Figure 7, when Weight = 55, the four items below are 47 or more, and the four items above are 61 or less. When the elements of SVL1512 that satisfy the dimension search condition "Weight = "47"" to "Weight = "61"" for the dimension "Weight" are found using (c) of Figure 9, the Hit interval on INV1514 for the dimension "Weight" is [1, 7].

[0070] An example of search processing when the search conditions are (Age, Height, Weight) = (17 ± 2 bits, 165 ± 2 bits, 55 ± 2 bits) and the similarity is 2 bits or more will be described below with reference to Figures 11 to 17. Figures 11 to 17 are diagrams for explaining an example of search processing in Example 1.

[0071] As described above, the Hit interval of the dimension "Age" is [8, 14], so for each element (record number) i whose storage position on the INV of the dimension "Age" is 8 to 14, －１ Processing is performed to determine whether [i] is included in the Hit section.

[0072] The example in FIG. 11 shows a search process when the storage position on INV of the dimension "Age" is 8. First, the record number i is i=INV １ [8] = 8. Next, as shown by arrow A in the figure, ２ －１ [i] = INV ２ －１ [8]=8, which is included in the Hit section. Next, as shown by arrow B in the figure, ３ －１ [i] = INV ３ －１ [8]=14, which is not included in the Hit section. Therefore, when the storage position on INV of the dimension "Age" is 8, nothing is added to the array that realizes the Hit column 2000.

[0073] The example in Figure 12 shows the search process when the storage position on INV of the dimension "Age" is 9. First, the record number i is i=INV １ [9] = 9. Next, as shown by arrow A in the figure, INV ２ －１ [i] = INV ２ －１ [9]=3, which is included in the Hit section. Next, as shown by arrow B in the figure, ３ －１ [i] = INV ３ －１Therefore, if the storage position on INV of the dimension "Age" is 9, the array element Hits[0]=9 is added to the array that realizes the Hit column 2000.

[0074] The example in Figure 13 shows the search process when the storage position on INV of the dimension "Age" is 10. First, the record number i is i=INV １

[10] = 10. Next, as shown by arrow A in the figure, INV ２ －１ [i] = INV ２ －１

[10] =13, which is not included in the Hit section. Therefore, if the storage position on INV of the dimension "Age" is 10, nothing is added to the array that realizes the Hit column 2000.

[0075] The example in Figure 14 shows the search process when the storage position on INV of the dimension "Age" is 11. First, the record number i is i=INV １

[11] = 11. Next, as shown by arrow A in the figure, INV ２ －１ [i] = INV ２ －１

[11] =9, which is included in the Hit section. Next, as shown by arrow B in the figure, ３ －１ [i] = INV ３ －１ Therefore, if the storage position on the INV of the dimension "Age" is 11, the array element Hits[1]=11 is added to the array that realizes the Hit column 2000.

[0076] The example in Figure 15 shows the search process when the storage position on INV of the dimension "Age" is 12. First, the record number i is i=INV １

[12] = 12. Next, as shown by arrow A in the figure, INV ２ －１ [i] = INV ２ －１

[12] =14, which is not included in the Hit section. Therefore, if the storage position on INV of the dimension "Age" is 12, nothing is added to the array that realizes the Hit column 2000.

[0077] The example in Figure 16 shows the search process when the storage position on INV of the dimension "Age" is 13. First, the record number i is i=INV １

[13] = 13. Next, as shown by arrow A in the figure, INV ２ －１ [i] = INV ２ －１

[13] =10, which is included in the Hit section. Next, as shown by arrow B in the figure, ３ －１ [i] = INV ３ －１

[13] =11, which is not included in the Hit section. Therefore, when the storage position on INV of the dimension "Age" is 13, nothing is added to the array that realizes the Hit column 2000.

[0078] The example in Figure 17 shows the search process when the storage position on INV of the dimension "Age" is 14. First, the record number i is i=INV １

[14] = 14. Next, as shown by arrow A in the figure, INV ２ －１ [i] = INV ２ －１

[14] = 0, and is not included in the Hit section. Therefore, when the storage position on INV of the dimension "Age" is 14, nothing is added to the array that realizes the Hit column 2000.

[0079] 18A, 18B, and 18C are diagrams showing examples of search results in Example 1. FIG. 18A shows the search results, FIG. 18B is a diagram for explaining a method for calculating the similarity with the target value for record number 9, and FIG. 18C is a diagram for explaining a method for calculating the similarity with the target value for record number 11. As shown in FIG. 18A, an array realizing the Hit column 2000 is obtained, and the multidimensional data (17, 155, 51) and (17, 169, 61) can be obtained from the record numbers Hits[0]=9 and Hits[1]=11 stored in this array.

[0080] Next, the difference between the multidimensional data of record numbers 9 and 11 and the target multidimensional data (Age, Height, Weight)=(17, 165, 55) is calculated.

[0081] First, record number 9 has 7 / 16 records in the dimension "Age" so 1.1926 bits, 4 / 16 records in the dimension "Height" so 2 bits, and 3 / 16 records in the dimension "Weight" so 2.415 bits, for a total similarity of 5.608 bits.

[0082] Next, for record number 11, the similarity is 1.1926 bits since it is 7 / 16 records in the dimension "Age", 2.415 bits since it is 3 / 16 records in the dimension "Height", and 2 bits since it is 4 / 16 records in the dimension "Weight", for a total similarity of 5.608 bits.

[0083] In this way, by calculating the total similarity for each hit record, the search results can be transmitted to the user device 10 in descending order of similarity. The search results may also be transmitted to the user device 10 together with the similarity.

[0084] In the explanation of FIGS. 11 to 17, the record number i is sequentially picked up from the interval on the INV of the dimension “Age”, and the INV of the dimension “Height” and the dimension “Weight” are －１ Although [i] was accessed, all dimensions are equal, and it is obvious that the dimension "Height" or the dimension "Weight" can be used as the starting point instead of the dimension "Age". Furthermore, if a search or sorting has already been performed in some way and there is an extracted record number string as a result, it can be used as the starting point to access the INV of the dimensions "Age", "Height" and "Weight". －１ 11 to 17 show how to implement an index that is effective for an arbitrary record number string (which can be called a subset).

[0085] The above describes Example 1 of the search device 20 according to this embodiment. In conventional technology, searching for similar vectors requires calculating the degree of similarity between each dimension of the target vector and the extracted vector, and then determining whether or not the vectors are similar based on the sum of these calculations. Specifically, the time required for a search is due to the process of calculating the degree of similarity between each dimension and the process of extracting all vectors from the tabular data.

[0086] The search device 20 according to this embodiment can solve the above problem by quickly calculating the similarity of each dimension and by reducing the number of vectors extracted from tabular data using a freely rearrangeable multidimensional index. Furthermore, the search device 20 according to this embodiment can calculate the similarity in steps O(1) in order notation, enabling efficient searches using the mechanism of a freely rearrangeable multidimensional index.

[0087] <Modifications> Modifications of the above embodiment will be described below. In a data set consisting of multidimensional data, NNC, INV －１ In the example where the record numbers are read from a continuous section of INV in the example where the record numbers are read from the continuous section of INV, an efficient search process can be performed.

[0088] However, if the values ​​on the INV that are read are widely dispersed, －１ Therefore, for a data set with a large number of records, file access efficiency is low, which may result in a decrease in performance.

[0089] The search device 20 according to the modified example performs processing to prevent the number of records in a data set from increasing beyond a predetermined number in order to prevent the values ​​on the INV from being dispersed over a wide range. Therefore, since the values ​​on the INV are dispersed over a wide range when the number of records is large, the search device 20 divides the data set into multiple modules to reduce the number of records included in each module.

[0090] Furthermore, the search device 20 according to the modified example minimizes the increase in processing effort due to module division by using a "public SVL / ACM" and a "cross-module ACM" instead of the SVL and ACM used in the embodiment. By using this method, the search device 20 according to the modified example maintains the speed of search processing even in data sets containing a huge number of records.

[0091] The search device 20 according to the modified example has four additional components compared to the search device 20 according to the embodiment. The first is a "public SVL / ACM" that allows external access to a data set without being aware of the module division. GSVL (Global SVL) corresponds to SVL, and GACM (Global ACM) corresponds to ACM. This allows the number of types of values ​​that appear in a column, the maximum and minimum values ​​that appear, and the number of times each value appears.

[0092] The second defines a Local ACM (LACM) that connects the public SVL and the cross-module ACM, where LACM[-1]≡0 and GSVL[i] is compatible with MACM[LACM[i-1]], MACM[LACM[i-1]+1], ..., MACM[LACM[i]-1].

[0093] The third is "Trans-module ACM (MACM)," which can determine how many times a corresponding GSVL value appears in which module using a calculation method similar to that of LACM. "Trans-module ACM" is used in search processing using similarity. The fourth is "GSVL#," which defines the position in the GSVL where the SVL value in each module appears.

[0094] <Example 2> Fig. 19 is a diagram illustrating an example of a dataset configured with multidimensional data according to Example 2. Fig. 20 is a diagram illustrating an example of GSVL, GACM, LACM, m#, and MACM for each dimension of a dataset configured with multidimensional data according to Example 2.

[0095] The data set shown in Figure 19 is composed of two-dimensional multidimensional data, the dimensions "Name" and "Age," and the numbers on the left side indicate the record number of each multidimensional data. For example, the record number for (Name, Age) = ("Bob", 22) is "0," and the record number for (Name, Age) = ("Alice", 26) is "1." The same applies to other multidimensional data.

[0096] 20, the column data of the dimension "Name" is represented by GSVL2111, GACM2112, LACM2113, m#2114, and MACM2115. Similarly, the column data of the dimension "Age" is represented by GSVL2211, GACM2212, LACM2213, m#2214, and MACM2215.

[0097] FIG. 21 is a diagram showing an example of module division of the data set shown in FIG. 20. In the diagram, m is the module number. As shown in the diagram, for the dimension "Name" and the dimension "Age", each data is divided into modules "0", "1", and "2", respectively. Below, an example of search processing in each divided module will be described.

[0098] As shown in the figure, the column data of the dimension “Name” is NNC2301, SVL2302, ACM2303, INV2304, INV2305, and so on, in module number m=0. －１ Similarly, in module number m=1, NNC2311, SVL2312, ACM2313, INV2314, INV －１ Similarly, in module number m=2, NNC2321, SVL2322, ACM2323, INV2324, INV －１ 2325 and GSVL#2326.

[0099] As shown in the figure, the column data of the dimension “Age” is NNC2401, SVL2402, ACM2403, INV2404, and INV2405 in module number m=0. －１Similarly, in module number m=1, NNC2411, SVL2412, ACM2413, INV2414, INV －１ Similarly, in module number m=2, NNC2421, SVL2422, ACM2423, INV2424, INV －１ 2425 and GSVL#2426.

[0100] 22 to 24 are diagrams illustrating an example of the search process in Example 2. The search condition is (Name, Age) = (Name > "C", Age > 20). The hit section was found using the same method as in Example 1.

[0101] The example in Fig. 22 is for the case where the module number is 0. First, as shown by the arrows in the figure, INV[2]=3 for the dimension "Name", and INV[3]=4 for the dimension "Age". －１ Since INV[3]=2, it is included in the Hit section. Therefore, an array element Hit[0]=3 is added to the array 3000 that realizes the Hit column. Also, INV[3]=2 for the dimension "Name", INV[0]=2 for the dimension "Age", and INV[1]=2 for the dimension "Age". －１ Since [2]=0, it is not included in the Hit section, and nothing is added to the array 3000 that realizes the Hit string.

[0102] The example in Fig. 23 is for the case where the module number is 1. First, as shown by the arrows in the figure, INV[1]=2 for the dimension "Name", and INV[2]=3 for the dimension "Age". －１ Since [2]=3, it is included in the Hit section. Therefore, an array element Hit[0]=2 is added to the array 3001 that realizes the Hit sequence. Also, INV[2]=0 for the dimension "Name", and INV[2]=0 for the dimension "Age". －１ Since [0]=2, it is included in the Hit section. Therefore, an array element Hit[1]=0 is added to the array 3001 that realizes the Hit column. Furthermore, INV[3]=3 for the dimension "Name", and INV[4]=3 for the dimension "Age". －１Since [3]=1, it is not included in the Hit section, and nothing is added to the array 3001 that realizes the Hit sequence.

[0103] The example in Fig. 24 is for the case where the module number is 2. First, as shown by the arrow in the figure, INV[2]=2 for the dimension "Name", and INV[3]=2 for the dimension "Age". －１ Since INV[2]=0, it is not included in the Hit section, and nothing is added to the array 3002 that realizes the Hit sequence. Also, INV[3]=0 for the dimension "Name", and INV[4]=0 for the dimension "Age". －１ Since [0]=3, it is included in the Hit section. Therefore, an array element Hit[0]=0 is added to the array 3002 that realizes the Hit string.

[0104] 25 is a diagram showing an example of a search result in Example 2. The search result is array 3010, which is data contained in arrays 3000, 3001, and 3002 that realize the respective Hit sequences when the module numbers are 0, 1, and 2, respectively. The search result response unit 212 transmits the record numbers contained in array 3010 to the user device 10 as the search result.

[0105] Furthermore, when a target value such as (Name, Age) = ("Cat" ± 2 bits, 20 ± 2 bits) is given, the hit interval on the GSVL is identified using GSVL and GACM. Next, each module references the hit interval on the GSVL using GSVL#, and sets the hit interval on the SVL. Next, the hit interval on the INV is set using ACM. Specifically, similar multidimensional data can be searched for using the procedure described in <Example 2>.

[0106] <Summary> The search device 20 according to this embodiment can solve the above-mentioned problems by quickly calculating the similarity of each dimension and by reducing the number of vectors extracted from tabular data using a freely rearrangeable multidimensional index. Furthermore, the search device 20 according to this embodiment can calculate the similarity in steps O(1) in order notation, enabling efficient searches using the mechanism of a freely rearrangeable multidimensional index.

[0107] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0108] This application is based on basic application No. 2024-086854, filed in Japan on May 29, 2024, the entire contents of which are incorporated herein by reference.

[0109] REFERENCE SIGNS LIST 1 Search system 10 User device 20 Search device 30 Communication network

Claims

1. A search device comprising: a dataset consisting of multidimensional data; a first array that stores, for each dimension, the data numbers of values ​​that appear in the dimension of the dataset in ascending order of the values; a second array in which the subscripts and elements of the first array are swapped; and means for calculating the values ​​that appear in each dimension and the occurrence probability of each dimension, wherein, when similarity is included in the search conditions for a dimension, a hit interval is set on the first array using the values ​​and the occurrence probability of each dimension, and data numbers that are commonly included in the hit interval between dimensions specified in the search conditions are identified based on the second array.

2. The search device of claim 1, which identifies a value of a specified dimension from the identified data number, and determines a new hit section and identifies data numbers commonly included in the new hit section based on new search conditions that specify the identified value of the specified dimension.

3. A search method comprising: a dataset consisting of multidimensional data; a first array that stores, for each dimension, the data numbers of values ​​that appear in the dimension of the dataset in ascending order of the values; a second array in which the subscripts and elements of the first array are swapped; and means for calculating the values ​​that appear in each dimension and the occurrence probability of each dimension, wherein, when similarity is included in the search criteria for a dimension, a hit interval is set on the first array using the values ​​and the occurrence probability of each dimension, and data numbers that are commonly included in the hit interval between dimensions specified in the search criteria are identified based on the second array.

4. A program for causing a computer having: a dataset consisting of multidimensional data; a first array that stores, for each dimension, the data numbers of values ​​that appear in the dimension of the dataset in ascending order of the values; a second array in which the subscripts and elements of the first array are swapped; and means for calculating the values ​​that appear in each dimension and the occurrence probability of each dimension, to execute a process in which, when similarity is included in the search conditions for a dimension, a hit interval is set on the first array using the values ​​and the occurrence probability of each dimension, and data numbers that are commonly included in the hit interval between dimensions specified in the search conditions are identified based on the second array.

5. A search device that calculates similarity for multidimensional data based on the co-occurrence probability of values ​​specified as search conditions in one or more dimensions to be searched out of each dimension of multidimensional data that constitutes a data set, and searches for similar data from the data set based on the similarity.

6. The search device according to claim 5, wherein the co-occurrence probability is calculated based on a section that satisfies the search condition in a third array that stores values ​​that appear in the corresponding dimension in the data set in ascending order.

7. A search device as described in claim 6, which converts the interval on the third array into an interval on the first array that satisfies the search condition based on a first array that stores data numbers of values ​​that appear in the dimension in ascending order of values ​​stored in the third array for the corresponding dimension.

8. A search device according to any one of claims 5 to 7, which calculates the logarithm of the co-occurrence probability in each of the one or more dimensions to be searched as the similarity of the corresponding dimension, and calculates the sum of the similarities of each of the one or more dimensions as the similarity for the multidimensional data.

9. A search method in which a computer executes the following process: calculates a similarity for the multidimensional data based on the co-occurrence probability of values ​​specified as search conditions in one or more dimensions to be searched out of each dimension of the multidimensional data that constitutes a data set; and searches for similar data from the data set based on the similarity.

10. A program that causes a computer to execute the following process: calculate similarity for multidimensional data based on the co-occurrence probability of values ​​specified as search conditions in one or more dimensions of multidimensional data that make up a data set, and search for similar data from the data set based on the similarity.

Images