Information processing device, information processing method, and information processing program
By generating groups for classifying objects and calculating approximate distances, the information processing device enhances search efficiency and reduces processing time in information retrieval systems.
Patent Information
- Application Number
- JP2022129735
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-08-16
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2042-08-16
AI Technical Summary
Conventional information search technologies are limited in efficiency, and further improvements are needed to enhance search processing speed and accuracy.
The information processing device generates groups for classifying objects using first and second group information, acquired through an acquisition unit, to facilitate efficient search processing by calculating approximate distances between nodes in a graph structure.
This approach enables faster and more efficient search processing by reducing the time required for distance calculations and optimizing memory usage, while maintaining search accuracy.
Smart Images

Figure 0007948791000002 
Figure 0007948791000003 
Figure 0007948791000004
Abstract
Description
Technical Field
[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.
Background Art
[0002] Conventionally, techniques for searching (retrieving) various types of information have been provided. For example, a technique is provided that generates a graph in which nodes corresponding to search targets are connected by edges, and performs a search using the generated graph. In addition, such a technique is used for, for example, image search and the like.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Patent Document 2
Patent Document 3
Non-Patent Documents
[0004]
Non-Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] However, the conventional technologies described above have room for improvement in terms of search efficiency. For example, the conventional technologies improve efficiency by adjusting the number of edges in the graph to change it into a graph that allows for efficient searching, thereby reducing search processing time. However, there are limits to how much efficiency can be achieved by changing the graph. Therefore, in order to generate information that can be used for efficient search processing, such as speeding up the search process, it is desirable to introduce the concept of groups that classify the objects to be searched and to generate groups that classify the objects.
[0006] This application has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that generate groups for classifying objects. [Means for solving the problem]
[0007] The information processing device according to the present application is characterized by comprising: an acquisition unit that acquires object information indicating a plurality of objects that are the subject of data retrieval; a generation unit that generates, based on the object information acquired by the acquisition unit, first group information indicating a plurality of first groups which are a first number of groups that classify the plurality of objects; and second group information indicating a plurality of second groups which are groups that have a different classification from the plurality of first groups and are a second number of groups which are greater than the first number that classify the plurality of objects. [Effects of the Invention]
[0008] According to one embodiment, the effect is to generate groups for classifying objects. [Brief explanation of the drawing]
[0009] [Figure 1] Figure 1 shows an example of information processing according to the first embodiment. [Figure 2] Figure 2 shows an example of data according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of data according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating a configuration example of an information processing system according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating a configuration example of an information processing apparatus according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of an object information storage unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a graph information storage unit according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a quantization information storage unit according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a codebook information storage unit according to the first embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of information processing according to the first embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of search processing according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of information processing according to the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating a configuration example of an information processing apparatus according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a quantization information storage unit according to the second embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of a blob information storage unit according to the second embodiment. [Figure 16] FIG. 16 is a flowchart illustrating an example of search processing according to the second embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of information processing according to a modification. [Figure 18] FIG. 18 is a diagram illustrating an example of a graph information storage unit according to the modification. [Figure 19] FIG. 19 is a flowchart illustrating an example of search processing according to the modification. [Figure 20] Figure 20 is a diagram showing another example of information processing according to a modification. [Figure 21] Figure 21 is a diagram showing an example of blob information generation processing. [Figure 22] Figure 22 is a flowchart showing an example of information processing according to a modification. [Figure 23] Figure 23 is a diagram showing an example of information processing according to the third embodiment. [Figure 24] Figure 24 is a diagram showing a configuration example of an information processing apparatus according to the third embodiment. [Figure 25] Figure 25 is a diagram showing an example of a blob connected graph information storage unit according to the third embodiment. [Figure 26] Figure 26 is a flowchart showing an example of information processing according to the third embodiment. [Figure 27] Figure 27 is a flowchart showing an example of search processing according to the third embodiment. [Figure 28] Figure 28 is a flowchart showing an example of search processing according to the third embodiment. [Figure 29] Figure 29 is a diagram showing an example of blob connected graph information according to a modification. [Figure 30] Figure 30 is a diagram showing an example of a blob information storage unit according to a modification. [Figure 31] Figure 31 is a diagram showing an example of a blob connected graph information storage unit according to a modification. [Figure 32] Figure 32 is a diagram showing an example of information processing according to the fourth embodiment. [Figure 33] Figure 33 is a diagram showing a configuration example of an information processing apparatus according to the fourth embodiment. [Figure 34] Figure 34 is a conceptual diagram relating to quantization according to the fourth embodiment. [Figure 35] Figure 35 is a diagram showing an example of an inverted index according to the fourth embodiment. [Figure 36] Figure 36 is a diagram showing a specific example of an inverted index according to the fourth embodiment. [Figure 37] Figure 37 is a flowchart showing an example of information processing according to the fourth embodiment. [Figure 38] Figure 38 is a flowchart showing an example of information processing according to the fourth embodiment. [Figure 39] Figure 39 shows an example of information processing according to the fifth embodiment. [Figure 40] Figure 40 shows an example of information processing according to the fifth embodiment. [Figure 41] Figure 41 shows an example of the configuration of an information processing device according to the fifth embodiment. [Figure 42] Figure 42 is a flowchart showing an example of information processing according to the fifth embodiment. [Figure 43] Figure 43 is a flowchart showing an example of the search process according to the fifth embodiment. [Figure 44] Figure 44 shows an example of information processing according to the sixth embodiment. [Figure 45] Figure 45 shows an example of the configuration of an information processing device according to the sixth embodiment. [Figure 46] Figure 46 shows an example of an inverted index according to the sixth embodiment. [Figure 47] Figure 47 is a flowchart showing an example of information processing according to the sixth embodiment. [Figure 48] Figure 48 is a flowchart showing an example of a search process according to the sixth embodiment. [Figure 49] Figure 49 shows an example of the generation of the first and second groups. [Figure 50] Figure 50 shows an example of the generation of the first and second groups. [Figure 51] Figure 51 shows an example of the generation of the first and second groups. [Figure 52] Figure 52 is a flowchart showing an example of information processing according to the sixth embodiment. [Figure 53]Figure 53 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]
[0010] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing apparatus, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.
[0011] (Embodiment) [1. First Embodiment] [1-1. Information Processing] An example of information processing according to the first embodiment will be explained using Figure 1. Figure 1 is a diagram showing an example of information processing according to the first embodiment. The information processing device 100 performs a search using a graph index (also simply called a "graph") which is a graph structure of multiple objects that are the target of data retrieval. Figure 1 shows a part of the search process when the information processing device 100 performs a nearest neighbor search using a graph in which nodes corresponding to each vector, which are vectorized objects that are the target of data retrieval, are connected by edges. The information processing device 100 uses each node of the graph as the object to be searched and searches for neighboring nodes of the given search query (vector) by traversing the graph. Through the search process, the information processing device 100 extracts a predetermined number of nodes (hereinafter also called the "search number"), which is the number of neighboring nodes to be extracted, as neighboring nodes of the search query. In the following explanation, the case in which image information is the target of data retrieval will be explained as an example, but the target of data retrieval may be various objects such as video information or audio information.
[0012] The information processing device 100 performs search processing on nodes corresponding to a vast amount of image information, ranging from millions to hundreds of millions of units, but only a portion of it (tens of nodes such as node N1 in Figure 1) is shown in the diagram. For example, as shown in the spatial information SP1 in Figure 1, the information processing device 100 acquires information about multiple nodes (vectors) such as nodes N1, N7, N9, etc. When it is written as "node N* (* is an arbitrary number)", it indicates that the node is identified by node ID "N*". For example, when it is written as "node N1", it is identified by node ID "N1". Each of the white circles (〇) in the spatial information SP1 represents a node.
[0013] In the spatial information SP1 shown in Figure 1, nodes mainly related to explanation are labeled, but each of the unlabeled white circles (〇) is also a node, and there are many more nodes than those shown. Furthermore, each node corresponds to each object (search target). For example, each of the multiple local features extracted from the image may be an object. Also, for example, various data with defined distances between objects may be objects.
[0014] For example, the spatial information SP1 in Figure 1 may be a Euclidean space. For instance, the spatial information SP1 may correspond to the number of dimensions of the object's vector and be a multidimensional space such as 100-dimensional or 1000-dimensional. In Figure 1, the state in which the vector (space) is divided into four sub-regions (subspaces) by direct product quantization, as shown in the spatial information SP1, is conceptually illustrated, but the direct product quantization will be discussed later.
[0015] In spatial information SP1, the dotted lines connecting the white circles (〇) that represent nodes indicate edges that link nodes together. In the example in Figure 1, for the sake of simplicity, we show the case where nodes are connected by undirected (bidirectional) edges (hereinafter simply referred to as "edges"). Here, an undirected edge means an edge that allows data to be traversed bidirectionally between connected nodes. For example, the white circle (〇) representing node N1 and the white circle (〇) representing node N4 are connected by a dotted line, indicating that node N1 and node N4 are connected by an edge. In other words, in the graph shown in spatial information SP1, it is possible to traverse bidirectionally between node N1 and node N4. Specifically, in spatial information SP1, it is possible to traverse from node N1 to node N4, and from node N4 to node N1.
[0016] In the example in Figure 1, for illustrative purposes, only the edges connecting the illustrated nodes are shown, but there are many other edges besides those shown. Thus, in the spatial information SP1 in Figure 1, only a portion of the edges are shown, but it is assumed to be, for example, a k-nearest neighbor graph. Note that the spatial information SP1 may be any type of graph.
[0017] Furthermore, graph edges are not limited to undirected edges; they can also be directed edges. In the case of directed edges, they can only be traversed from the node that references the directed edge to the node that references it. For example, if two nodes are connected by two directed edges—a first edge that references one node and the other the destination, and a second edge that references one node and the other the destination—then the two nodes are in the same state as if they were connected by an undirected (bidirectional) edge.
[0018] From here, the search process will be explained using Figure 1. In the example in Figure 1, it is assumed that the information processing device 100 has already acquired a graph GR1 as shown in the spatial information SP1. Note that the information processing device 100 may generate the graph GR1 using various conventional techniques as appropriate. First, before explaining the search process, we will explain the Cartesian product quantization of vectors (spaces).
[0019] Figure 1 conceptually illustrates the state in which a vector (space) is divided into four subspaces AR11 to AR14 by direct product quantization, as shown in spatial information SP1. The subspaces AR11 to AR14 shown in Figure 1 are conceptual diagrams for explaining the distances between vectors of each node (object), but each subspace AR11 to AR14 is a multidimensional space. For example, the subspace AR11 shown in Figure 1 is illustrated in a two-dimensional form because it is illustrated on a plane, but it is a multidimensional space such as 100-dimensional or 1000-dimensional.
[0020] Each of the subspaces AR11 to AR14 represents the space corresponding to each of the four divisions (subvectors) of a vector that has been divided into four parts. For example, a 100-dimensional vector can be divided into 100 parts. For example, subspace AR11 represents the space (also called the "first subspace") of the dimension corresponding to the first subvector (also called the "first subvector") of the four subvectors (also called "sub-vectors") obtained by dividing a vector into four parts. Subspace AR11 represents the subspace (first subspace) corresponding to the division position at the beginning of the vector. In the case of search query QE1, subspace AR11 is the space corresponding to the first subquery QE1-1, which is the first subvector (first subvector).
[0021] Furthermore, subspace AR12 represents the space (also called the "second sub-space") of the dimension corresponding to the second sub-vector from the beginning (also called the "second sub-vector") when the vector is divided into four sub-vectors. Subspace AR12 represents the subspace (second sub-space) corresponding to the second division position from the beginning of the vector. In the case of search query QE1, subspace AR12 is the space corresponding to the second sub-query QE1-2, which is the second sub-vector from the beginning (second sub-vector).
[0022] Furthermore, subspace AR13 represents the space (also called the "third sub-space") of the dimension corresponding to the third sub-vector from the beginning (also called the "third sub-vector") when the vector is divided into four sub-vectors. Subspace AR13 represents the subspace (third sub-space) corresponding to the third division position from the beginning of the vector. In the case of search query QE1, subspace AR13 is the space corresponding to the third sub-query QE1-3, which is the third sub-vector (third sub-vector) from the beginning.
[0023] Furthermore, subspace AR14 represents the space of the dimension corresponding to the fourth subvector from the beginning (i.e., the last subvector) (also called the "fourth subvector") when the vector is divided into four subvectors. Subspace AR14 represents the subspace (fourth subspace) corresponding to the fourth division position from the beginning of the vector. In the case of search query QE1, subspace AR14 is the space corresponding to the fourth subquery QE1-4, which is the fourth subvector (fourth subvector) from the beginning.
[0024] In the example in Figure 1, subspaces AR11 to AR14 are shown with similar shapes, but the shapes of each subspace AR11 to AR14 may be different, and the way the regions are divided in each subspace AR11 to AR14 may also be different. Furthermore, the connection relationships between nodes by edges are assumed to be common to subspaces AR11 to AR14. That is, graph GR1 is a graph corresponding to spatial information SP1 and is common to subspaces AR11 to AR14.
[0025] In addition, in the example in Figure 1, a lookup table is generated for each divided subvector (subvector). For example, the four subvectors, the first subvector, the second subvector, the third subvector, and the fourth subvector, are clustered, and a separate lookup table (codebook information) is generated for each.
[0026] In Figure 1, the first subvectors are clustered into nine groups corresponding to codebooks CD11 to CD19, as shown in the subspace AR11, and a representative vector (centroid) is calculated for each of the codebooks CD11 to CD19. For example, the first subvectors of nodes N7, N9, etc. are clustered into the group corresponding to codebook CD11. In this case, the first subvectors of nodes N7, N9, etc. are vector quantized to the vector of codebook CD11 using the codebook information corresponding to the first subvector (also called "first codebook information") in the distance calculation.
[0027] Furthermore, the second sub-vectors are clustered into multiple groups corresponding to codebooks CD21-CD24, etc. (see Figures 2 and 9), and a representative vector (centroid) is calculated for each of the codebooks CD21-CD24, etc. In this case, the second sub-vector of each node is vector-quantized into the vector of the corresponding codebook using the codebook information corresponding to the second sub-vector (also called "second codebook information") during distance calculation.
[0028] For the third sub-vectors, they are clustered into multiple groups corresponding to codebooks CD31-CD34, etc. (see Figures 2 and 9), and a representative vector (centroid) is calculated for each of the codebooks CD31-CD34, etc. In this case, the third sub-vector of each node is vector-quantized into the vector of the corresponding codebook using the codebook information corresponding to the third sub-vector (also called "third codebook information") in the distance calculation (calculation).
[0029] For the fourth sub-vector, it is clustered into multiple groups corresponding to codebooks CD41-CD44, etc. (see Figures 2 and 9), and a representative vector (centroid) corresponding to each of the codebooks CD24-CD44, etc. is calculated. In this case, the fourth sub-vector of each node is vector-quantized into the vector of the corresponding codebook using the codebook information corresponding to the fourth sub-vector (also called "fourth codebook information") in the distance calculation (calculation).
[0030] The generation of codebook information, including the process of finding representative vectors, is carried out using various techniques as appropriate. Since the generation of codebook information is based on conventional techniques, a detailed explanation will be omitted. Furthermore, the lookup table is not limited to the above; for example, a single lookup table (codebook information) may be used for the entirety of the divided subvectors (sub-vectors), but this point will be discussed later.
[0031] From here, we will explain the search process targeting the search query QE1. First, the information processing device 100 obtains the search query QE1 (step S11). For example, the information processing device 100 obtains the search query QE1 from the terminal device 10 used by the user (see Figure 4).
[0032] The information processing device 100 then divides the vector, which is the search query QE1, into four parts. That is, the information processing device 100 divides the search query QE1 into four subqueries. In Figure 1, the information processing device 100 divides the search query QE1 into four subvectors: the first subquery QE1-1, which is the first subvector; the second subquery QE1-2, which is the second subvector; the third subquery QE1-3, which is the third subvector; and the fourth subquery QE1-4, which is the fourth subquery. Specifically, the information processing device 100 designates "45,23,2…" as the first subquery QE1-1, "127,34,5…" as the second subquery QE1-2, "20,98,110…" as the third subquery QE1-3, and "12,45,4…" as the fourth subquery QE1-4.
[0033] The information processing device 100 then calculates the distance between each sub-vector of the search query QE1 and the corresponding codebook vector. The information processing device 100 calculates the distance (difference) between each sub-vector of the search query QE1 and each codebook vector, as shown in the codebook information TB1 to TB4 in Figure 2. Figure 2 is a diagram showing an example of data according to the first embodiment.
[0034] The codebook information TB1 indicates the first codebook information corresponding to the first subvector, and the information processing device 100 calculates the distance between each vector of codebooks CD11 to CD19 corresponding to the first subvector and the first partial query QE1-1. In Figure 2, as shown in the codebook information TB1, the information processing device 100 calculates the distance between codebook CD11 and the first partial query QE1-1 as distance DS11. Similarly, the information processing device 100 calculates the distances between each of the codebooks CD12 to CD14, etc. and the first partial query QE1-1 as distances DS12 to DS14, etc.
[0035] Similarly, each of the codebook information TB2 to TB4 represents the second to fourth codebook information corresponding to each of the second to fourth subvectors. The information processing device 100 calculates the distance between each of the codebooks CD21 to CD24, etc. and the second subquery QE1-2 as distances DS21 to DS24, etc., as shown in the codebook information TB2. The information processing device 100 calculates the distance between each of the codebooks CD31 to CD34, etc. and the third subquery QE1-3 as distances DS31 to DS34, etc., as shown in the codebook information TB3. The information processing device 100 calculates the distance between each of the codebooks CD41 to CD44, etc. and the fourth subquery QE1-4 as distances DS41 to DS44, etc., as shown in the codebook information TB4.
[0036] Note that while Figure 2 shows an abstract representation for illustrative purposes, distances DS11 to DS44, etc., are assumed to be specific values. Distances are expressed as floating-point values, but may be compressed to a 1-byte integer type using scale-offset-compression (see "https: / / www.unidata.ucar.edu / blogs / developer / entry / compression_by_scaling_and_offfset", etc.) for speedup using SIMD (Single Instruction, Multiple Data) as described later. The information processing device 100 may calculate the distance between each codebook and the query when it obtains the search query QE1, or when the information is needed. In the search process, the information processing device 100 uses the codebook information TB1 to TB4 as a lookup table to calculate the distance between each node and the search query QE1, i.e., the approximate distance (also called the "first distance"). Thus, in the search process, the information processing device 100 can perform efficient search processing by calculating and processing an approximate distance (first distance) rather than the true distance (also called the "second distance") between each node and the search query when searching the graph.
[0037] The information processing device 100 performs a search operation targeting the search query QE1 (step S12). The information processing device 100 obtains the search results for the search query QE1 by performing a search operation targeting the search query QE1 as shown in Figure 11 using graph GR1. Details of the search operation shown in Figure 11 will be described later. By performing a search operation targeting the search query QE1, the information processing device 100 extracts nodes with a search count as neighboring nodes to the search query QE1.
[0038] In the search process targeting the search query QE1, the information processing device 100 selects a predetermined node from among the nodes to be the starting point (origin) of the search in graph GR1 (hereinafter also referred to as the "starting node"). For example, the information processing device 100 selects the starting node using a predetermined index, such as a tree structure index. In the example in Figure 1, the information processing device 100 selects node N7 as the starting node. Note that for simplicity of explanation, Figure 1 shows the case where only node N7 is selected as the starting node using a predetermined index, but the information processing device 100 may select multiple nodes as starting nodes, or may select the starting node using various methods such as randomness.
[0039] Here, in the search processing targeting the search query QE1, the information processing device 100 calculates the distance between the search query and the node to which an edge from one node is connected (hereinafter also referred to as a "connected node") by parallel processing (step S13). In Figure 1, as shown in the batch processing information LT1, the information processing device 100 calculates the distance to the search query QE1 in parallel for nodes N9, N12, N54, N85, etc., which are nodes to which an edge from node N7 is connected (connected nodes).
[0040] For example, node N9, as shown in node information INF1, has its first sub-vector associated with codebook CD12, its second sub-vector with codebook CD23, its third sub-vector with codebook CD35, and its fourth sub-vector with codebook CD47. Note that the size of the variable CD is determined by the size of the codebook; for example, if the codebook size is 16, the variable CD only needs to be 4 bits, allowing for significant data compression. Therefore, the information processing device 100 calculates the distance between node N9 and search query QE1 using the distances DS12 (codebook CD12), DS23 (codebook CD23), DS35 (codebook CD35), and DS47 (codebook CD47). For example, the information processing device 100 calculates the sum of the distances DS12 (codebook CD12), DS23 (codebook CD23), DS35 (codebook CD35), and DS47 (codebook CD47) as the distance between node N9 and search query QE1.
[0041] For example, node N12, as shown in node information INF2, has its first sub-vector associated with codebook CD14, its second sub-vector with codebook CD29, its third sub-vector with codebook CD31, and its fourth sub-vector with codebook CD45. For example, the information processing device 100 calculates the distance between node N12 and search query QE1 as the sum of the distances DS14 (codebook CD14), DS29 (codebook CD29), DS31 (codebook CD31), and DS45 (codebook CD45). Similarly, the information processing device 100 uses node information INF3 of node N54 to calculate the distance between node N54 and search query QE1, and uses node information INF4 of node N85 to calculate the distance between node N85 and search query QE1.
[0042] For example, the information processing device 100 calculates the distance between each of the nodes N9, N12, N54, and N85 and the search query QE1 simultaneously by parallelizing SIMD operations. This allows the information processing device 100 to speed up distance calculations by parallelizing them, enabling efficient search processing.
[0043] Note that Figure 1 shows a case where the distance is calculated in parallel for four nodes for the sake of simplicity, but the number of nodes to be parallelized is determined based on the specifications of the information processing device 100. For example, if the number of nodes that the information processing device 100 can process in a batch using SIMD (also called the "batch processing unit") is "4", the processing will be the same as in Figure 1. However, if the number of nodes that can be processed in a batch using SIMD (batch processing unit) is "16", the information processing device 100 will calculate the distance in parallel for 16 nodes. Also, if the batch processing unit is "32", the information processing device 100 will calculate the distance in parallel for 32 nodes. In this way, the information processing device 100 can perform efficient search processing by performing distance calculations for a number of nodes corresponding to the number of nodes that can be processed in a batch using SIMD (batch processing unit) in a batch.
[0044] The information processing device 100 performs distance calculations for multiple nodes in parallel as described above, and then performs a search process using graph GR1 as shown in Figure 11, targeting the search query QE1, thereby extracting nodes with a search count as neighboring nodes to the search query QE1.
[0045] As described above, the information processing device 100 can perform efficient search processing by calculating the approximate distance (first distance) between the vector of each vector-quantized node and the search query, and then performing the search processing using the first distance. Furthermore, the information processing device 100 can perform efficient search processing by performing distance calculations for a number of nodes that can be parallelized all at once. As described above, the information processing device 100 can speed up processing by calculating the approximate distance between a node in the graph and multiple connected nodes all at once, thereby repeatedly using a limited memory area (lookup table) (loading into the memory cache).
[0046] Furthermore, in the example shown in Figure 1, the information processing device 100 can perform search processing using vectors divided by Cartesian product quantization, enabling more efficient search processing compared to performing search processing without dividing the vectors. For example, the information processing device 100 calculates the distance between short vectors using Cartesian product quantization, thereby reducing the size of the vectors subject to distance calculation. In addition, the information processing device 100 can limit the memory space accessed by the lookup table used during distance calculation to the lookup table of the divided sub-vectors. The information processing device 100 can reduce the vector size while suppressing a decrease in search accuracy through Cartesian product quantization. Note that although Figure 1 illustrates the processing when Cartesian product quantization is performed, this is merely an example, and the information processing device 100 may also perform search processing using vectors that have not undergone Cartesian product quantization.
[0047] [1-1-1. Others] The process described above is merely one example, and the information processing device 100 may use various information and methods to perform search processing for efficient search. This point will be explained in detail below.
[0048] (Storage method for edge (connected node)) Regarding the information of edges (connected nodes) linked to each node, there are two possible storage methods (storage modes) for the edges (connected nodes) at the node, such as the following first and second storage modes. The storage mode may be selected depending on the usage.
[0049] For example, in the first storage configuration, each node may be associated with and stored with the ID of the connected node (object), and the information of the Cartesian product quantized object may be stored in a separate table. For example, the data storage configurations shown in Figures 7 and 8 correspond to the first storage configuration. In the first storage configuration, memory usage can be reduced.
[0050] Furthermore, in the second storage mode, for example, each node directly holds quantized objects. In the second storage mode, the quantized information of a node's connected nodes (objects) is stored in association with each node. For example, the second storage mode corresponds to the storage of data where the quantized information of nodes N9, N12, N54, and N85, which are connected nodes of node N7 (node information INF1 to INF4 in Figure 2), is stored in association with node N7. In the second storage mode, since objects are accessed sequentially during retrieval, performance degradation can be suppressed.
[0051] (Look-up table) In the example described above, we showed a case where each divided sub-vector is individually clustered and a lookup table (codebook information) is generated for each. However, the lookup table can be in any form, and this is not the only example.
[0052] For example, clustering may be performed on all subvectors to generate a single lookup table. In the example above, for instance, clustering may be performed on all four subvectors—the first subvector, the second subvector, the third subvector, and the fourth subvector—to generate a single lookup table.
[0053] Alternatively, lookup tables may be generated for each of the multiple classes by partially merging individual subvectors. For example, subvectors with similar variances may be merged to form multiple classes, and a lookup table may be generated for each of the multiple classes. In the example above, for example, if the variances of the first and third subvectors are similar, and the variances of the second and fourth subvectors are similar, the first and third subvectors may be merged to form the first class, and the second and fourth subvectors may be merged to form the second class. In this case, two lookup tables may be generated: a lookup table for the first class (codebook information) and a lookup table for the second class (codebook information).
[0054] (Transposition) The data retention described above is merely one example, and the information processing device 100 may retain data in various ways to enable efficient data retrieval, etc. For example, the information processing device 100 may retain transposed data. This point will be explained using Figure 3. Figure 3 is a diagram showing an example of data according to the first embodiment.
[0055] For example, if data is stored as shown in node information INF1 to INF4 and node information INF1 to INF4 is processed sequentially, the lookup tables codebook information TB1 to TB4 will be repeatedly referenced, as shown in step S14 of Figure 2.
[0056] Therefore, as shown in step S15, transposed data TR1 to TR4 are generated by transposing the node information INF1 to INF4. For example, the information processing device 100 generates transposed data TR1 to TR4 by transposing the node information INF1 to INF4.
[0057] In Figure 3, the first data TR1 is generated, which is a list of codebooks corresponding to the first sub-vectors of nodes N9, N12, N54, and N85 from the node information INF1 to INF4. Specifically, the first data TR1 is generated, which is a list of codebooks CD12 corresponding to the first sub-vector of node N9, CD14 corresponding to the first sub-vector of node N12, CD13 corresponding to the first sub-vector of node N54, and CD18 corresponding to the first sub-vector of node N85.
[0058] Similarly, the second data TR2 is generated, which is a list of codebooks corresponding to the second sub-vectors of nodes N9, N12, N54, and N85 from the node information INF1 to INF4. The third data TR3 is also generated, which is a list of codebooks corresponding to the third sub-vectors of nodes N9, N12, N54, and N85 from the node information INF1 to INF4. Finally, the fourth data TR4 is generated, which is a list of codebooks corresponding to the fourth sub-vectors of nodes N9, N12, N54, and N85 from the node information INF1 to INF4.
[0059] Furthermore, mapping information is generated that indicates which data point in each list of transposed data TR1 to TR4 corresponds to which node. In the example in Figure 3, mapping information is generated that indicates that the first data point in each list of transposed data TR1 to TR4 corresponds to node N9, the second data point corresponds to node N12, the third data point corresponds to node N54, and the fourth data point corresponds to node N85. By referring to the mapping information, the information processing device 100 can identify which node each data point in each list of transposed data TR1 to TR4 corresponds to. For example, the information processing device 100 generates mapping information.
[0060] The information processing device 100 processes data using transposed data TR1 to TR4, which are obtained by transposing node information INF1 to INF4. For example, when the information processing device 100 refers to a lookup table using transposed data TR1, it will only refer to codebook information TB1, thus referring to only one piece of codebook information TB1. Similarly, when the information processing device 100 refers to a lookup table using transposed data TR2, it will only refer to codebook information TB2, thus referring to only one piece of codebook information TB2. The same applies to transposed data TR3 and TR4, where only one piece of codebook information is referred to. As a result, the information processing device 100 can efficiently refer to data, enabling efficient search processing.
[0061] In this way, by transposing the data of neighboring objects (connected nodes) of a node and storing the data in the node in the order grouped by subclass, similar to how lookup tables for each subclass (subvector, etc.) are referenced during batch distance calculation, the locality of reference is further increased, enabling faster processing.
[0062] Note that Figure 3 shows the generation of transposed data for four nodes for simplicity of explanation, but the unit (number of nodes) for generating transposed data is determined based on the specifications of the information processing device 100. For example, if the batch processing unit of the information processing device 100 is "4", the processing will be the same as in Figure 3, but if the batch processing unit is "16", transposed data will be generated with 16 nodes as one unit. Also, if the batch processing unit is "32", transposed data will be generated with 32 nodes as one unit. In this way, by using transposed data generated according to the number of items that the information processing device 100 can process in batch using SIMD (batch processing unit), the information processing device 100 can efficiently refer to the data, thus enabling efficient search processing.
[0063] For example, much of the time spent on the search process described above is spent on distance calculation, but distance calculation can be sped up by parallel computing using SIMD as described above. When distance calculation is parallelized in this way, the time spent fetching object data becomes the main part of the search process. If this fetch time can be reduced, further speed improvements can be achieved. One way to reduce data fetching time is to perform prefetching, but there are limitations to this method.
[0064] On the other hand, the information processing device 100 can suppress data fetching and achieve further speed improvements by increasing the locality of reference as described above and enabling efficient data retrieval.
[0065] (Search results) Furthermore, when returning search results using the second distance (true distance) described above, two methods are possible: the first method and the second method described below.
[0066] For example, as a first method, the search may be performed with a search count greater than the specified search count, and at the end of the search, the true distance calculation may be performed and the results sorted by distance, with the specified search count objects being the search results. In this case, the information processing device 100 sets the extended search count (also called the "extended search count") as the search count so as to extract nodes that are greater than the specified search count (also called the "first number"), and performs the search processing shown in Figure 11 to extract nodes with the extended search count, i.e., nodes that are greater than the specified search count, as neighbor candidate nodes. The information processing device 100 then calculates the second distance (true distance) for the neighbor candidate nodes and extracts the nodes with the first number from among the neighbor candidate nodes that have the shortest second distance as neighbor nodes of the search query.
[0067] Alternatively, for example, a second method could be used: during the search, a second distance (true distance) could be calculated based on the first distance (approximate distance) only if the node (object) enters the search range or search area; the approximate distance could then be replaced with the true distance to return the search results based on the true distance.
[0068] Here, the search range is defined by "r" in Figure 11, and the exploration range is defined by "r(1+ε)" using the search range coefficient "ε" in Figure 11. For example, when calculating the second distance (true distance) when a node enters the search range, the information processing device 100 calculates the second distance (true distance) of a node when the first distance (approximate distance) of that node becomes less than or equal to "r". Also, when calculating the second distance (true distance) when a node enters the exploration range, the information processing device 100 calculates the second distance (true distance) of a node when the first distance (approximate distance) of that node becomes less than or equal to "r(1+ε)".
[0069] For example, to improve accuracy, the condition may be set to check if the result is within the search range. Similarly, to speed up processing, the condition may be set to check if the result is within the search range. Note that the above are merely examples, and the condition to be used may be set appropriately depending on the value of the search range coefficient "ε" and the purpose of the processing.
[0070] (Vector quantization) Furthermore, the information processing device 100 may perform two-stage vector quantization as shown in Patent Document 3. The information processing device 100 may also calculate the distance between the search query and each node using the following equation (1).
[0071]
number
[0072] Here, the value on the left side of equation (1) above represents, for example, the squared distance between the search query and the node. Also, for example, "x" in equation (1) above corresponds to the query. Also, for example, "y" in equation (1) above corresponds to the node. Also, for example, "q" on the right side of equation (1) above c"(y)" represents the representative vector (centroid) of "y". For example, if the information processing device 100 does not have vector data for the node for "y" in equation (1) above, it may use the numerical value of the centroid of the subregion to which each node belongs. Also, for example, "yq c (y) represents the residual vector. Also, for example, in the right-hand side of equation (1) above, "q p " indicates a predetermined quantizer (function).
[0073] Furthermore, for example, "j" in the right-hand side of equation (1) above may be the number of partitioned spaces. For example, in the example in Figure 1, "j" in the right-hand side of equation (1) above may be the number of partitioned spaces "4". Furthermore, for example, "u" in the right-hand side of equation (1) above j The parentheses () indicate the partial residual vectors between the vectors within the parentheses. For example, the information processing device 100 may use equation (1) above to calculate the squared distance between the query and the node in each subspace and sum them up to calculate the distance between the query and the node.
[0074] [1-2. Configuration of the Information Processing System] As shown in Figure 4, the information processing system 1 includes a terminal device 10, an information providing device 50, and an information processing device 100. The terminal device 10, the information providing device 50, and the information processing device 100 are connected to each other via a predetermined network N, either by wired or wireless means. Figure 4 is a diagram showing an example configuration of the information processing system according to the first embodiment. Note that the information processing system 1 shown in Figure 4 may include multiple terminal devices 10, multiple information providing devices 50, and multiple information processing devices 100.
[0075] Terminal device 10 is an information processing device used by the user. Terminal device 10 accepts various operations from the user. In the following, terminal device 10 may be referred to as the user. That is, in the following, the user can be read as terminal device 10. The above-mentioned terminal device 10 can be implemented as, for example, a smartphone, a tablet device, a notebook PC (Personal Computer), a desktop PC, a mobile phone, or a PDA (Personal Digital Assistant).
[0076] The information providing device 50 is an information processing device that stores information for providing various information to users, etc. For example, the information providing device 50 stores object IDs based on character information, etc., collected from various external devices such as web servers. For example, the information providing device 50 is an information processing device that provides image search services to users, etc. For example, the information providing device 50 stores various information for providing image search services. For example, the information providing device 50 provides vector information corresponding to images that are the target of the image search service to the information processing device 100. The information providing device 50 also receives object IDs, etc., indicating images corresponding to queries from the information processing device 100 by sending queries to the information processing device 100.
[0077] The information processing device 100 is a computer that provides a search service. The information processing device 100 provides a specified number of search objects corresponding to the search query as search results. The information processing device 100 uses a graph in which the nodes (objects) to be searched are connected by edges to provide nodes (objects) in the vicinity of the search query as search results. In the search process, which searches for nodes in the vicinity of the search query using a graph in which groups of nodes corresponding to each of multiple objects are connected by edges, the information processing device 100 calculates the distance between multiple nodes selected according to predetermined criteria and the search query using vector information of multiple nodes that have been vector-quantized.
[0078] For example, when the information processing device 100 receives a query (search query) from the terminal device 10, it searches for objects similar to the search query and provides the search results to the terminal device. Furthermore, the data that the information processing device 100 provides to the terminal device may be the data itself, such as image information, or it may be information for referencing corresponding data, such as a URL (Uniform Resource Locator). Also, the search query and the search target (object) may be any type of data, such as images, audio, or text data.
[0079] [1-3. Configuration of Information Processing Device] Next, the configuration of the information processing device 100 according to the first embodiment will be described using Figure 5. Figure 5 is a diagram showing an example of the configuration of the information processing device 100 according to the first embodiment. As shown in Figure 5, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also have an input unit (for example, a keyboard or mouse) that receives various operations from the administrator of the information processing device 100, and a display unit (for example, a liquid crystal display) for displaying various information.
[0080] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network (for example, network N in Figure 4) by wire or wireless connection and transmits and receives information with the terminal device 10 and the information providing device 50.
[0081] (Storage unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. As shown in Figure 5, the storage unit 120 according to the first embodiment includes an object information storage unit 121, a graph information storage unit 122, a quantization information storage unit 123, and a codebook information storage unit 124.
[0082] (Object information storage unit 121) The object information storage unit 121 according to the first embodiment stores various information about an object. For example, the object information storage unit 121 stores an object ID and vector data. Figure 6 shows an example of the object information storage unit according to the first embodiment. The object information storage unit 121 shown in Figure 6 includes items such as "object ID" and "vector information".
[0083] The "Object ID" indicates identification information used to identify an object. The "Vector Information" indicates vector information corresponding to the object identified by the Object ID. In other words, in the example in Figure 6, the vector data (vector information) corresponding to the object is registered and associated with the Object ID that identifies the object.
[0084] For example, in the example in Figure 6, the object (target) identified by object ID "OB1" is associated with multidimensional vector information of the form "10, 24, 51, 2...".
[0085] Furthermore, the object information storage unit 121 may store various types of information depending on the purpose, not limited to those mentioned above.
[0086] (Graph information storage unit 122) The graph information storage unit 122 according to the first embodiment stores various information related to the graph. For example, the graph information storage unit 122 stores the generated graph. Figure 7 shows an example of the graph information storage unit according to the first embodiment. The graph information storage unit 122 shown in Figure 7 has items such as "node ID", "object ID", and "connected node information".
[0087] The "Node ID" indicates identification information for identifying each node (object) in the graph. The "Object ID" indicates identification information for identifying an object. If the Node ID and Object ID are the same, the Object ID is stored in the "Node ID" field, and the "Object ID" field does not need to be included in the graph information storage unit 122. For example, if used as both Object ID and Node ID, the Object ID is stored in the "Node ID" field, and the "Object ID" field does not need to be included in the graph information storage unit 122.
[0088] Furthermore, "connection node information" indicates information about nodes that can be traced from the corresponding node (referenced nodes). For example, "connection node information" includes information such as "referenced node." "Referenced node" indicates information for identifying the referenced node that is connected by an edge and can be traced from that node. In other words, in the example in Figure 7, the referenced node that can be traced from that node by an edge is associated with the node ID (object ID) that identifies the node and is registered. Note that "connection node information" may also include information for identifying the edge connected to the referenced node (edge ID), etc.
[0089] In the example in Figure 7, the node identified by node ID "N1" (node N1) corresponds to the object (target) identified by object ID "OB1". Furthermore, an edge is connected from node N1 to the node identified by node ID "N4" (node N4), indicating that it is possible to traverse from node N1 to node N4.
[0090] Furthermore, the node identified by node ID "N2" (node N2) corresponds to the object (target) identified by object ID "OB2". Additionally, an edge is connected from node N2 to the node identified by node ID "N6" (node N6), indicating that it is possible to traverse from node N2 to node N6.
[0091] Furthermore, the node identified by node ID "N7" (node N7) corresponds to the object (target) identified by object ID "OB7". Additionally, edges connect node N7 to nodes N9, N12, N54, and N85, indicating that it is possible to traverse from node N2 to nodes N9, N12, N54, and N85 respectively.
[0092] The graph information storage unit 122 is not limited to the above and may store various types of information depending on the purpose. For example, the graph information storage unit 122 may store the lengths of the edges connecting each node (vector). That is, the graph information storage unit 122 may store information indicating the distance between each node (vector). The graph information storage unit 122 is not limited to the above and may store graph information using various data structures.
[0093] Furthermore, the graph may include a program module that takes a query as input, searches for nodes by traversing edges in the graph, and extracts and outputs nodes similar to the query. In other words, the graph may be intended for use as a program module that performs search processing using the graph. For example, graph GR1 may be a program that, when vector data is input as a query, extracts and outputs nodes from the graph that correspond to vector data similar to that vector data. For example, graph GR1 may be data used as a program module that searches for similar images corresponding to a query image. For example, graph GR1 causes the computer to function so that, based on the input query, it extracts and outputs nodes in the graph that are similar to that query.
[0094] (Quantization information storage unit 123) The quantization information storage unit 123 according to the first embodiment stores various information related to the allocation process. Figure 8 shows an example of the quantization information storage unit according to the first embodiment. In the example in Figure 8, the quantization information storage unit 123 has items such as "node ID", "object ID", and "quantization information".
[0095] The "Node ID" indicates identification information for identifying each node (object) in the graph. The "Object ID" indicates identification information for identifying an object. If the Node ID and Object ID are the same, the Object ID is stored in the "Node ID" field, and the "Object ID" field does not need to be included in the quantization information storage unit 123.
[0096] Furthermore, "quantization information" indicates the information of the quantized vector of each node (object). For example, "quantization information" includes information such as "elements" and "codebook ID". "Elements" indicate the placement of the corresponding object in the vector. In the example in Figure 8, "elements" include "#1", "#2", "#3", and "#4". In this case, the vector of each node (object) is divided into four parts, and each divided sub-vector is quantized by the codebook. Note that the number of divisions is not limited to four; for example, if the number of divisions is six, "elements" will include "#1", "#2", "#3", "#4", "#5", and "#6". "Codebook ID" indicates information for identifying the codebook corresponding to each element (sub-vector).
[0097] In the example in Figure 8, the vector of node N9 (object OB9) is shown to be quantized by the codebook (codebook CD12) identified by codebook ID "CD12" as the first of the four divided subvectors. Furthermore, the vector of node N9 (object OB9) is shown to be quantized by the codebook (codebook CD23) identified by codebook ID "CD23" as the second of the four divided subvectors.
[0098] Furthermore, the vector of node N9 (object OB9) is shown to be quantized by the codebook (codebook CD35) identified by codebook ID "CD35" for the third subvector from the beginning of the four divided subvectors. Furthermore, the vector of node N9 (object OB9) is shown to be quantized by the codebook (codebook CD47) identified by codebook ID "CD47" for the fourth subvector from the beginning (i.e., the last subvector) of the four divided subvectors.
[0099] Furthermore, the quantized information storage unit 123 may store various types of information depending on the purpose, not limited to those mentioned above.
[0100] (Codebook information storage unit 124) The codebook information storage unit 124 according to the first embodiment stores various information about codebooks. For example, the codebook information storage unit 124 stores codebook IDs and vector information for each codebook. Figure 9 is a diagram showing an example of the codebook information storage unit according to the first embodiment. In the example in Figure 9, the codebook information storage unit 124 stores a lookup table that shows the correspondence between each codebook and a vector.
[0101] The codebook information storage unit 124 stores codebook information TB1 used to quantize the first subvector of the four divided subvectors, and codebook information TB2 used to quantize the second subvector from the beginning of the four divided subvectors. The codebook information storage unit 124 also stores codebook information TB3 used to quantize the third subvector from the beginning of the four divided subvectors, and codebook information TB4 used to quantize the fourth (i.e., last) subvector from the beginning of the four divided subvectors, etc.
[0102] In the example in Figure 9, the codebook information TB1 stores codebook information such as the codebook identified by codebook ID "CD11" (codebook CD11) and the codebook identified by codebook ID "CD12" (codebook CD12). For example, codebook CD11 indicates that multidimensional vector information of "5,13..." is associated with it. Similarly, codebook CD12 indicates that multidimensional vector information of "27,51..." is associated with it.
[0103] The codebook information storage unit 124 is not limited to the above and may store various types of information depending on the purpose. For example, the codebook information storage unit 124 may store information indicating the difference (distance) between each codebook and the search query.
[0104] (Control unit 130) Returning to the explanation of Figure 5, the control unit 130 is a controller, and is realized by various programs (corresponding to an example of an information processing program) stored in the memory device inside the information processing device 100 being executed using RAM as the working area by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), etc. Furthermore, the control unit 130 is a controller, and is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), for example.
[0105] As shown in Figure 5, the control unit 130 includes an acquisition unit 131, a generation unit 132, a search processing unit 133, and a provision unit 134, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 5, and other configurations are also acceptable as long as they perform the information processing described later.
[0106] (Acquisition part 131) The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires various types of information from the storage unit 120. For example, the acquisition unit 131 acquires various types of information from the object information storage unit 121, the graph information storage unit 122, the quantization information storage unit 123, the codebook information storage unit 124, etc. The acquisition unit 131 also acquires various types of information from external information processing devices. The acquisition unit 131 acquires various types of information from the terminal device 10 and the information providing device 50.
[0107] The acquisition unit 131 acquires the graph. For example, the information processing device 100 may acquire spatial information SP1 in Figure 1. For example, the information processing device 100 may acquire the graph from an external device such as the information providing device 50.
[0108] The acquisition unit 131 acquires search queries for multiple objects that are the target of data retrieval. For example, the acquisition unit 131 acquires information about search query QE1. For example, the acquisition unit 131 acquires search queries related to image retrieval. For example, the acquisition unit 131 acquires queries from the terminal device 10 being used. For example, the acquisition unit 131 acquires queries from the information providing device 50 that received the query from the terminal device 10 being used.
[0109] (Generation unit 132) The generation unit 132 generates various types of information. For example, the generation unit 132 generates various types of information (data) from the information (data) stored in the storage unit 120. For example, the generation unit 132 generates various types of information from the information (data) stored in the object information storage unit 121, the graph information storage unit 122, the quantization information storage unit 123, the codebook information storage unit 124, etc.
[0110] The generation unit 132 may generate a graph as shown in the graph information storage unit 122. For example, the generation unit 132 generates spatial information SP1. The generation unit 132 may also generate information related to vector quantization as shown in the quantization information storage unit 123. For example, the generation unit 132 generates information in which each object, such as node N1 (object OB1), has been vector-quantized. The generation unit 132 may also generate information related to a codebook as shown in the codebook information storage unit 124. The generation unit 132 may also generate a lookup table for the codebook. For example, the generation unit 132 generates information related to a codebook, such as codebook information TB1 to TB4. Note that if the information processing device 100 obtains the information shown in the graph information storage unit 122, the quantization information storage unit 123, and the codebook information storage unit 124 from an external device such as the information providing device 50, the information processing device 100 does not need to have the generation unit 132.
[0111] (Search processing unit 133) The search processing unit 133 provides a search service for objects. The search processing unit 133 searches for various types of information. The search processing unit 133 searches for various types of information. For example, the search processing unit 133 searches for objects by searching a graph. For example, when a query is obtained by the acquisition unit 131, the search processing unit 133 searches for objects similar to the query by searching a graph. For example, the search processing unit 133 extracts objects similar to the query by searching a graph based on the processing procedure shown in Figure 11. Note that if the information processing device 100 does not provide a search service, it does not need to have a search processing unit 133.
[0112] The search processing unit 133 selects various information during the search process. The search processing unit 133 extracts various information during the search process. The search processing unit 133 determines various information during the search process. The search processing unit 133 decides on various information during the search process. The search processing unit 133 modifies various information during the search process. The search processing unit 133 updates various information during the search process.
[0113] The search processing unit 133 uses a graph in which groups of nodes corresponding to each of multiple objects are connected by edges to search for nodes near a search query. In this search process, it calculates the distance between multiple nodes selected according to predetermined criteria and the search query using vector information of multiple nodes that have been vector-quantized. The search processing unit 133 performs the calculation of the distance between multiple nodes and the search query in parallel and in a batch. The search processing unit 133 performs parallel processing of the calculation of the distance between multiple nodes and the search query for a batch processing number determined based on the specifications of the information processing device 100.
[0114] The search processing unit 133 calculates the distance between multiple nodes and the search query using a codebook to which each of the multiple nodes is associated with a corresponding representative vector. The search processing unit 133 calculates the distance between multiple nodes connected by edges from one node and the search query. The search processing unit 133 calculates the distance between multiple nodes and the search query using node information to which reference information indicating the representative vectors associated with each of the multiple nodes is stored and associated with one node.
[0115] The search processing unit 133 calculates the distance between multiple nodes and the search query using vector information of multiple nodes, each of which has been divided into multiple subvectors by Cartesian product quantization. The search processing unit 133 calculates the distance between multiple nodes and the search query using multiple codebooks associated with representative vectors corresponding to the division positions of the multiple subvectors into which each of the multiple nodes has been divided. The search processing unit 133 calculates the distance between multiple nodes and the search query using node information, in which reference information indicating the representative vectors to which each of the multiple subvectors of each of the multiple nodes is associated is stored and associated with a single node.
[0116] The search processing unit 133 calculates the distance between multiple nodes and the search query using reference information that associates a list of representative vectors, each of which is associated with each of the multiple subvectors of each node. The search processing unit 133 calculates the distance between multiple nodes and the search query using reference information that includes transpose information, where the representative vectors corresponding to each of the multiple subvectors of each node are listed by division position, and correspondence information indicating the corresponding position in the list for each of the multiple nodes. The search processing unit 133 calculates the distance between multiple nodes and the search query using a codebook that associates each of the representative vectors corresponding to each of the multiple subvectors of each node.
[0117] The search processing unit 133 calculates a second distance, which is not vector-quantized, for predetermined nodes among the nodes processed during the search process, unlike the first distance, which is vector-quantized. The search processing unit 133 extracts a second number of nodes, which is greater than the first number of nodes to be extracted as neighboring nodes of the search query, as neighbor candidate nodes, and calculates the second distance for these neighbor candidate nodes. The search processing unit 133 extracts the first number of nodes from among the neighbor candidate nodes, starting with those with the shortest second distance, as neighboring nodes of the search query.
[0118] The search processing unit 133 calculates a second distance for nodes whose first distance is within a predetermined threshold. The search processing unit 133 calculates a second distance for nodes within a search range that indicates the range to be extracted as neighboring nodes. The search processing unit 133 calculates a second distance for nodes within a search range that indicates the range to be searched.
[0119] (Provider 134) The information provision unit 134 provides various types of information. For example, the information provision unit 134 transmits various types of information to the terminal device 10 and the information provision device 50. For example, the information provision unit 134 provides the object ID corresponding to the search query as a search result. The information provision unit 134 transmits the search results to the terminal device 10. The information provision unit 134 provides the object ID retrieved by the search processing unit 133 to the terminal device 10 as a search result corresponding to the search query.
[0120] Furthermore, the providing unit 134 may provide the object IDs retrieved by the search processing unit 133 to the information providing device 50. For example, the providing unit 134 provides the object IDs extracted by the search processing unit 133 to the information providing device 50. The providing unit 134 provides the object IDs extracted by the search processing unit 133 to the information providing device 50 as information indicating a vector corresponding to the query.
[0121] [1-4. Information Processing Flow] Next, the information processing procedure by the information processing system 1 according to the first embodiment will be explained using Figure 10. Figure 10 is a flowchart showing an example of information processing according to the first embodiment.
[0122] As shown in Figure 10, the information processing device 100 obtains a search query for multiple objects that are the target of data retrieval (step S101). In the example in Figure 1, the information processing device 100 obtains the search query QE1.
[0123] Then, in a search process in which the information processing device 100 searches for neighboring nodes of a search query using a graph in which groups of nodes corresponding to each of multiple objects are connected by edges, the distance between the search query and multiple nodes selected according to predetermined criteria is calculated using the vector information of the multiple nodes which have been vector-quantized (step S102). In the example in Figure 1, the information processing device 100 calculates the distance between the multiple nodes N9, N12, N54, N85 and the search query QE1 using the vector information of the multiple nodes N9, N12, N54, N85 which have been vector-quantized.
[0124] [1-5. Search Processing Examples] Here, an example of the search process according to the first embodiment will be explained using Figure 11 as an example. Figure 11 is a flowchart of an example of the search process according to the first embodiment. The search process described below is performed by the search processing unit 133 of the information processing device 100. Also, the term "object" below may be read as "node". In the following, the information processing device 100 (search processing unit 133) performs the search process. Note that if a search service is not provided, the information processing device 100 does not need to have a search processing unit 133. The search query for the process described below may be an additional node, a target node, or an object specified by the user.
[0125] Here, the neighbor object set N(G,y) is the set of neighboring objects associated with node y by the edges assigned to it. "G" may be predetermined graph data (for example, graph GR1 shown in spatial information SP1). For example, the information processing device 100 performs a k-nearest neighbor search process.
[0126] For example, the information processing device 100 sets the radius r of the hypersphere to ∞ (infinity) (step S300) and extracts a subset S from the existing set of objects (step S301). For example, the information processing device 100 may extract the object (node) selected as the root node as the subset S. Also, for example, the hypersphere is a virtual sphere that indicates the search range. The objects included in the set of objects S extracted in step S301 are simultaneously included in the initial set of the search result set R.
[0127] Next, the information processing device 100 extracts the object with the shortest distance from object y to the search query object among the objects included in the object set S, and identifies it as object s (step S302). For example, if the only object (node) selected as the root node is an element of S, the information processing device 100 extracts the root node as object s. Next, the information processing device 100 removes object s from the object set S (step S303).
[0128] Next, the information processing device 100 determines whether the distance d(s,y) between object s and object y exceeds r(1+ε) (step S304). Here, ε is an extension element, and r(1+ε) is a value that indicates the radius of the search range (only nodes within this range are searched. The accuracy can be improved by making it larger than the search range). If the distance d(s,y) between object s and object y exceeds r(1+ε) (step S304: Yes), the information processing device 100 outputs the object set R as the set of neighboring objects of object y (step S305), and terminates the process.
[0129] If the distance d(s,y) between object s and search query object y does not exceed r(1+ε) (step S304: No), the information processing device 100 selects one object from among the objects that are elements of the neighboring object set N(G,s) of object s that is not included in object set C, and stores the selected object u in object set C (step S306). Object set C is provided for convenience to avoid duplicate searches and is set to an empty set at the start of processing.
[0130] Next, the information processing device 100 determines whether the distance d(u,y) between object u and object y is less than or equal to r(1+ε) (step S307). If the distance d(u,y) between object u and object y is less than or equal to r(1+ε) (step S307: Yes), the information processing device 100 adds object u to the object set S (step S308). If the distance d(u,y) between object u and object y is not less than or equal to r(1+ε) (step S307: No), the information processing device 100 performs the determination (processing) in step S309.
[0131] Next, the information processing device 100 determines whether the distance d(u,y) between object u and object y is less than or equal to r (step S309). If the distance d(u,y) between object u and object y exceeds r (step S309: No), the information processing device 100 performs the determination (processing) in step S315. That is, if the distance d(u,y) between object u and object y is not less than or equal to r, the information processing device 100 performs the determination (processing) in step S315.
[0132] If the distance d(u,y) between object u and object y is less than or equal to r (step S309: Yes), the information processing device 100 adds object u to the object set R (step S310). Then, the information processing device 100 determines whether the number of objects included in the object set R exceeds ks (step S311). The predetermined number ks is an arbitrarily determined natural number. For example, ks may be the number of search results or the number of objects to be extracted. Also, for example, if there is no upper limit on the number of objects to be extracted in a range search, ks may be set to infinity. For example, ks may be 4. If the number of objects included in the object set R does not exceed ks (step S311: No), the information processing device 100 performs the determination (processing) in step S313.
[0133] If the number of objects in the object set R exceeds ks (step S311: Yes), the information processing device 100 removes the object that is furthest from object y from the object set R (step S312).
[0134] Next, the information processing device 100 determines whether the number of objects in the object set R matches ks (step S313). If the number of objects in the object set R does not match ks (step S313: No), the information processing device 100 performs the determination (processing) in step S315. If the number of objects in the object set R matches ks (step S313: Yes), the information processing device 100 sets the distance between the object with the longest (farthest) distance from object y and object y as the new r (step S314).
[0135] Then, the information processing device 100 determines whether it has finished selecting all objects from the elements of the neighboring object set N(G,s) of object s and storing them in the object set C (step S315). If it has not finished selecting all objects from the elements of the neighboring object set N(G,s) of object s and storing them in the object set C (step S315: No), the information processing device 100 returns to step S306 and repeats the process.
[0136] If all objects have been selected from the objects that are elements of the neighbor object set N(G,s) of object s and stored in the object set C (step S315: Yes), the information processing device 100 determines whether the object set S is an empty set (step S316). If the object set S is not an empty set (step S316: No), the information processing device 100 returns to step S302 and repeats the process. If the object set S is an empty set (step S316: Yes), the information processing device 100 outputs the object set R and terminates the process (step S317). For example, the information processing device 100 may select objects (nodes) included in the object set R as neighbor nodes corresponding to the additional node (input object y). For example, the information processing device 100 may extract (select) objects (nodes) included in the object set R as neighbor nodes corresponding to the target node (input object y). Also, for example, the information processing device 100 may provide objects (nodes) included in the object set R as search results corresponding to the search query (input object y) to the terminal device that performed the search.
[0137] [2. Second Embodiment] Next, a second embodiment will be described. In the second embodiment, neighboring objects are grouped by clustering or the like, and each group (hereinafter also referred to as "blob") is the target for distance calculation as a whole. That is, in the second embodiment, distance calculation is performed as a whole on a blob basis. Points that are the same as in the first embodiment will be omitted from explanation as appropriate. In the second embodiment, the information processing system 1 has an information processing device 100A instead of the information processing device 100.
[0138] [2-1. Information Processing] First, an overview of the information processing according to the second embodiment will be explained using Figure 12. Figure 12 is a diagram showing an example of the information processing according to the second embodiment.
[0139] In the example shown in Figure 12, it is assumed that the information processing device 100A has already acquired a graph GR21 as shown in spatial information SP21. For example, the spatial information SP21 in Figure 1 may be a Euclidean space. For example, the spatial information SP21 may correspond to the number of dimensions of the object vector and be a multidimensional space such as 100 dimensions or 1000 dimensions. Note that in the example shown in Figure 12, the explanation is based on a figure where direct product quantization has not been performed for simplicity, but direct product quantization of the vector (space) may be performed as in Figure 1.
[0140] First, let's explain the information shown in Figure 12. In the spatial information SP21 of Figure 12, the dotted lines connecting the white circles (〇) which are nodes indicate the edges that connect the nodes. In Figure 12, as in Figure 1, undirected edges are shown as an example, but the edges of graph GR21 are not limited to undirected edges; directed edges may also be used. Note that the notes and edges are the same as in Figure 1, so a detailed explanation will be omitted.
[0141] In the spatial information SP21 shown in Figure 12, the regions enclosed by straight lines are areas where neighboring objects have been grouped together through clustering, and these regions are called blobs. In the examples shown below, blobs are the processing units for batch distance calculations. Figure 12 shows the case where the data has been clustered into 10 regions (blobs) BL1 to BL10. For example, blob BL1 is a blob to which multiple nodes such as N7, N9, N85, and N126 belong. The black dots in blobs BL1 to BL10 represent the centroids (representative vectors) of each blob.
[0142] Blobs BL1 to BL10 are generated using various clustering methods as appropriate. For example, k-means clustering can be used for classifying into blobs. Alternatively, instead of using the centroids obtained from the mean coordinates of each cluster in a single iteration (assignment) of k-means for the next iteration, the centroids can be replaced with the objects closest to each centroid before proceeding to the next iteration. In this case, the final cluster centroids obtained by k-means will be existing objects, i.e., nodes. This increases the tendency for clusters (blobs) and the edges (neighboring nodes) of each node to coincide, further improving search performance.
[0143] In the example shown in Figure 12, the information processing device 100A performs a search using the information of multiple classified blobs BL1 to BL10 by clustering multiple nodes (objects) as shown in the spatial information SP21.
[0144] From here, we will explain the search process targeting the search query QE2. First, the information processing device 100A obtains the search query QE2 (step S21). For example, the information processing device 100A obtains the search query QE2 from the terminal device 10 used by the user (see Figure 4).
[0145] The information processing device 100A then calculates the distance between the vector of the search query QE2 and the vector of the codebook. Although not shown in the diagram, if Cartesian product quantization is not performed, the information processing device 100A calculates the distance (difference) between the vector of the search query QE2 and the vector of the codebook used to quantize the entire vector. The information processing device 100A then associates the distance (difference) between the vector of the search query QE2 and the vector of each codebook with information for identifying each codebook and stores it as codebook information TB21. If Cartesian product quantization is performed, the information processing device 100A calculates the distance (difference) between each sub-vector of the search query QE2 and the vector of each codebook, as shown in the codebook information TB1 to TB4 in Figure 2, similar to Figure 1. The calculation of the distance between the search query vector and the codebook vector is the same as in Figures 1 and 2, so a detailed explanation is omitted.
[0146] The information processing device 100A performs a search operation targeting the search query QE2 (step S22). The information processing device 100A obtains the search results for the search query QE2 by performing a search operation targeting the search query QE2 as shown in Figure 16 using graph GR21. Details of the search operation shown in Figure 16 will be described later. By performing a search operation targeting the search query QE2, the information processing device 100A extracts nodes with a search count as neighboring nodes to the search query QE2.
[0147] In the search process targeting the search query QE2, the information processing device 100A selects a predetermined node from among the nodes to be the starting node (starting node) for the search of graph GR21. In the example in Figure 12, the information processing device 100A selects node N9 as the starting node. In the example in Figure 12, the information processing device 100A starts the search process, for example, with node N9 as the processing target.
[0148] Here, in the search processing targeting the search query QE2, the information processing device 100A calculates the distance between the search query and multiple nodes belonging to the blob to which the node to be processed belongs, in parallel (step S23). In Figure 12, as shown in the batch processing information LT2, the information processing device 100A calculates the distance to the search query QE2 in parallel for nodes N7, N9, N85, N126, etc., which belong to the blob BL1 to which the node to be processed N9 belongs.
[0149] The information processing device 100A calculates the distance between each node N7, N9, N85, and N126 and the search query QE2, using the distance between each codebook indicated by the codebook information TB21 and the search query QE2. For example, the information processing device 100A refers to the codebook information TB21 and calculates the distance of the codebook corresponding to the vector of node N7 as the distance between node N7 and the search query QE2. Note that the distance calculation when Cartesian product quantization is performed is the same as in Figures 1 and 2, so a detailed explanation is omitted.
[0150] The number of units that can be processed in a batch is determined based on the specifications of the information processing device 100A, similar to the case of the information processing device 100. For example, if the number of units that the information processing device 100A can process in a batch using SIMD (the number of units that can be processed in a batch) is "4", then, as in Figure 1, the distance is calculated in parallel for four nodes. For example, the information processing device 100A calculates the distance between each of the nodes N7, N9, N85, and N126 and the search query QE2 in a batch by parallelizing the SIMD calculation. In this way, the information processing device 100A can speed up distance calculation by parallelizing the distance calculation, enabling efficient search processing. Furthermore, the information processing device 100A uses graph GR21 to traverse the nodes to which the edge from node N7 is connected (for example, nodes N12 and N64), and performs a batch distance calculation for the blobs to which those nodes belong (for example, blobs BL2 and BL3) to perform the search processing. Furthermore, the information processing device 100A suppresses the repeated use of the same blob in the batch distance calculation during the search process, but details will be described later.
[0151] Note that Figure 12 shows a case where the distance is calculated in parallel for four nodes for the sake of simplicity, but the number of nodes to be parallelized is determined based on the specifications of the information processing device 100A. This point is the same as in Figures 1 and 2, so a detailed explanation is omitted. Also, the information processing device 100A may use transposed data, as in the example shown in Figure 3.
[0152] As described above, the information processing device 100A can perform efficient search processing by calculating an approximate distance (second distance) between the vector of each vector-quantized node and the search query, and then performing the search processing using the second distance. Furthermore, the information processing device 100A can perform efficient search processing by performing distance calculations for a number of nodes that can be parallelized in a single operation. As described above, the information processing device 100A can perform efficient search processing by performing calculations in a single operation using blobs as the unit of batch processing.
[0153] In the search process, the information processing device 100A traverses the graph GR21 and performs the above-described process on the nodes to be processed, thereby extracting nodes with a search count as neighboring nodes to the search query QE2. For example, the information processing device 100A obtains search results using either the first method or the second method described above, similar to the information processing device 100.
[0154] The process described above is merely one example, and the information processing device 100A may use various information and methods to perform search processing for efficient search processing. This point will be explained in detail below. For example, the information processing device 100A may perform clustering and generate blobs such that the number of nodes belonging to each blob is equal to the number that the information processing device 100 can process collectively using SIMD (a unit that can be processed collectively).
[0155] Furthermore, the information processing device 100A may use information about blobs to suppress redundant processing. The information processing device 100A may have a blob distance calculation flag (hereinafter simply referred to as "flag") that indicates whether the distance calculation for a blob has been performed, and a table that indicates which blob each object belongs to. In this case, the information processing device 100A may use the flag to manage whether each blob has been processed or not. For example, immediately before processing each neighboring node, the information processing device 100A may identify the blob to which the node belongs using the table, determine using the flag whether the blob has already had its distance calculated in a batch, and if it has not been calculated, perform a batch calculation and then process the individual nodes. This point will also be explained in Figures 15 and 16. This allows the information processing device 100A to suppress redundant processing.
[0156] [2-2. Configuration of Information Processing Device] Next, the configuration of the information processing device 100A according to the second embodiment will be described using Figure 13. Figure 13 is a diagram showing an example of the configuration of the information processing device according to the second embodiment. As shown in Figure 13, the information processing device 100A has a communication unit 110, a storage unit 120A, and a control unit 130A. Note that in the information processing device 100A, the same points as in the information processing device 100 will be omitted from the explanation as appropriate.
[0157] (Storage unit 120A) The storage unit 120A is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120A according to the second embodiment, as shown in Figure 13, has an object information storage unit 121, a graph information storage unit 122, a quantization information storage unit 123A, a codebook information storage unit 124, and a blob information storage unit 125.
[0158] (Quantization information storage unit 123A) The quantization information storage unit 123A according to the second embodiment stores various information related to the allocation process. Figure 14 shows an example of the quantization information storage unit according to the second embodiment. In the example in Figure 14, the quantization information storage unit 123A has items such as "node ID", "object ID", "blob ID", and "quantization information".
[0159] The "Node ID" indicates identification information for identifying each node (object) in the graph. The "Object ID" indicates identification information for identifying an object. If the Node ID and Object ID are the same, the Object ID is stored in the "Node ID" field, and the "Object ID" field does not need to be included in the quantization information storage unit 123A.
[0160] The "Blob ID" is information used to identify the blob to which a node (object) belongs.
[0161] Furthermore, "quantization information" indicates the information of the quantized vector of each node (object). For example, "quantization information" includes information such as "elements" and "codebook ID". "Elements" indicate the placement of the corresponding object in the vector. In the example in Figure 14, it is shown that "elements" include "#1", "#2", "#3", and "#4". In this case, the vector of each node (object) is divided into four parts, and each divided sub-vector is quantized by the codebook.
[0162] In the example in Figure 14, node N1 (object OB1) is identified by blob ID "BL9" (blob BL9). For example, if the vector is divided into four parts by Cartesian product quantization, the quantization information of node N1 will store information similar to the four codebooks shown in Figure 8. Also, for example, if Cartesian product quantization is not performed, the quantization information of node N1 will store information indicating one codebook for quantizing the vector of node N1.
[0163] Furthermore, the quantized information storage unit 123A may store various types of information depending on the purpose, not limited to those mentioned above.
[0164] (Blob information storage unit 125) The blob information storage unit 125 according to the second embodiment stores information about blobs. For example, the blob information storage unit 125 stores various information that identifies the object associated with each blob. Figure 15 shows an example of the blob information storage unit according to the second embodiment. In the example in Figure 15, the blob information storage unit 125 includes items such as "blob ID," "node ID," and "vector information."
[0165] The "Blob ID" indicates identification information for identifying a blob. The "Node ID" indicates the node (object) associated with the blob identified by the Blob ID. The "Vector Information" indicates the vector information of the blob. For example, the "Vector Information" indicates the vector corresponding to the centroid of the blob.
[0166] In the example shown in Figure 15, the nodes (objects) associated with the blob identified by blob ID "BL1" (blob BL1) are nodes N7, N9, N85, N126, etc. This indicates that blob BL1 is associated with multidimensional vector information of the form "51, 4, 102, 33...".
[0167] Furthermore, it indicates that the nodes (objects) associated with the blob identified by blob ID "BL9" (blob BL9) are nodes N1, N4, N5, etc. It also indicates that blob BL9 is associated with multidimensional vector information of the form "12, 55, 12, 6...".
[0168] The blob information storage unit 125 may store various types of information depending on the purpose, not limited to those described above. The blob information storage unit 125 may also store a flag indicating whether or not each blob has been processed for distance calculation. For example, the blob information storage unit 125 may set the flag value of each blob to either a value indicating that it has not been processed for distance calculation (also called the "unprocessed flag value") or a value indicating that it has been processed for distance calculation (also called the "processed flag value"). For example, at the start of the search process, the blob information storage unit 125 may set the flag value of each blob to a value indicating that distance calculation for that blob has not yet been processed (e.g., 0), and change the flag value of blobs that have been processed for distance calculation to a value indicating that distance calculation has been processed (e.g., 1).
[0169] For example, the information processing device 100A refers to the flag value of the blob being processed and determines whether or not to perform a batch distance calculation for that blob. The information processing device 100A refers to the flag value of each blob, and if the flag value of the blob is the unprocessed flag value, it determines that the blob has not yet undergone batch distance calculation and performs a batch distance calculation for the nodes belonging to that blob. On the other hand, if the flag value of the blob is the processed flag value, the information processing device 100A determines that the blob has already undergone batch distance calculation and does not perform a batch distance calculation for the nodes belonging to that blob.
[0170] (Control Unit 130A) Returning to the explanation of Figure 13, the control unit 130A is a controller, and is realized by, for example, a CPU, MPU, or GPU executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the information processing device 100A using RAM as the working area. Alternatively, the control unit 130A is a controller and can be realized by, for example, an integrated circuit such as an ASIC or FPGA.
[0171] As shown in Figure 13, the control unit 130A includes an acquisition unit 131, a generation unit 132A, a search processing unit 133A, and a provisioning unit 134, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130A is not limited to the configuration shown in Figure 13, and other configurations are also acceptable as long as they perform the information processing described later.
[0172] (Generation unit 132A) The generation unit 132A generates various types of information in the same manner as the generation unit 132.
[0173] The generation unit 132A may generate information related to vector quantization, as shown in the quantization information storage unit 123. For example, the generation unit 132 generates information indicating that node N1 (object OB1) belongs to blob BL9. The generation unit 132 may also generate information related to blobs, as shown in the blob information storage unit 125. For example, the generation unit 132 generates information indicating that nodes N7, N9, N85, N126, etc., belong to blob BL1. Note that if the information processing device 100 acquires the information shown in the graph information storage unit 122, quantization information storage unit 123A, codebook information storage unit 124, and blob information storage unit 125 from an external device such as an information providing device 50, the information processing device 100 does not need to have the generation unit 132A.
[0174] (Search processing unit 133A) The search processing unit 133A performs various processes related to the search process, similar to the search processing unit 133.
[0175] The search processing unit 133A, in a search process that searches for nodes near a search query, targets a group of nodes corresponding to each of multiple objects, and uses information from multiple blobs that classify the multiple objects to calculate the distance between multiple nodes belonging to a single blob and the search query using vector information of multiple nodes that have been vector-quantized. The search processing unit 133A performs the calculation of the distance between multiple nodes and the search query in parallel and in a batch. The search processing unit 133A performs parallel processing of the calculation of the distance between multiple nodes and the search query in batch processing numbers determined based on the specifications of the information processing device 100A.
[0176] The search processing unit 133A calculates the distance between the search query and multiple nodes belonging to a blob adjacent to the blob to which the search query belongs. In a search process that searches for neighboring nodes of the search query using a graph in which multiple nodes are connected by edges, the search processing unit 133A calculates the distance between the search query and multiple nodes belonging to a blob to which connected nodes, connected by edges from the target node being processed in the search process, belong.
[0177] The search processing unit 133A calculates the distance between the search query and multiple nodes belonging to a blob other than the blob to which the target node belongs. The search processing unit 133A calculates the distance between the search query and multiple nodes in a search process that searches for neighboring nodes of the search query using a blob graph in which each of the multiple nodes is connected to other blobs other than the blob to which each of the multiple nodes belongs. The search processing unit 133A calculates the distance between the search query and multiple nodes in a search process that searches for neighboring nodes of the search query using a blob graph, which is a transformed graph generated by connecting an edge from one node to a blob other than the blob to which the node connected by an edge from one of the multiple nodes belongs, in a pre-transformation graph in which multiple nodes are connected by edges.
[0178] The search processing unit 133A calculates the distance between the search query and multiple nodes belonging to a blob that is a blob connected by edges from the target node to be processed in the search process. The search processing unit 133A uses information indicating processed blobs, which are blobs that have already been processed, to determine whether to process the blob. If the blob is a processed blob, the search processing unit 133A determines not to process the blob.
[0179] The search processing unit 133A calculates a second distance, which is not vector-quantized, for predetermined nodes among the nodes processed during the search process, unlike the first distance, which is vector-quantized. The search processing unit 133A extracts a second number of nodes, which is greater than the first number of nodes to be extracted as neighboring nodes of the search query, as neighbor candidate nodes, and calculates the second distance for these neighbor candidate nodes. The search processing unit 133A extracts the first number of nodes from among the neighbor candidate nodes, starting with those with the shortest second distance, as neighboring nodes of the search query.
[0180] The search processing unit 133A calculates a second distance for nodes whose first distance is within a predetermined threshold. The search processing unit 133A calculates a second distance for nodes within a search range that indicates the range to be extracted as neighboring nodes. The search processing unit 133A calculates a second distance for nodes within a search range that indicates the range to be searched.
[0181] [2-3. Search Processing Example] Here, an example of the search process according to the second embodiment will be explained using Figure 16 as an example. Figure 16 is a flowchart of an example of the search process according to the second embodiment. The search process described below is performed by the search processing unit 133A of the information processing device 100A. Note that explanations of points similar to those in the first embodiment, such as in Figure 11, will be omitted as appropriate. For example, in Figure 16, points similar to those in Figure 11 will be given the same step numbers, and explanations will be omitted as appropriate.
[0182] Here, the neighbor object set N(G,y) is the set of neighboring objects associated with node y by the edges assigned to it. "G" may be predetermined graph data (for example, the graph GR21 shown in spatial information SP21). For example, the information processing device 100A performs a k-nearest neighbor search process.
[0183] For example, the search process shown in Figure 16 differs from the search process shown in Figure 11 in that it performs the process shown in step S304a after step S304. Thus, the search process shown in Figure 16 is a normal graph search that, when it reaches a blob that has not yet been accessed, calculates the distance between the object (node) in that blob and the query all at once, and then processes each node.
[0184] Specifically, in the search process shown in Figure 16, if the blob to which object s belongs has not yet had its distance calculated, the information processing device 100A executes all of the following (the four processes shown after "-" in step S304a in Figure 16; hereinafter referred to as "the first process" to "the fourth process") (step S304a). In step S304a, the information processing device 100A executes the process of calculating the distance of all objects in that blob at once (the first process). Also in step S304a, the information processing device 100A executes the process of setting the distance calculation flag for the blob (the second process). Also in step S304a, the information processing device 100A executes the process of storing all objects in the blob for which the distance calculation has been performed in C (the third process). Also in step S304a, the information processing device 100A executes the process of processing all objects in the blob one by one from step S307 to step S314, treating all objects in the blob as u (the fourth process). Subsequently, the information processing device 100A performs the processing from step S306 onward.
[0185] [2-4. Variations] Now, let's describe some modifications. In the modifications according to the second embodiment, a graph including the concept of a blob may be used. For example, in the modifications according to the second embodiment, a graph in which the reference destination by an edge from a node is a blob may be used. Note that the same points as in the first and second embodiments will be omitted from the explanation as appropriate. The information processing device 100A according to the modifications has a graph information storage unit 122A instead of a graph information storage unit 122.
[0186] [2-4-1. Information Processing] First, we will explain the overview of information processing related to modification using Figure 17. Figure 17 is a diagram showing an example of information processing related to modification.
[0187] The information processing device 100A converts graph GR21, in which nodes are connected by edges, into graph GR31, in which the references from the edges of the nodes are blobs (step S31). In other words, as shown in Figure 17, the information processing device 100A generates graph GR31, which is a graph in which the concept of blobs has been introduced (a graph for blobs), from graph GR21, which is a normal graph.
[0188] In the example shown in Figure 17, the information processing device 100A generates a graph GR31 that includes directed edges from nodes to blobs, as shown in the spatial information SP31. In graph GR31, edges between each node are removed, so graph GR31 includes directed edges from nodes to blobs, but not edges between nodes. The arrow lines shown in Figure 17 indicate directed edges from the node at the source of the arrow to the blob at the destination of the arrow. That is, the arrow lines shown in Figure 17 indicate directed edges that reference the node at the source of the arrow and the blob at the destination of the arrow. For example, in Figure 17, it is shown that edges are connected from node N1 to two blobs, blob BL8 and blob BL10.
[0189] For example, the information processing device 100A converts the edges of each node from edges to connected nodes (neighboring nodes) to edges to the blob to which the connected node belongs. Note that the information processing device 100A does not generate edges to the blob to which each node belongs. In the example in Figure 17, since nodes N9 and N126 belong to the same blob BL1, the information processing device 100A does not generate edges from node N9 to blob BL1, nor from node N126 to blob BL1. On the other hand, in the example in Figure 17, since nodes N9 and N18 belong to different blobs BL1 and BL4, the information processing device 100A generates edges from node N9 to blob BL4, and from node N18 to blob BL1.
[0190] In the example shown in Figure 17, the information processing device 100A performs a search using a graph GR31 that includes directed edges from nodes (objects) to blobs, as shown in the spatial information SP31. For example, the information processing device 100A obtains the search results for search query QE2 by performing a search using graph GR21 as shown in Figure 19. Details of the search process shown in Figure 19 will be described later. Note that the search process in Figure 17 is similar to that in Figure 12 in that it performs a search using the concept of blobs, and is the same as the search process shown in Figure 12 except for the processing related to edges.
[0191] [2-4-2. Graphs] Next, we will explain the overview of the modified graph using Figure 18. Figure 18 is a diagram showing an example of a graph information storage unit related to the modified graph. For example, the graph information storage unit 122A shown in Figure 18 stores a graph in which the concept of blobs has been introduced (a graph for blobs). The graph information storage unit 122A shown in Figure 18 has items such as "node ID", "object ID", and "blob information". Thus, the graph information storage unit 122A related to the modified graph differs from the graph information storage unit 122 in Figure 7 in that it has "blob information" instead of "connected node information". Note that explanations of points that are the same as those of the graph information storage unit 122 in Figure 7 will be omitted as appropriate.
[0192] Furthermore, "blob information" indicates information about blobs (referenced blobs) that can be traced from the corresponding node. For example, "blob information" includes information such as "referenced destination." "Referenced destination" indicates information for identifying the referenced destination (blob) that is connected by an edge and can be traced from that node. In other words, in the example in Figure 18, the referenced destination (blob) that can be traced from that node by an edge is associated with the node ID (object ID) that identifies the node and is registered. Note that "blob information" may also include information for identifying the edge connected to the referenced destination (edge ID), etc.
[0193] In the example in Figure 18, the node identified by node ID "N1" (node N1) corresponds to the object (target) identified by object ID "OB1". Furthermore, an edge connects node N1 to the blob identified by blob ID "BL8" (blob BL8), indicating that it is possible to traverse blob BL8 from node N1. Additionally, an edge connects node N1 to the blob identified by blob ID "BL10" (blob BL10), indicating that it is possible to traverse blob BL10 from node N1.
[0194] Furthermore, the node identified by node ID "N2" (node N2) corresponds to the object (target) identified by object ID "OB2". Additionally, an edge is connected from node N2 to the blob identified by blob ID "BL5" (blob BL5), indicating that it is possible to traverse from node N1 to blob BL5.
[0195] Furthermore, the graph information storage unit 122A may store various types of information depending on the purpose, not limited to those mentioned above.
[0196] Furthermore, the graph may include a program module that takes a query as input, searches for nodes by traversing edges in the graph, and extracts and outputs nodes similar to the query. In other words, the graph may be intended to be used as a program module that performs search processing using the graph. For example, graph GR31 may be a program that, when vector data is input as a query, extracts and outputs nodes from the graph that correspond to vector data similar to that vector data. For example, graph GR31 may be data used as a program module that searches for similar images corresponding to a query image. For example, graph GR31 causes the computer to function so that, based on the input query, it extracts and outputs nodes in the graph that are similar to that query.
[0197] [2-4-3. Search Processing Example] From here, an example of the search process related to the modified example will be explained using Figure 19 as an example. Figure 19 is a flowchart of an example of the search process related to the modified example. The search process described below is performed by the search processing unit 133A of the information processing device 100A. Note that explanations of points that are the same as in the first or second embodiment, such as in Figures 11 and 16, will be omitted as appropriate. For example, in Figure 19, points that are the same as in Figure 11 will be given the same step numbers, and explanations will be omitted as appropriate.
[0198] In Figure 19, the set N(G,y) differs from the search processes in Figures 11 and 16 in that it is a set of blobs associated with the edge assigned to node y. In Figure 19, N(G,s) and C are blob sets. That is, the "neighborhood object set N" in Figure 11 is read as "blob set N" in Figure 19, and the "object set C" in Figure 11 is read as "blob set C" in Figure 19. Also, the word "object" in the process related to the above conversion to blobs should be read as "blob" as appropriate. "G" may be graph data (blob graph) in which the concept of blobs is introduced (for example, graph GR31 shown in spatial information SP31). For example, the information processing device 100A performs a k-neighbor search process.
[0199] For example, in the search process shown in Figure 19, the process in step S306 is modified from Figure 11 as follows. In the search process in Figure 19, the information processing device 100A selects one blob from N(G, s) that is not included in the blob set C, stores the selected blob in the blob set C, calculates the objects within the blob all at once, and sets that set as B (also called "target blob object set B") (step S306).
[0200] Furthermore, the search process shown in Figure 19 differs from the search process shown in Figure 11 in that it performs the process shown in step S306a after step S306. Specifically, in the search process in Figure 19, the information processing device 100A selects one object u from the object set B within the target blob (step S306a). After that, the information processing device 100A performs the processes from step S307 onwards.
[0201] Furthermore, the search process shown in Figure 19 differs from the search process shown in Figure 11 in that it performs the process shown in step S314a before step S315. Specifically, in the search process in Figure 19, the information processing device 100A determines whether or not all objects have been selected from the object set B within the target blob (step S314a).
[0202] If all objects have been selected from the object set B within the target blob (step S314a: Yes), the information processing device 100A performs the process in step S315. On the other hand, if all objects have not been selected from the object set B within the target blob (step S314a: No), the information processing device 100A returns to step S306a and repeats the process.
[0203] [2-4-4. Other examples of generation processes] In the example described above, a blob graph was generated by converting the edges of each node from edges to connected nodes to edges to the blob to which the connected nodes belong. However, the information processing device 100A may generate the blob graph using various information as appropriate. For example, the information processing device 100A may generate the blob graph using a nearest neighbor search index (also simply called an "index") that searches multiple objects. Note that explanations of points similar to those described above will be omitted as appropriate.
[0204] An example of generating a graph for blobs is explained using Figure 20. Figure 20 is a diagram showing another example of information processing related to a modified example. In the following explanation, a graph is used as an example of an index, but the index can be any type that searches multiple objects. For example, the index may be an index that uses a hashing description such as a hash table (hash index), an index with a tree structure (tree index), etc.
[0205] Furthermore, information indicating an object or a node corresponding to that object (also called an "object node") may be described as the first type of information, and information used to classify multiple objects may be described as the second type of information. In addition, information indicating a blob may be described as the third type of information, and a graph in which object nodes are connected by edges may be described as the fourth type of information. For example, graph GR21 in Figure 20 has object nodes such as node N1 connected by edges and corresponds to the fourth type of information. Also, graph GR31 in Figure 17 and graph GR32 in Figure 20 are different from the fourth type of information; they are blob graphs (also called "group graphs") in which object nodes and blobs are connected by edges. That is, a group graph is a graph in which edges are connected from object nodes to blobs, with object nodes as the source and blobs (groups) as the destination.
[0206] The information processing device 100A uses graph GR21, in which object nodes are connected by edges, as an index (fourth piece of information) to generate graph GR32, which is a group graph where the references from the object nodes by edges are blobs (step S41). In Figure 20, the information processing device 100A generates graph GR32 which includes directed edges from nodes to blobs, as shown in spatial information SP32.
[0207] For example, the information processing device 100A selects one object from multiple objects, searches for neighboring objects of that object using graph GR21, and generates graph GR32 by performing a connection process that connects edges from the object node corresponding to the object to other groups to which neighboring objects belong. For example, the information processing device 100A randomly selects one object from multiple objects, performs a connection process on the selected object, and generates graph GR32.
[0208] For example, the information processing device 100A performs the following processes (1-1) to (1-4) to generate the graph GR32. Note that "object" may be interpreted as "node corresponding to the object."
[0209] (1-1): Get (select) a random object NX. (1-2): Search for a constant number of neighboring objects of object NX. (1-3): Get the blob to which each neighboring object belongs. (1-4): Generate edges to the blob obtained from object NX.
[0210] For example, in process (1-1), the information processing device 100A randomly selects one object NX from a plurality of objects stored in the object information storage unit 121.
[0211] For example, in process (1-2), the information processing device 100A performs a search process as shown in Figure 11, targeting object NX (query) and using graph GR21, thereby extracting k objects as neighboring objects of object NX.
[0212] For example, in process (1-3), the information processing device 100A refers to the blob information stored in the blob information storage unit 125 and obtains the blob to which the neighboring objects (nodes) of object NX are associated.
[0213] For example, in process (1-4), the information processing device 100A generates an edge from object NX to the blob acquired. For example, the information processing device 100A generates graph GR32 by associating information indicating the acquired blob with object NX in the quantized information storage unit 123A.
[0214] For example, the information processing device 100A repeatedly performs the above processes (1-1) to (1-4) until there are no more objects to process, thereby generating the graph GR32. Note that if the blobs shown in the spatial information SP21 are generated by searching, the search results from the blob generation process may be reused to generate the graph GR32. An example of generating blobs by searching will be described later.
[0215] For example, the information processing device 100A performs search processing using a graph GR32 that includes directed edges from nodes (objects) to blobs, as shown in the spatial information SP32. For example, the information processing device 100A obtains search results for search query QE2 by performing search processing using graph GR32 as shown in Figure 19, targeting search query QE2.
[0216] Furthermore, although the above example showed the generation of blobs by clustering, blobs may be generated by various methods other than clustering. For example, the information processing device 100A may generate information indicating the blob (third information) using second information such as an index.
[0217] The information processing device 100A generates third-party information representing blobs, using graph GR21 as an index, as shown in Figure 21. Figure 21 is a diagram illustrating an example of the blob information generation process. The spatial information SP20 in Figure 21 shows the state before the blob information in spatial information SP21 is generated, and graph GR21 in Figure 21 is the same graph as graph GR21 in Figure 20. Note that the process shown in Figure 21 is a part of the first method described later, but the details will be described later.
[0218] For example, the information processing device 100A generates third-party information by classifying multiple objects into multiple blobs through a search using the graph GR21. For example, the information processing device 100A selects one node (also called the "processing target node") from multiple nodes in the graph GR21, and generates third-party information by classifying the objects corresponding to each of the neighboring nodes (nodes connected to the processing target node by edges) into a single group. For example, the information processing device 100A selects a processing target node from multiple nodes through a predetermined process, performs a classification process on the selected processing target node, and generates third-party information.
[0219] For example, the information processing device 100A may select a node to be processed based on the number of neighboring nodes (connected nodes) connected to the node by edges. For example, the information processing device 100A may select a node to be processed in order from the node with the largest number of neighboring nodes. The method of selecting a node to be processed in order from the node with the largest number of neighboring nodes and generating third information indicating a blob is also called the "first method". Alternatively, for example, the information processing device 100A may select a node to be processed in order from the node with the smallest number of neighboring nodes (connected nodes). The method of selecting a node to be processed in order from the node with the smallest number of neighboring nodes and generating a blob is also called the "second method". Alternatively, for example, the information processing device 100A may select a node to be processed randomly. The method of selecting a node to be processed randomly and generating third information indicating a blob is also called the "third method". Below, an overview of the processing procedures for each of the first, second, and third methods will be described.
[0220] [2-4-4-1. 1st method] First, let's explain the first method. For example, in the first method, the information processing device 100A performs the following processes (2-1) to (2-3) to generate third information indicating a blob.
[0221] (2-1): Obtain the node ND starting with the node with the largest number of neighboring nodes (edges). (2-2): The neighboring nodes of node ND and node ND are considered as one blob. (2-3): Remove the blob node and its edges.
[0222] For example, the information processing device 100A performs processes (2-1) to (2-3) at all nodes.
[0223] For example, in process (2-1), the information processing device 100A refers to the graph GR21 and sequentially obtains a node ND. The information processing device 100A may also obtain a node ND using a list of nodes arranged in descending order of the number of neighboring nodes.
[0224] For example, in process (2-2), the information processing device 100A refers to the graph being processed (graph GR21, etc.) and generates information that sets the neighboring nodes of node ND and node ND as a single blob. Note that the information processing device 100A may set not all neighboring nodes as blobs, but only neighboring nodes up to a constant K as blobs. For example, instead of setting all neighboring nodes of node ND together with node ND as a blob, the information processing device 100A may set K neighboring nodes and node ND as a single blob.
[0225] For example, in process (2-3), the information processing device 100A deletes the nodes that were designated as blobs in process (2-2) and the edges of those nodes so that nodes are not selected more than once. For example, if a node designated as a blob is a neighboring node of another node, the information processing device 100A deletes that node (and its edges). Note that the above method is merely an example, and process (2-3) is not limited to methods that delete edges, etc., but can be any method that enables the generation of blobs. For example, the information processing device 100A may generate blobs by maintaining a list of nodes that already belong to a blob and managing the nodes to be excluded. For example, the information processing device 100A may manage nodes that already belong to a blob by setting a predetermined flag (such as a deletion flag) on the node designated as a blob. When using flags in this way, the information processing device 100A refers to the flag of each node during processing to determine whether to process that node.
[0226] A specific example of the first method described above will be explained using Figure 21. In Figure 21, the information processing device 100A acquires node N7 in graph GR21, which has the maximum number of neighboring nodes (4) (step S51). In Figure 21, node N1 in graph GR21 also has 4 neighboring nodes, but the information processing device 100A acquires node N7. If there are multiple nodes with the same number of neighboring nodes, the information processing device 100A may randomly acquire a node from among those multiple nodes.
[0227] Then, the information processing device 100A designates the four neighboring nodes of node N7, N9, N12, N54, and N126, along with node N7 itself, as a single blob BL11 (step S52). The information processing device 100A generates third information BLT1 indicating that the five nodes N7, N9, N12, N54, and N126 belong to a single blob BL11. Then, the information processing device 100A removes the five nodes N7, N9, N12, N54, and N126, as well as the edges of each node, from graph GR21. As a result, graph GR21 is updated to graph GR21-1, as shown in spatial information SP20-1.
[0228] Then, the information processing device 100A acquires node N1 from graph GR21, which has 4 neighboring nodes (step S53). Then, the information processing device 100A designates the 4 neighboring nodes N4, N5, N88, and N99, along with node N1 itself, as a single blob BL12 (step S54). The information processing device 100A generates third information BLT2 indicating that the 5 nodes N1, N4, N5, N88, and N99 belong to a single blob BL12. Then, the information processing device 100A removes the 5 nodes N1, N4, N5, N88, and N99, as well as the edges of each node, from graph GR21-1.
[0229] The information processing device 100A repeats the above-described process until there are no more nodes to process, generating a third piece of information that represents multiple blobs.
[0230] [2-4-4-2. Second method] Next, the second method will be explained. For example, in the second method, the information processing device 100A performs the following processes (3-1) to (3-3) to generate third information indicating the blob.
[0231] (3-1): Obtain the node ND starting with the node with the smallest number of neighboring nodes (edges) for each node. (3-2): The neighboring nodes of node ND and node ND are considered as one blob. (3-3): Remove the blob node and its edges.
[0232] For example, the information processing device 100A performs processes (3-1) to (3-3) at all nodes.
[0233] For example, in process (3-1), the information processing device 100A refers to the graph GR21 and sequentially acquires a node ND. The information processing device 100A may also acquire a node ND using a list of nodes arranged in descending order of the number of neighboring nodes.
[0234] For example, in process (3-2), the information processing device 100A refers to the graph being processed (graph GR21, etc.) and generates information that treats the neighboring nodes of node ND and node ND as a single blob. Note that the information processing device 100A may treat not all neighboring nodes as blobs, but only neighboring nodes up to a constant K as blobs. For example, instead of treating all neighboring nodes of node ND as a blob along with node ND, the information processing device 100A may treat K neighboring nodes and node ND as a single blob.
[0235] For example, in process (3-3), the information processing device 100A deletes the nodes that were designated as blobs in process (3-2) and the edges of those nodes so that nodes are not selected more than once. For example, if a node designated as a blob is a neighbor node of another node, the information processing device 100A deletes that node (and its edges). Note that the second method is the same as the first method except that the order in which nodes are acquired is from the node with the smallest number of neighbor nodes (edges), so a detailed explanation of specific examples will be omitted. For example, process (3-3) is not limited to deleting edges, etc., as in the first method, but can be any method that can generate a blob. For example, the information processing device 100A may generate a blob by maintaining a list of nodes that already belong to a blob and managing the nodes to be excluded, as in the first method.
[0236] [2-4-4-3. Third method] Next, the third method will be described. For example, in the case of the third method, the information processing apparatus 100A performs the following processes (4-1) to (4-3) to generate third information indicating a blob.
[0237] (4-1): Sequentially acquire nodes ND at random (4-2): Treat the neighboring nodes of a node ND and the node ND as one blob (4-3): Delete the nodes treated as a blob and edges of the nodes
[0238] For example, the information processing apparatus 100A executes the processes (4-1) to (4-3) for all nodes. The third method is the same as the first method and the second method except that the node acquisition order is random, and detailed description thereof will be omitted. For example, if all nodes have the same number of edges, the information processing apparatus 100A may execute the generation of the third information in a random order (the third method).
[0239] [2-4-4-4. Fourth Method] Note that the above-described first method to third method are merely examples, and the information processing apparatus 100A may generate blobs through various processes. For example, the information processing apparatus 100A may generate a blob by a search using a neighborhood search index (index) that targets a plurality of objects. The method of generating a blob by a search using an index is also referred to as "fourth method".
[0240] Hereinafter, the fourth method will be described. Note that descriptions of points identical to the above-described contents will be omitted as appropriate. In the following, a case where a graph is used as an example of the index will be described as an example, but as described above, the index may be any index as long as it targets a plurality of objects. For example, the index may be any index such as a hash index or a tree index.
[0241] Next, we will explain the case in which the information processing device 100A generates third information indicating a blob by searching using graph GR21 as an index, as shown in Figure 21. For example, the information processing device 100A generates graph GR32 by performing the following processes (5-1) to (5-4).
[0242] (5-1): Get (select) a random object NX. (5-2): Search for a constant number of neighboring objects of object NX. (5-3): Object NX and its neighbors are treated as a single blob. (5-4): Execute processing to avoid duplicate processing.
[0243] For example, in process (5-1), the information processing device 100A randomly obtains an object NX from the graph GR21.
[0244] For example, in process (5-2), the information processing device 100A performs a search process as shown in Figure 11, targeting object NX (query) and using graph GR21, thereby extracting k objects as neighboring objects of object NX. Alternatively, the information processing device 100A may scan (search) all objects and extract neighboring objects of object NX.
[0245] For example, in process (5-3), the information processing device 100A generates information that groups the neighboring nodes of neighboring objects of object NX and object NX into a single blob.
[0246] For example, in process (5-4), the information processing device 100A executes a process to avoid duplicate processing so that an object does not end up belonging to multiple blobs due to being searched for multiple times. For example, the information processing device 100A removes the object that has been designated as a blob from the index. For example, the information processing device 100A removes the object that has been designated as a blob from graph GR21.
[0247] Furthermore, the information processing device 100A may perform processing to avoid duplicate processing by any method, as long as a single object does not belong to multiple blobs. For example, the information processing device 100A may suppress the processing of objects that already belong to a blob by setting a predetermined flag (such as a deletion flag) on the object that has been made into a blob. In this case, when using flags, the information processing device 100A refers to the flag of each object during processing to determine whether to process that object. For example, when the information processing device 100A uses search results to generate the group graph described above, it may perform processing to avoid duplicate processing by using a flag method.
[0248] [2-4-5. Information Processing Device Related to Modified Examples] In the information processing device 100A according to a modified example of the second embodiment, the acquisition unit 131 also performs the following processing. The acquisition unit 131 acquires first information indicating a plurality of objects that are the target of data retrieval, and second information used to classify the plurality of objects. The acquisition unit 131 acquires second information which is an index for searching the plurality of objects. The acquisition unit 131 acquires second information which is a graph in which a plurality of nodes corresponding to each of the plurality of objects are connected by edges. The acquisition unit 131 acquires fourth information which is an index for searching the plurality of objects. The acquisition unit 131 acquires fourth information which is a graph in which a plurality of object nodes corresponding to each of the plurality of objects are connected by edges.
[0249] Furthermore, in the information processing device 100A according to a modified example of the second embodiment, the generation unit 132A also performs the following processing. The generation unit 132A uses the second information acquired by the acquisition unit 131 to classify the multiple objects indicated by the first information and generates third information indicating multiple groups used for batch processing in a search process targeting multiple objects. The generation unit 132A uses the second information to generate third information that classifies the multiple objects into multiple groups. The generation unit 132A generates third information that classifies the multiple objects into multiple groups through a search using the second information.
[0250] The generation unit 132A generates third-party information by classifying the nodes to be classified, which are at least some of the neighboring nodes connected to the node by edges, and the group of objects corresponding to each of the nodes, into a single group. The generation unit 132A generates third-party information by classifying the nodes to be classified, which are a predetermined number of neighboring nodes of the node, and the group of objects corresponding to each of the nodes, into a single group. The generation unit 132A generates third-party information by classifying the nodes to be classified, which are all of the neighboring nodes of the node, and the group of objects corresponding to each of the nodes, into a single group.
[0251] The generation unit 132A generates third-party information by selecting one node from a group of nodes based on the number of neighboring nodes, which are nodes connected to each of the multiple nodes by edges, and classifying the nodes to be classified for that node and the group of objects corresponding to each of those nodes into a single group. The generation unit 132A generates third-party information by selecting one node in order from the largest number of neighboring nodes, and classifying the nodes to be classified for that node and the group of objects corresponding to each of those nodes into a single group. The generation unit 132A generates third-party information by selecting one node in order from the smallest number of neighboring nodes, and classifying the nodes to be classified for that node and the group of objects corresponding to each of those nodes into a single group.
[0252] The generation unit 132A generates third-party information by randomly selecting one node from multiple nodes and classifying the node to be classified and the group of objects corresponding to each node into a single group. The generation unit 132A removes the node to be classified and the group of processed nodes that are the node that have been classified into a single group from the graph and generates third-party information by repeating the classification process using the graph after the processed node group has been removed. The generation unit 132A generates third-party information by repeating the classification process using the graph after the processed node group has been removed until multiple objects are classified into one of the groups. The generation unit 132A generates third-party information that shows multiple mutually exclusive groups to which each object belongs.
[0253] The generation unit 132A uses the fourth information to generate a group graph in which edges are connected from object nodes, which are nodes corresponding to at least some of the multiple objects, to other groups among the multiple groups, excluding the group to which the object nodes belong. The generation unit 132A selects one object from the multiple objects, uses the fourth information to search for neighboring objects of the one object among the multiple objects, and generates a group graph by connecting edges from one object node corresponding to the one object to other groups to which the neighboring objects belong. The generation unit 132A randomly selects one object from the multiple objects, searches for neighboring objects of the one object, and generates a group graph by connecting edges from one object node corresponding to the one object to other groups to which the neighboring objects belong. The generation unit 132A uses the fourth information to generate a group graph.
[0254] [2-4-6. Information Processing Flow] Next, we will explain the information processing procedure related to the modification using Figure 22. Figure 22 is a flowchart showing an example of information processing related to the modification.
[0255] As shown in Figure 22, the information processing apparatus 100A acquires first information indicating a plurality of objects that are targets of data search (step S401). For example, the information processing apparatus 100A acquires the first information indicating the plurality of objects from the object information storage unit 121.
[0256] Further, the information processing apparatus 100A acquires second information used for classifying the plurality of objects (step S402). For example, the information processing apparatus 100A acquires a graph as the second information from the graph information storage unit 122A.
[0257] Then, the information processing apparatus 100A uses the second information to classify the plurality of objects indicated by the first information, and generates third information indicating a plurality of groups used for performing batch processing in search processing targeting the plurality of objects (step S403). For example, the information processing apparatus 100A generates the third information indicating a plurality of blobs whose respective belonging objects are mutually exclusive.
[0258] [3. Third Embodiment] In the above-described example, the description has been given using two types of graphs: a graph in which nodes corresponding to objects (object nodes) are connected by edges (also referred to as "object graph"), and a graph in which object nodes and groups (blobs) are connected by edges (blob graph). However, the information processing system 1 may use various types of graphs. For example, unlike the object graph and the blob graph, the information processing system 1 may generate a graph in which groups (blobs) are connected (hereinafter also referred to as "group connection graph" or "blob connection graph"), and use the blob connection graph for search.
[0259] This point will be described below as a third embodiment. In the third embodiment, the information processing system 1 includes an information processing apparatus 100B in place of the information processing apparatus 100 or the information processing apparatus 100A. Note that descriptions of points that are the same as the contents described above in the first embodiment, the second embodiment, and the like will be omitted as appropriate.
[0260] The information processing device 100B uses an index (index information) to generate a group-linked graph in which edges are connected from a first group (first blob) to which one object belongs to a second group (second blob) to which neighboring objects of one object belong. Below, we will explain the case where a graph is used as an example of an index, but as mentioned above, the index can be any type of index that searches multiple objects. For example, the index can be a hash index, a tree index, or any other type of index.
[0261] [3-1. Information Processing] First, we will explain the overview of the information processing according to the third embodiment using Figure 23. Figure 23 is a diagram showing an example of the information processing according to the third embodiment.
[0262] In the example shown in Figure 23, it is assumed that the information processing device 100B has already acquired the graph GR21 as shown in the spatial information SP21. The information processing device 100B may generate the graph GR21 using various graph generation techniques as appropriate. Since the graph GR21 is the same as the graph GR21 shown in Figure 12, etc., its explanation is omitted.
[0263] The information processing device 100B generates information indicating blobs BL1 to BL10, etc., by any method. The information processing device 100B may classify each node into one of the blobs BL1 to BL10 by any clustering method such as k-means. The information processing device 100B may generate the blobs by any method as long as it is possible to generate blobs. For example, the information processing device 100B may generate information indicating blobs BL1 to BL10, etc., by any of the first to fourth methods described above.
[0264] The information processing device 100B generates a blob-connected graph using the graph GR21 and blob information as shown in the spatial information SP21 (step S61). In the example in Figure 23, the information processing device 100B generates a graph GR41, which is a blob-connected graph in which a blob (first blob) and another blob (second blob) are connected by edges, as shown in the spatial information SP41. The arrow lines shown in Figure 23 indicate directed edges from the blob at the beginning of the arrow (first blob) to the blob at the end of the arrow (second blob). That is, the arrow lines shown in Figure 23 indicate directed edges that refer to the blob at the beginning of the arrow and refer to the blob at the end of the arrow. For example, in Figure 23, edges are shown to be connected from blob BL1 to five other blobs: blob BL2, blob BL3, blob BL4, blob BL5, and blob BL8.
[0265] For example, the information processing device 100B searches graph GR21 and generates graph GR41 based on the search results. For example, the information processing device 100B performs the following processes (6-1) to (6-4) to generate graph GR41, which is a blob-connected graph. Note that the following objects may be read as nodes corresponding to the objects.
[0266] (6-1): Get (select) a random object NX. (6-2): Search for a constant number of neighboring objects of object NX. (6-3): Get the blob to which each neighboring object belongs. (6-4): Generate an edge to the blob obtained from the blob to which object NX belongs.
[0267] For example, in process (6-1), the information processing device 100B randomly selects one object NX from a plurality of objects stored in the object information storage unit 121.
[0268] For example, in process (6-2), the information processing device 100B performs a search process as shown in Figure 11, targeting object NX (query) and using graph GR21, thereby extracting k objects as neighboring objects of object NX.
[0269] For example, in process (6-3), the information processing device 100B refers to the blob information stored in the blob information storage unit 125 and obtains the blob to which the neighboring objects (nodes) of object NX are associated.
[0270] For example, in process (6-4), the information processing device 100B generates an edge from the blob to which object NX belongs to the acquired blob. The information processing device 100B generates an edge from the blob to which object NX belongs to the blob to which all neighboring objects belong. For example, the information processing device 100B generates the graph GR41 by associating the information indicating the blob (blob ID) acquired as the reference destination of the blob to which object NX belongs with the information indicating the blob to which object NX belongs (blob ID) and registering it in the blob-linked graph information storage unit 126.
[0271] For example, the information processing device 100B repeatedly performs the above processes (6-1) to (6-4) until there are no more objects to process, thereby generating graph GR41. Note that if blob Y is already registered as a reference to blob X, the information processing device 100B may skip registration even if an object belonging to blob Y is searched (extracted) again as a neighboring object of an object belonging to blob X, thereby preventing the same blob from being registered multiple times as a reference to each blob.
[0272] For example, the information processing device 100B performs search processing using a graph GR41 that connects blobs with edges, as shown in the spatial information SP41. For example, the information processing device 100B obtains search results for search query QE2 by performing search processing using the blob-connected graph GR41 as shown in Figures 27 and 28, targeting the search query QE2.
[0273] The above-described method for generating a blob-connected graph is merely one example, and the information processing device 100B may generate blob-connected graphs by various methods. For example, the information processing device 100B may generate a blob-connected graph without searching for a graph. If the blobs shown in spatial information SP21 are generated by searching, the search results from the blob generation process may be reused to generate a blob-connected graph such as graph GR41.
[0274] For example, the information processing device 100B may convert the connection relationships of object nodes in graph GR21 into connection relationships of blobs to which the object nodes belong, thereby generating a blob-connected graph. For example, an edge is connected from node N1 belonging to blob BL9 to node N88 belonging to blob BL8 and to a node belonging to blob BL10, and an edge is connected from node N4 belonging to blob BL9 to a node belonging to blob BL7. Therefore, the information processing device 100B generates a blob-connected graph in which edges are connected from blob BL9 to the three blobs BL7, blob BL8, and blob BL10. In this way, the information processing device 100B may generate a blob-connected graph in which blobs are connected by edges based on the edges that connect object nodes.
[0275] [3-2. Configuration of Information Processing Equipment] Next, the configuration of the information processing device 100B according to the third embodiment will be described using Figure 24. Figure 24 is a diagram showing an example of the configuration of the information processing device according to the third embodiment. As shown in Figure 24, the information processing device 100B has a communication unit 110, a storage unit 120B, and a control unit 130B. Note that in the information processing device 100B, the same points as in the information processing device 100 or the information processing device 100A will be omitted from the explanation as appropriate.
[0276] (Storage unit 120B) The storage unit 120B is implemented by, for example, semiconductor memory elements such as RAM and flash memory, or storage devices such as hard disks and optical discs. The storage unit 120B according to the third embodiment, as shown in Figure 24, includes an object information storage unit 121, a graph information storage unit 122, a quantization information storage unit 123, a codebook information storage unit 124, a blob information storage unit 125, and a blob-linked graph information storage unit 126.
[0277] (Blob-linked graph information storage unit 126) The blob-connected graph information storage unit 126 according to the third embodiment stores various information related to the blob-connected graph. For example, the blob-connected graph information storage unit 126 stores the generated blob-connected graph. Figure 25 shows an example of the blob-connected graph information storage unit according to the third embodiment. The blob-connected graph information storage unit 126 shown in Figure 25 has items such as "blob ID" and "connected blob information".
[0278] "Blob ID" indicates identification information for identifying each blob (group) in the graph. "Connected Blob Information" indicates information about blobs that can be traced from the corresponding blob (referenced blobs). For example, "Connected Blob Information" includes information such as "Referenced Blob". "Referenced Blob" indicates information for identifying the referenced blob that is connected by an edge and can be traced from that blob. In other words, in the example in Figure 25, the referenced blobs that can be traced from the blob by an edge are associated with the blob ID that identifies the blob. Note that "Connected Blob Information" may also include information for identifying the edge connected to the referenced blob (edge ID), etc.
[0279] In the example in Figure 25, we can see that the blob identified by blob ID "BL1" (blob BL1) has edges connected to the five blobs identified by blob IDs "BL2", "BL3", "BL4", "BL5", and "BL8". In other words, it is possible to traverse from blob BL1 to each of the five blobs BL2, BL3, BL4, BL5, and BL8.
[0280] The blob-connected graph information storage unit 126 is not limited to the above and may store various types of information depending on the purpose. Furthermore, the blob-connected graph may include a program module that takes a query as input, searches for objects (object nodes) by traversing edges in the blob-connected graph, and extracts and outputs objects similar to the query. In other words, the blob-connected graph may be intended for use as a program module that performs search processing using the blob-connected graph. For example, graph GR41, which is a blob-connected graph, may be a program that, when vector data is input as a query, extracts and outputs objects corresponding to vector data similar to that vector data using the blob-connected graph. For example, graph GR41 may be data used as a program module that searches for similar images corresponding to a query image. For example, graph GR41 causes the computer to function so that, based on the input query, it extracts and outputs objects similar to that query in the blob-connected graph.
[0281] (Control Unit 130B) Returning to the explanation of Figure 24, the control unit 130B is a controller, and is realized by, for example, a CPU, MPU, or GPU executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the information processing device 100B using RAM as the working area. The control unit 130B is also a controller, and is realized by, for example, an integrated circuit such as an ASIC or FPGA.
[0282] As shown in Figure 24, the control unit 130B includes an acquisition unit 131, a generation unit 132B, a search processing unit 133B, and a provisioning unit 134, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130B is not limited to the configuration shown in Figure 24, and other configurations are also acceptable as long as they perform the information processing described later.
[0283] The acquisition unit 131 according to the third embodiment acquires blob information showing multiple blobs in which multiple objects to be searched are classified, and index information showing an index for which the multiple objects are searched. The acquisition unit 131 acquires index information showing a graph in which multiple nodes corresponding to each of the multiple objects are connected by edges. The acquisition unit 131 acquires blob information showing multiple blobs in which multiple objects are classified by clustering processing. The acquisition unit 131 acquires blob information showing multiple blobs in which objects belonging to each blob are mutually exclusive. The acquisition unit 131 acquires a query.
[0284] (Generation unit 132B) The generation unit 132B generates various types of information in the same manner as the generation unit 132 or the generation unit 132A.
[0285] The generation unit 132B uses the index information acquired by the acquisition unit 131 to generate a blob connection graph in which edges are connected from a first blob to which one of the multiple objects belongs to a second blob to which neighboring objects of that object belong. The generation unit 132B selects one object from the multiple objects, uses the index information to search for the previous neighboring objects of that object among the multiple objects, and generates a blob connection graph by connecting edges from the first blob to which the object belongs to to the second blob to which neighboring objects belong.
[0286] The generation unit 132B randomly selects one object from multiple objects, searches for neighboring objects of that object, and generates a blob-connected graph by connecting edges from the first blob to which the object belongs to to the second blob to which the neighboring objects belong. The generation unit 132B uses the graph to generate a blob-connected graph in which edges are connected from the first blob to the second blob.
[0287] The generation unit 132B uses a graph to search for neighboring objects of one object and generates a blob-connected graph by connecting edges from the first blob to which one object belongs to the second blob to which the neighboring objects belong. The generation unit 132B generates a blob-connected graph by connecting edges from the first blob to the second blob to which the neighboring objects corresponding to neighboring nodes, which are nodes connected by edges to a node corresponding to one object in the graph, belong.
[0288] The generation unit 132B generates a blob connection graph by connecting the first representative point, which is the representative point of the first blob, and the second representative point, which is the representative point of the second blob, with edges from the first blob. The generation unit 132B generates a blob connection graph by connecting the first representative point, which is the centroid of the first blob, and the second representative point, which is the centroid of the second blob, with edges. The generation unit 132B generates a blob connection graph by connecting the first representative point, which is the center point calculated based on the objects belonging to the first blob, and the second representative point, which is the center point calculated based on the objects belonging to the second blob, with edges.
[0289] (Search processing unit 133B) The search processing unit 133B performs various processing related to the search process, similar to the search processing unit 133 or the search processing unit 133A.
[0290] The search processing unit 133B performs search processing using the blob-connected graph generated by the generation unit 132B. The search processing unit 133B performs search processing to find neighboring objects of the search query using the blob-connected graph. The search processing unit 133B performs search processing to find neighboring objects of the search query by traversing the edges that connect blobs in the blob-connected graph.
[0291] The search processing unit 133B, in the search process, uses blob information indicating multiple blobs to calculate the distance between a target object, which is an object belonging to one blob, and the search query, using the vector information of the vector-quantized target object. The search processing unit 133B performs the calculation of the distance between the target object and the search query in parallel and in a batch. The search processing unit 133B performs parallel processing of the calculation of the distance between the target object and the search query for a batch processing number determined based on the specifications of the information processing device.
[0292] [3-3. Information Processing Flow] Next, the procedure for information processing according to the third embodiment will be explained using Figure 26. Figure 26 is a flowchart showing an example of information processing according to the third embodiment.
[0293] As shown in Figure 26, the information processing device 100B acquires group information that indicates multiple groups into which multiple objects to be retrieved are classified (step S501). For example, the information processing device 100B acquires blob information that indicates multiple blobs into which nodes (object nodes) corresponding to multiple objects to be retrieved are classified.
[0294] The information processing device 100B obtains index information that indicates an index for searching multiple objects (step S502). For example, the information processing device 100B obtains a graph as an index in which nodes corresponding to multiple objects (object nodes) are connected by edges.
[0295] The information processing device 100B uses index information to generate a group-connected graph in which edges are connected from a first group to which one of the multiple objects belongs to a second group to which neighboring objects of that object belong (step S503). For example, the information processing device 100B uses index information to generate a blob-connected graph in which edges are connected from a first blob to which one of the multiple objects belongs to a second blob to which neighboring objects of that object belong.
[0296] [3-4. Search Processing Example] Here, an example of the search process according to the third embodiment will be described using Figures 27 and 28 as an example. Figures 27 and 28 are flowcharts illustrating an example of the search process according to the third embodiment. The search process described below is performed by the search processing unit 133B of the information processing device 100B. Note that explanations of points similar to those described in the first and second embodiments will be omitted as appropriate.
[0297] In Figures 27 and 28, N(G, s) and C are sets of blobs (blob set). Also, "G" may be graph data in which blobs are connected by edges (blob-connected graph) (for example, graph GR41 shown in spatial information SP41). For example, the information processing device 100B performs k-nearest neighbor search processing.
[0298] First, let's explain the process shown in Figure 27 (the main process). For example, the information processing device 100B sets the radius r of the hypersphere to ∞ (infinity) and sets the blob set C to the empty set (Φ) (step S601). Then, the information processing device 100B extracts a sub-blob set B from the existing blob set (all blobs) (step S602). For example, the information processing device 100B may extract the blob to which the object (node) selected as the root node belongs as the sub-blob set B. Alternatively, for example, the information processing device 100B may randomly extract blobs as the sub-blob set B.
[0299] Then, the information processing device 100B executes a determination process (step S603). The information processing device 100B executes the determination processes in steps S701 to S717 shown in Figure 28. Details of the determination processes shown in Figure 28 will be described later.
[0300] Then, the information processing device 100B sets a subset blob set B in the blob set S (step S604).
[0301] The information processing device 100B selects blob s with the smallest query distance d from among the blobs included in the blob set S (step S605). Here, "query distance" refers to the shortest distance between all objects within the blob and the query object (query). For example, if the object that is the query (search query) is y, the information processing device 100B selects blob s from the blob set S that has the shortest query distance to object y. Note that the query distance is not limited to the above, and may also be the distance between the query and a representative point (centroid, etc.) of the blob. Then, the information processing device 100B removes blob s from the blob set S (step S606).
[0302] Then, the information processing device 100B determines whether the query distance d of blob s exceeds r(1+ε) (step S607). If the query distance d of blob s exceeds r(1+ε) (step S607: Yes), the information processing device 100B outputs the object set R as the neighbor object set of object y (step S608) and terminates the process.
[0303] If the query distance d of blob s does not exceed r(1+ε) (step S607: No), the information processing device 100B determines whether blob s is included in the blob set C (step S609).
[0304] If blob s is included in the blob set C (step S609: Yes), the information processing device 100B returns to step S605 and repeats the process.
[0305] If blob s is not included in blob set C (step S609: No), the information processing device 100B adds blob s to blob set C (step S610).
[0306] Then, the information processing device 100B sets the neighborhood blob set N(G,s) of blob s in the subset blob set B (step S611). The neighborhood blob set N(G,s) is, for example, the set of blobs associated with blob s (neighboring blobs). For example, the neighborhood blob set N(G,s) is the set of blobs to which edges from blob s are connected (neighboring blobs). Then, the information processing device 100B performs a determination process (step S612). Details will be described later, but for example, the information processing device 100B performs the determination process of steps S701 to S717 shown in Figure 28, similar to step S603 described above.
[0307] The information processing device 100B then determines whether the blob set S is an empty set (Φ) (step S613). If the blob set S is not an empty set (step S613: No), the information processing device 100B returns to step S605 and repeats the process. If the blob set S is an empty set (step S613: Yes), the information processing device 100B outputs the object set R and terminates the process (step S614). For example, the information processing device 100B may select an object (node) included in the object set R as a neighbor node corresponding to the additional node (input object y). For example, the information processing device 100B may extract (select) an object (node) included in the object set R as a neighbor node corresponding to the target node (input object y). Alternatively, for example, the information processing device 100B may provide an object (node) included in the object set R as a search result corresponding to the search query (input object y) to the terminal device that performed the search.
[0308] From here, we will explain the process (decision process) shown in Figure 28. First, the information processing device 100B sets a subset blob set B in the blob set T (step S701).
[0309] Then, the information processing device 100B retrieves one blob b from the blob set T and removes blob b from the blob set T (step S702). Then, the information processing device 100B sets min, a variable used for the decision process, to ∞ (infinity) (step S703).
[0310] Then, the information processing device 100B retrieves one object u from blob b and deletes object u from blob b (step S704).
[0311] Then, the information processing device 100B determines whether the distance d(u,y) between object u and object y is less than min (step S705).
[0312] If the distance d(u,y) between object u and object y is less than min (step S705: Yes), the information processing device 100B sets (updates) the value of min to the distance d(u,y) (step S706) and performs the process in step S707.
[0313] If the distance d(u,y) between object u and object y is not less than min (step S705: No), the information processing device 100B does not perform the process in step S706 and performs the process in step S707.
[0314] The information processing device 100B determines whether the distance d(u,y) between object u and object y is less than or equal to r(1+ε) (step S707).
[0315] If the distance d(u,y) between object u and object y is less than or equal to r(1+ε) (step S707: Yes), the information processing device 100B adds blob b to the blob set S (step S708) and performs the process in step S709.
[0316] If the distance d(u,y) between object u and object y is not less than or equal to r(1+ε) (step S707: No), the information processing device 100B does not perform the process in step S708 and performs the process in step S709.
[0317] The information processing device 100B determines whether the distance d(u,y) between object u and object y is less than or equal to r (step S709).
[0318] If the distance d(u,y) between object u and object y is less than or equal to r (step S709: Yes), the information processing device 100B adds object u to the object set R (step S710) and performs the process in step S711.
[0319] If the distance d(u,y) between object u and object y is not less than or equal to r (step S709: No), the information processing device 100B does not perform the process in step S710 and performs the process in step S711.
[0320] The information processing device 100B determines whether the number of objects in the object set R exceeds ks (step S711). The predetermined number ks is an arbitrary natural number. For example, ks may be the number of search results or the number of objects to be extracted. Also, for example, if there is no upper limit on the number of objects to be extracted in a range search, ks may be set to infinity. For example, ks may be 4.
[0321] If the number of objects in the object set R exceeds ks (step S711: Yes), the information processing device 100B removes the object furthest from object y from the object set R (step S712), and then performs the process in step S713.
[0322] If the number of objects in the object set R does not exceed ks (step S711: No), the information processing device 100B does not perform the processing in step S712 and performs the processing in step S713.
[0323] The information processing device 100B determines whether the number of objects in the object set R matches ks (step S713).
[0324] If the number of objects in the object set R matches ks (step S713: Yes), the information processing device 100B sets the distance between the object furthest from object y and object y to r (step S714), and then performs the process in step S715.
[0325] If the number of objects in the object set R does not match ks (step S713: No), the information processing device 100B does not perform the process in step S714 and performs the process in step S715.
[0326] The information processing device 100B determines whether blob b is an empty set (Φ) (step S715). If blob b is not an empty set (step S715: No), the information processing device 100B returns to step S704 and repeats the process.
[0327] Furthermore, if blob b is an empty set (step S715: Yes), the information processing device 100B sets min as the query distance of blob b within the subset blob set B (step S716). For example, the information processing device 100B sets min as the query distance, which is the shortest distance among all the distances between all objects in blob b and the query.
[0328] The information processing device 100B determines whether the blob set T is an empty set (Φ) (step S717). If the blob set T is not an empty set (step S717: No), the information processing device 100B returns to step S702 and repeats the process.
[0329] Furthermore, if the blob set T is empty (step S717: Yes), the determination process is terminated.
[0330] [3-5. Variant Examples] The blob-connected graph may be configured in various ways, not limited to the examples described above. For example, the blob-connected graph may be a graph in which representative points of each group (blob) are connected by edges. This point will be explained below as a modification according to the third embodiment. Points that are the same as those in the first, second, and third embodiments will be omitted from explanation as appropriate. The information processing device 100B according to the modification of the third embodiment has a blob information storage unit 125A and a blob-connected graph information storage unit 126A instead of the blob information storage unit 125 and the blob-connected graph information storage unit 126.
[0331] [3-5-1. Information Processing] First, we will explain the overview of information processing related to the modified example using Figure 29. Figure 29 is a diagram showing an example of blob-connected graph information related to the modified example. In the example of Figure 29, the blobs are shown to be generated by clustering.
[0332] In the example shown in Figure 29, the information processing device 100B generates a blob-connected graph by connecting the centroids of each blob. The information processing device 100B generates a graph GR42, which is a blob-connected graph by connecting the centroids C1 to C10 corresponding to each of the blobs BL1 to BL10 with edges, as shown in the spatial information SP41. The information processing device 100B generates the graph GR42 by appropriately using various conventional techniques related to graph generation.
[0333] For example, the information processing device 100B generates a graph GR42 in which centroid C1 of blob BL1 is connected by edges to five centroids: centroid C2 of blob BL2, centroid C3 of blob BL3, centroid C4 of blob BL4, centroid C7 of blob BL7, and centroid C8 of blob BL8. Alternatively, for example, the information processing device 100B generates a graph GR42 in which centroid C2 of blob BL2 is connected by edges to three centroids: centroid C1 of blob BL1, centroid C3 of blob BL3, and centroid C4 of blob BL4.
[0334] The information processing device 100B may generate graph GR42 by any method. The information processing device 100B may generate graph GR42 using various indices. For example, the information processing device 100B may generate graph GR42 using graph GR21, in which object nodes are linked. In this case, the information processing device 100B can generate graph GR42 by the same process as graph GR41 in Figure 23, so a detailed explanation is omitted. For example, the information processing device 100B may generate graph GR42 by the same process as the example shown in Figure 23. Note that graph GR42 is shown as an example of undirected (bidirectional) edges, but directed edges may also be used.
[0335] In this modified form, the information processing device 100B generates a centroid graph as a blob-connected graph. The information processing device 100B then uses the generated blob-connected graph to perform a normal search on the centroids, and distance calculations are performed on a blob-by-blob basis. In this way, the information processing device 100B can simplify processing by representing the blobs with centroids. If the blobs are not clustered, the information processing device 100B may calculate a center point based on the objects belonging to each blob and use the calculated center point as the representative point.
[0336] In the example shown in Figure 29, the information processing device 100B performs search processing using a graph GR42 in which centroids, which are representative points of blobs, are connected by edges, as shown in spatial information SP42. For example, the information processing device 100B obtains search results for search query QE2 by performing search processing using graph GR21 as shown in Figures 27 and 28, targeting search query QE2. This is the same as in the example described above, so a detailed explanation is omitted.
[0337] [3-5-2. Information] Next, an overview of the information related to the modified example will be explained using Figures 30 and 31. Figure 30 is a diagram showing an example of a blob information storage unit related to the modified example. Figure 31 is a diagram showing an example of a blob linked graph information storage unit related to the modified example.
[0338] First, we will explain the modified blob information storage unit 125A shown in Figure 30. The modified blob information storage unit 125A shown in Figure 30 includes items such as "blob ID," "node ID," "vector information," and "centroid ID." Thus, the modified blob information storage unit 125A differs from the blob information storage unit 125 in Figure 15 in that it has a "centroid ID." Note that explanations of points that are the same as those of the blob information storage unit 125 in Figure 15 will be omitted as appropriate.
[0339] The "Centroid ID" indicates identification information used to identify the centroid of each blob. The "Vector Information" indicates vector information of the centroid, which is the representative point of the blob.
[0340] In the example shown in Figure 30, the centroid of blob BL1, identified by blob ID "BL1", is the centroid identified by centroid ID "C1" (centroid C1). The centroid of blob BL9, identified by blob ID "BL9", is the centroid identified by centroid ID "C9" (centroid C9).
[0341] Furthermore, the blob information storage unit 125A may store various types of information depending on the purpose, not limited to those mentioned above.
[0342] Next, we will describe the blob-connected graph information storage unit 126A according to the modified example shown in Figure 31. The blob-connected graph information storage unit 126A according to the modified example shown in Figure 30 has items such as "centroid ID" and "connected centroid information". Note that explanations of points similar to those of the blob-connected graph information storage unit 126 in Figure 25 will be omitted as appropriate.
[0343] The "Centroid ID" indicates identification information for identifying the centroid, which is the representative point of each blob in the graph. The "Connected Centroid Information" indicates information about centroids that can be traced from the corresponding centroid (referenced centroids). For example, the "Connected Centroid Information" includes information such as "Referenced Destination". The "Referenced Destination" indicates information for identifying the referenced destination (centroid) that is connected by an edge and can be traced from that centroid. In other words, in the example in Figure 31, the referenced destination (centroid) that can be traced from that centroid by an edge is registered in association with the centroid ID that identifies the centroid. Note that the "Connected Centroid Information" may also include information for identifying the edge connected to the referenced destination (edge ID), etc.
[0344] In the example in Figure 31, we can see that edges are connected from the centroid identified by centroid ID "C1" (centroid C1) to five centroids identified by centroid IDs "C2", "C3", "C4", "C7", and "C8", respectively. In other words, it is possible to traverse from centroid C1 to each of the five centroids C2, C3, C4, C7, and C8.
[0345] The blob-connected graph information storage unit 126A is not limited to the above and may store various types of information depending on the purpose. Furthermore, the blob-connected graph may include a program module that takes a query as input, searches for objects (object nodes) by traversing edges in the blob-connected graph, and extracts and outputs objects similar to the query. In other words, the blob-connected graph may be intended for use as a program module that performs search processing using the blob-connected graph. For example, graph GR42, which is a blob-connected graph, may be a program that, when vector data is input as a query, extracts and outputs objects corresponding to vector data similar to that vector data using the blob-connected graph. For example, graph GR42 may be data used as a program module that searches for similar images corresponding to a query image. For example, graph GR42 causes the computer to function so that, based on the input query, it extracts and outputs objects similar to that query in the blob-connected graph.
[0346] As described above, in each embodiment and modification, the information processing system 1 performs the following processes. For example, the information processing system 1 generates blobs by vector clustering. For example, the information processing system 1 generates a graph index using the centroids of the blobs as nodes.
[0347] For example, information processing system 1 acquires a certain number of arbitrary vectors and performs Cartesian product quantization clustering. In other words, information processing system 1 divides the vectors into subvectors, performs clustering on each subvector, and generates a codebook.
[0348] For example, Information Processing System 1 performs a Cartesian product quantization on objects belonging to a blob unit and generates an inverted index. For example, Information Processing System 1 generates a graph (blob graph) that includes edges from objects to blobs. For example, Information Processing System 1 searches for neighboring objects using a general Cartesian product quantization method. The general Cartesian product quantization method referred to here may be, for example, calculating the distance between an object and all centroids of a blob, obtaining a certain number of top blobs, and then obtaining neighboring objects by calculating the distance of objects within the blob using Cartesian product quantization. Note that if Information Processing System 1 has generated a graph of centroids as described above, it is possible to search for and obtain a certain number of blobs using the graph. Alternatively, Information Processing System 1 may generate a graph (blob-connected graph) in which blobs are connected by edges, rather than the graph (blob graph) that includes edges from objects to blobs as described above.
[0349] [4. Fourth Embodiment] The example above showed the use of blobs as one example of a group for classifying objects, but the group for classifying objects is not limited to blobs; various graphs for classifying objects may be used. For example, in addition to blobs (the first group), clusters, which are the second group that serve as the unit for quantizing objects, may also be used.
[0350] This point will be explained below in the fourth embodiment. In the fourth embodiment, the information processing system 1 has an information processing device 100C instead of the information processing device 100, information processing device 100A, or information processing device 100B. Note that explanations of points similar to those described above in the first, second, and third embodiments will be omitted as appropriate.
[0351] The information processing device 100C generates blob information (first group information) that indicates multiple blobs that classify multiple objects, and cluster information (second group information) that indicates clusters that are groups classified differently from blobs.
[0352] [4-1. Information Processing] First, an overview of the information processing according to the fourth embodiment will be explained using Figure 32. Figure 32 is a diagram showing an example of information processing according to the fourth embodiment. In Figure 32, the information processing device 100C classifies objects into multiple blobs and then classifies them into clusters based on those blobs. However, objects may be classified into clusters first and then into blobs, but this will be discussed later.
[0353] First, the information processing device 100C generates blob information that classifies multiple objects into multiple blobs (step S71). In the example in Figure 32, the information processing device 100C generates blob information that classifies multiple objects into blobs BL1 to BL10, etc., by any method. The information processing device 100C may classify each node into one of the blobs BL1 to BL10 by any clustering method such as k-means. The information processing device 100C may generate blobs by any method as long as it is possible to generate blobs. For example, the information processing device 100C may generate blobs using an index such as a graph. For example, the information processing device 100C may generate information indicating blobs BL1 to BL10, etc., by any method such as the first to fourth methods described above.
[0354] In Figure 32, the information processing device 100C generates a blob-connected graph GR41, as shown in the spatial information SP41, but it is not necessary to generate the graph GR41. For example, if the information processing device 100C does generate the graph GR41, it does so by processing the same information as in Figure 23, but a detailed explanation is omitted as it is the same as in the explanation in Figure 23.
[0355] Then, the information processing device 100C generates cluster information that classifies the multiple blobs into multiple clusters (step S72). For example, the information processing device 100C generates cluster information that classifies the multiple blobs, such as blobs BL1 to BL10, into clusters using any method. The information processing device 100C may classify the blobs into any of the multiple clusters, such as blobs BL1 to BL10, using any clustering method such as k-means. The information processing device 100C may generate the blobs by any method that enables the generation of blobs. For example, the information processing device 100C may generate blobs using an index such as a graph.
[0356] In the example shown in Figure 32, the information processing device 100C generates cluster information that classifies blobs BL1 to BL10 into clusters CL1 to CL4, etc. For example, the information processing device 100C classifies blobs BL1 and BL7 into cluster CL1. That is, the information processing device 100C classifies objects belonging to blob BL1 and objects belonging to blob BL7 into cluster CL1. Note that in Figures 32 and 34, the boundaries of the blobs are shown to be on the boundaries of the clusters, but depending on the division method, they may not coincide, and depending on the method, the space may not necessarily be divided by the boundaries.
[0357] Furthermore, the information processing device 100C classifies blobs BL2 and blob BL4 into cluster CL2. That is, the information processing device 100C classifies objects belonging to blob BL2 and objects belonging to blob BL4 into cluster CL2.
[0358] Furthermore, the information processing device 100C classifies blobs BL3, BL8, and BL10 into cluster CL3. That is, the information processing device 100C classifies objects belonging to blob BL3, objects belonging to blob BL8, and objects belonging to blob BL10 into cluster CL3.
[0359] Furthermore, the information processing device 100C classifies blobs BL5, BL6, and BL9 into cluster CL4. That is, the information processing device 100C classifies objects belonging to blob BL5, objects belonging to blob BL6, and objects belonging to blob BL9 into cluster CL4.
[0360] For example, the information processing device 100C performs quantization processing using cluster information. The information processing device 100C performs quantization processing on multiple objects using cluster information. For example, the information processing device 100C performs quantization for each object belonging to each of the clusters CL1 to CL4, but this will be explained later.
[0361] As described above, the information processing device 100C generates blob information that shows multiple blobs that classify multiple objects that are the target of data retrieval, and cluster information that shows multiple clusters that are classified differently from the multiple blobs. As a result, the information processing device 100C can generate two types of groups that classify multiple objects, and can generate groups that classify objects.
[0362] [4-1-1. Other processing examples] Note that the process shown in Figure 32 is merely one example, and the information processing device 100C may generate blob information and cluster information by any method of processing, as long as it is capable of generating them.
[0363] For example, the information processing device 100C may generate cluster information to classify objects into clusters, and then generate blob information to classify objects into blobs. In this case, the information processing device 100C may generate cluster information to classify each object into one of the multiple clusters by clustering, which classifies multiple objects into clusters. For example, the information processing device 100C may classify each object into one of the clusters CL1 to CL4 by any clustering method such as k-means.
[0364] Furthermore, the information processing device 100C may generate blob information indicating a blob by dividing at least one of the clusters CL1 to CL4 and generating two or more blobs. For example, the information processing device 100C may divide each cluster and generate blobs by clustering the objects belonging to each cluster using any clustering method such as k-means. For example, the information processing device 100C may generate blob BL1 and blob BL7 by dividing cluster CL1 into two blobs. For example, the information processing device 100C may cluster the objects belonging to cluster CL1 with 2 clusters using k-means, thereby dividing cluster CL1 and generating blob BL1 and blob BL7.
[0365] For example, the information processing device 100C may generate blobs BL2 and BL4 by dividing cluster CL2 into two blobs. For example, the information processing device 100C may generate blobs BL3, BL8 and BL10 by dividing cluster CL3 into three blobs. For example, the information processing device 100C may generate blobs BL5, BL6 and BL9 by dividing cluster CL4 into three blobs.
[0366] [4-2. Configuration of Information Processing Device] Next, the configuration of the information processing device 100C according to the fourth embodiment will be described using Figure 33. Figure 33 is a diagram showing an example of the configuration of the information processing device according to the fourth embodiment. As shown in Figure 33, the information processing device 100C has a communication unit 110, a storage unit 120C, and a control unit 130B. Note that in the information processing device 100C, the same points as in the information processing device 100, information processing device 100A, or information processing device 100B will be omitted from the explanation as appropriate.
[0367] (Storage unit 120C) The storage unit 120C is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120C according to the fourth embodiment, as shown in Figure 33, includes an object information storage unit 121, a graph information storage unit 122, a quantization information storage unit 123, a codebook information storage unit 124, a blob information storage unit 125, a blob-linked graph information storage unit 126, and a cluster information storage unit 127.
[0368] The cluster information storage unit 127 according to the fourth embodiment stores information about the second group of clusters. The cluster information storage unit 127 stores information about the clusters that serve as the units of quantization. The cluster information storage unit 127 may store identification information (cluster ID, etc.) that identifies each cluster, associated with information indicating the first group of blobs belonging to that cluster (blob ID, etc.). The cluster information storage unit 127 may also store identification information (cluster ID, etc.) that identifies each cluster, associated with information indicating the objects belonging to that cluster (object ID, etc.). The cluster information storage unit 127 may also store identification information (cluster ID, etc.) that identifies each cluster, associated with information for identifying the base point of the cluster, such as the centroid (base point ID, etc.), and its vector, etc.
[0369] (Control Unit 130C) The control unit 130C is a controller, and is implemented, for example, by a CPU, MPU, GPU, etc., executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the information processing device 100C using RAM as the working area. Alternatively, the control unit 130C is a controller and can be implemented, for example, by an integrated circuit such as an ASIC or FPGA.
[0370] As shown in Figure 33, the control unit 130C includes an acquisition unit 131, a generation unit 132C, a search processing unit 133C, and a provision unit 134, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130C is not limited to the configuration shown in Figure 33, and other configurations are also acceptable as long as they perform the information processing described later.
[0371] In the fourth embodiment, the acquisition unit 131 acquires object information indicating multiple objects that are the target of data retrieval. The acquisition unit 131 acquires index information indicating an index that targets the multiple objects for retrieval.
[0372] The acquisition unit 131 acquires processing information indicating multiple blobs that classify multiple objects that are the target of data retrieval, and multiple clusters that are groups that have a different classification from the multiple blobs, and which classify objects belonging to the same blob in the multiple blobs into the same group. The acquisition unit 131 acquires processing information indicating a first number of blobs that classify multiple objects, and clusters that classify multiple objects into a second number that is less than the first number. The acquisition unit 131 acquires processing information indicating multiple clusters in which groups of objects belonging to two or more blobs are classified into one cluster.
[0373] The acquisition unit 131 acquires processing information indicating multiple blobs used to perform a first process, which is a batch process in a search process targeting multiple objects. The acquisition unit 131 also acquires processing information indicating multiple clusters used to perform a second process, which is a quantization process targeting multiple objects.
[0374] The acquisition unit 131 acquires processing information that associates multiple first identification pieces, each identifying one of the multiple blobs, with multiple second identification pieces, each identifying one of the multiple clusters. The acquisition unit 131 acquires processing information in which one second identification piece is associated with two or more first identification pieces.
[0375] The acquisition unit 131 acquires processing information associated with the first identification information of the blob to which each of the multiple objects belongs. The acquisition unit 131 acquires processing information associated with the first identification information of the quantized object information of each of the multiple objects that has been quantized. The acquisition unit 131 acquires processing information associated with the first identification information of the quantized object information generated by vector quantization, which quantizes each subvector obtained by dividing the vector corresponding to each of the multiple objects by Cartesian product quantization. The acquisition unit 131 acquires processing information associated with the first identification information of the quantized object information generated based on each subcentroid corresponding to each sub-region of each centroid corresponding to quantization. The acquisition unit 131 acquires processing information associated with the first identification information of the quantized object information generated based on the residual vector generated by each centroid corresponding to quantization and each subcentroid corresponding within each sub-region of said centroid.
[0376] (Generation unit 132C) The generation unit 132C generates various types of information in the same manner as the generation unit 132, generation unit 132A, or generation unit 132B.
[0377] The generation unit 132C generates blob information that shows multiple blobs for classifying multiple objects, based on the object information acquired by the acquisition unit 131. The generation unit 132C also generates cluster information that shows multiple clusters, which are groups that have a different classification from the multiple blobs, and which are groups that classify objects belonging to the same blob in the multiple blobs into the same group.
[0378] The generation unit 132C generates blob information that classifies multiple objects into a first-number blob, and cluster information that classifies multiple objects into a second-number cluster that is less than the first-number cluster. The generation unit 132C also generates cluster information that shows multiple clusters in which groups of objects belonging to two or more blobs are classified into a single cluster.
[0379] The generation unit 132C generates blob information indicating multiple blobs used for batch processing in search processing targeting multiple objects. The generation unit 132C generates blob information indicating multiple blobs used for batch calculation of distances between multiple objects. The generation unit 132C generates blob information indicating multiple blobs used for parallel processing of distance calculation between search queries and objects used in search processing.
[0380] The generation unit 132C generates cluster information indicating multiple clusters used for processing quantization of multiple objects. The generation unit 132C generates cluster information indicating multiple clusters that will serve as units of quantization. The generation unit 132C generates cluster information indicating multiple clusters used for quantizing the vectors corresponding to each of the multiple objects.
[0381] The generation unit 132C performs classification of multiple clusters after classifying multiple blobs. The generation unit 132C generates blob information indicating multiple blobs by clustering multiple objects. The generation unit 132C generates blob information indicating multiple blobs using index information. The generation unit 132C generates cluster information indicating multiple clusters using blob information indicating blobs. The generation unit 132C generates cluster information indicating multiple clusters by clustering multiple blobs.
[0382] The generation unit 132C classifies multiple blobs after classifying multiple clusters. The generation unit 132C generates cluster information indicating multiple clusters by clustering multiple objects. The generation unit 132C generates blob information indicating multiple blobs using the cluster information indicating clusters. The generation unit 132C generates blob information indicating multiple blobs by dividing at least one cluster from the multiple clusters and generating two or more blobs. The generation unit 132C generates first group information indicating multiple first groups by clustering objects belonging to each second group and dividing each second group into two or more first groups.
[0383] (Search processing unit 133C) The search processing unit 133C functions as a processing unit that executes a first process targeting objects belonging to a blob and a second process targeting objects belonging to a cluster. The search processing unit 133C performs various processes related to the search process, similar to the search processing unit 133, search processing unit 133A, or search processing unit 133B. The search processing unit 133C performs processing using the information generated by the generation unit 132C.
[0384] The search processing unit 133C uses the processing information acquired by the acquisition unit 131 to perform a first process targeting objects belonging to a blob and a second process targeting objects belonging to a cluster. The search processing unit 133C performs the second process targeting a group of objects belonging to a cluster. The search processing unit 133C performs the first process targeting a group of objects belonging to a blob.
[0385] The search processing unit 133C executes a first process that calculates the distance to a group of objects belonging to a single blob all at once. The search processing unit 133C executes a first process that is a parallel process of calculating the distance between each of the objects belonging to a single blob and the search query used in the search process.
[0386] The search processing unit 133C performs a second process on a group of objects belonging to a cluster. The search processing unit 133C performs a second process, which is quantization, on a group of objects belonging to a cluster. The search processing unit 133C performs a second process that quantizes the vector corresponding to each of the objects belonging to a cluster.
[0387] If a cluster contains two or more blobs, the search processing unit 133C performs a second process on the objects belonging to those two or more blobs. If a cluster contains two or more blobs, the search processing unit 133C performs quantization on all objects belonging to those two or more blobs.
[0388] The search processing unit 133C executes the first and second processes by referring to processing information. The search processing unit 133C identifies the object to be processed by referring to processing information. In a search process targeting multiple objects, the search processing unit 133C performs the first process on a blob of one of two or more first identification pieces of information, and generates reference information to be referenced in the search process with respect to a cluster of one second identification piece of information. However, when performing the first process on a blob of another first identification piece of information other than the first first identification piece of information, the search processing unit 133C does not generate reference information.
[0389] The search processing unit 133C performs search processing using a blob-connected graph. The search processing unit 133C performs search processing to find neighboring objects of the search query using the blob-connected graph. The search processing unit 133C performs search processing to find neighboring objects of the search query by traversing the edges that connect blobs in the blob-connected graph.
[0390] The search processing unit 133C, in the search process, uses blob information indicating multiple blobs to calculate the distance between a target object, which is an object belonging to one blob, and the search query, using the vector information of the vector-quantized target object. The search processing unit 133C performs the calculation of the distance between the target object and the search query in parallel and in a batch. The search processing unit 133C performs the calculation of the distance between the target object and the search query in parallel, for a batch processing number determined based on the specifications of the information processing device.
[0391] The search processing unit 133C performs search processing using a blob-connected graph and the processes shown in Figures 27 and 28. For example, the search processing unit 133C performs search processing using a graph GR41 that connects blobs with edges, as shown in spatial information SP51. For example, the search processing unit 133C obtains search results for search query QE3 by performing search processing using the blob-connected graph GR41 as shown in Figures 27 and 28. For example, the search processing unit 133C performs a first process on the blob that is the target of processing by traversing the blob-connected graph.
[0392] The search processing unit 133C generates reference information for the cluster to which the blob being processed belongs if reference information has not yet been generated. For example, the search processing unit 133C generates a distance lookup table (LUT) as reference information. For example, the search processing unit 133C generates a distance lookup table on a cluster basis during the search process. That is, the search processing unit 133C generates a distance lookup table on a cluster basis, not on a blob basis, during the search process. This allows the information processing device 100C to suppress an increase in search time.
[0393] The search processing unit 133C uses the generated reference information to perform a distance calculation on all objects belonging to the blob being processed. For example, if reference information has already been generated for the cluster to which the blob being processed belongs, the search processing unit 133C does not generate any new reference information. Instead, the search processing unit 133C uses the already generated reference information to perform a distance calculation on all objects belonging to the blob being processed.
[0394] [4-3. Processing and Information Examples] From here, we will describe examples of the processing and information used in the processing in the fourth embodiment.
[0395] First, we will explain the concept of quantization using Figure 34. Figure 34 is a conceptual diagram of quantization according to the fourth embodiment.
[0396] For example, base point CL1 indicates the base point used for quantizing cluster CL1. In this case, base point CL1 represents the centroid of cluster CL1, which includes blobs BL1 and BL7. Similarly, base point CL2 indicates the base point used for quantizing cluster CL2. In this case, base point CL2 represents the centroid of cluster CL2, which includes blobs BL2 and BL4.
[0397] For example, base point CL3 indicates the base point used for quantizing cluster CL3. In this case, base point CL3 represents the centroid of cluster CL3, which includes blobs BL3, BL8, and BL10. Similarly, for example, base point CL4 indicates the base point used for quantizing cluster CL4. In this case, base point CL4 represents the centroid of cluster CL4, which includes blobs BL5, BL6, and BL9.
[0398] In this case, the information processing device 100C calculates residual vectors within each of the four clusters CL1 to CL4. Then, the information processing device 100C performs quantization using the residual vectors obtained within each of the four clusters CL1 to CL4. For example, for objects belonging to cluster CL1, the information processing device 100C performs quantization using the residual vector from the base point CL1. A detailed explanation of quantization using residual vectors is omitted here, but the information processing device 100C performs quantization using residual vectors by appropriately employing techniques such as those shown in Patent Document 3.
[0399] Note that Figure 34 shows a state where the vector has not been divided into subvectors for illustrative purposes. However, the information processing device 100C may derive a base point for each subspace divided by direct product quantization and perform quantization using residual vectors based on the derived base points. For example, when the information processing device 100C divides a vector into four subvectors and performs direct product quantization, there is a base point for each subspace corresponding to each subvector, and quantization is performed using residual vectors from those base points.
[0400] The following describes an example of object quantization and processing by the information processing device 100C, with reference to Figure 34. In Figure 34, the case where the query object (search query) is search query QE3 is used as an example. Note that detailed explanations of aspects of object (vector) quantization and Cartesian product quantization described below that are the same as those disclosed in Patent Document 3 will be omitted as appropriate.
[0401] As shown in the spatial information SP52 of Figure 34, the distances between the search query QE3 and each vector corresponding to each object belonging to cluster CL1, such as node N7, are different. However, the information processing device 100C does not use the vectors corresponding to each object belonging to cluster CL1, such as node N7. In this case, the distance between the search query QE3 and each vector is considered to be, for example, the distance between the search query QE3 and the vector corresponding to the base point CL1 of cluster CL1. That is, the vectors corresponding to each object belonging to cluster CL1 are quantized into the vector corresponding to the base point CL1. Specifically, the information processing device 100C calculates the residual vector, which is the difference between each vector from the base point CL1, and performs Cartesian product quantization on that residual vector (further dividing it into subspaces and quantizing them).
[0402] For example, the information processing device 100C further divides each cluster into multiple sub-regions and generates subcentroids corresponding to the multiple sub-regions as a codebook. For example, the information processing device 100C may generate subcentroids corresponding to the multiple sub-regions into which each cluster is divided based on residual vector information for each cluster.
[0403] For example, the information processing device 100C may determine the number of sub-regions of each cluster based on the number of objects contained in each cluster, or it may determine the number of sub-regions based on a predetermined setting value. For example, the information processing device 100C may determine the sub-regions by appropriately using various conventional techniques related to clustering. For example, the information processing device 100C calculates residual vectors from the objects contained in each sub-region of cluster CL1 and the base point CL1 of cluster CL1, and generates subcentroids of the sub-regions from these residual vectors. For example, the information processing device 100C uses information on the residual vectors between the base point of the cluster and the subcentroids of each sub-region. For example, the information processing device 100C uses information on the residual vectors between the base point CL1 of cluster CL1 and the subcentroids of each sub-region. For example, the information processing device 100C stores information on the subcentroids of the sub-regions to which each object belongs, in association with each object.
[0404] For example, the information processing device 100C can use the information described above to quantize the position of each object to the position of the centroid of the subregion to which its vector belongs. This allows the information processing device 100C to refine the distance between the search query QE3 and each object to the distance between the search query QE3 and the vector corresponding to the subcentroid of the subregion to which each object belongs.
[0405] For example, the information processing device 100C uses information that shows the vector of the base point of each cluster and the residual vector related to the partial centroid. This allows the information processing device 100C to calculate the distance to the partial centroid for, for example, a search query QE3. Therefore, the information processing device 100C can determine the distance between, for example, the search query QE3 and each object as the distance to the partial centroid, which is an even closer approximation than the cluster base point.
[0406] Furthermore, the information processing device 100C may process each object's vector by dividing it into multiple subvectors. That is, the information processing device 100C may perform processing using techniques related to so-called direct product quantization. In this respect, it is the same as in the first embodiment described above, so redundant explanations will be omitted as appropriate.
[0407] For example, in Figure 34, the information processing device 100C divides the vector of the search query QE3 into multiple subvectors (also called "subvectors of the search query QE3"). In this case, the spatial information SP52 is also divided into multiple subspaces (also called "subspaces of the spatial information SP52"). For example, the information processing device 100C may divide the vector of the search query QE3 into four parts, similar to Figure 1. In this case, the spatial information SP52 is also divided into four subspaces, and each cluster is divided into multiple subspaces.
[0408] For example, the information processing device 100C stores information indicating the subcentroid (codebook) corresponding to the area to which each object belongs in four subspaces, associating it with each object. For example, the information processing device 100C calculates the distance between each object and the search query QE3 by summing the distances between the four codebooks corresponding to each object and the search query QE3.
[0409] Next, an overview of the inverted index will be explained using Figures 35 and 36. Figure 35 is a diagram showing an example of the inverted index according to the fourth embodiment. Figure 36 is a diagram showing a specific example of the inverted index according to the fourth embodiment.
[0410] First, let's explain the inverted index IND1 shown in Figure 35. The inverted index IND1 shown in Figure 35 includes items such as "Blob ID," "Cluster ID," "Number of Quantized Objects," "Quantized Object #1," "Quantized Object #2," "Quantized Object #3," and "Quantized Object #4." Note that while Figure 35 illustrates the first four items, "Quantized Object #1" through "Quantized Object #4," the inverted index IND1 includes items such as "Quantized Object #5," "Quantized Object #6," etc., corresponding to the number of objects belonging to each blob.
[0411] "Blob ID" indicates identification information for identifying the blob, which is the first group. "Cluster ID" indicates identification information for identifying the cluster, which is the second group. "Number of Quantized Objects" indicates the number of objects belonging to the blob. "Quantized Object #1" to "Quantized Object #4" indicate the quantized objects. For example, "Quantized Object #1" to "Quantized Object #4" store a list of information indicating the codebooks corresponding to each subvector after the objects have been quantized using the Cartesian product.
[0412] In the example in Figure 35, the blob with blob ID "1" belongs to the cluster with cluster ID "2" and has four objects. Furthermore, for the blob with blob ID "1", "quantization object #1" stores "4, 2, 8, 10", indicating that the object is composed of four subvectors, each quantized using codebooks "4", "2", "8", and "10". For example, for the object corresponding to "quantization object #1" of blob ID "1", the first subvector is quantized using codebook "4", the second using codebook "2", the third using codebook "8", and the last using codebook "10".
[0413] Furthermore, the inverted index IND1 may store various types of information depending on the purpose, not limited to those mentioned above. For example, the inverted index IND1 may store information indicating each quantization object (such as an object ID) associated with each quantization object.
[0414] Next, we will explain the inverted index IND2 shown in Figure 36. Figure 36 shows a specific example of an inverted index corresponding to the example shown in Figure 32, etc. Note that explanations of points similar to those of the inverted index IND1 will be omitted as appropriate. The inverted index IND2 shown in Figure 36 includes items such as "Blob ID", "Cluster ID", "Number of Quantized Objects", and "Quantized Object #1". Note that only "Quantized Object #1" is shown in Figure 36, but like the inverted index IND1, it includes a number of items corresponding to the number of objects belonging to each blob.
[0415] In the inverted index IND2, similar to the spatial information SP51 shown in Figure 32, blob BL1, identified by blob ID "BL1", is shown to belong to cluster CL1, which has cluster ID "CL1". It also indicates that the number of objects belonging to blob BL1 is "NM1". Note that in Figure 36, an abstract code such as "NM1" is used, but the "quantized object count" stores a specific number (e.g., 10 or 100) similar to the inverted index IND1.
[0416] Blob BL1 also indicates that "quantization object #1" stores "CD51, CD64, CD77, CD82", and that this object is composed of four subvectors, each quantized by the codebooks "CD51", "CD64", "CD77", and "CD82".
[0417] Furthermore, in the inverted index IND2, similar to the spatial information SP51 shown in Figure 32, blob BL2, identified by blob ID "BL2", belongs to cluster CL2, which has cluster ID "CL2". Also, in the inverted index IND2, similar to the spatial information SP51 shown in Figure 32, blob BL7, identified by blob ID "BL7", belongs to cluster CL1, which has cluster ID "CL1".
[0418] The inverted index IND2 is not limited to the above and may store various types of information depending on the purpose. For example, the inverted index IND2 may store information indicating each quantization object (such as an object ID) associated with each quantization object. In this way, the inverted index is assigned information identifying a cluster (cluster ID), and the information processing device 100C calculates a residual vector using the inverted index. For example, the residual vector is used when generating and searching the index, but since this is the same as the processing in the prior art described in Patent Document 3, a detailed explanation will be omitted as appropriate.
[0419] [4-4. Information Processing Flow] Next, the information processing procedure according to the fourth embodiment will be described using Figures 37 and 38. Figures 37 and 38 are flowcharts showing an example of information processing according to the fourth embodiment.
[0420] First, let's explain Figure 37. For example, Figure 37 is a flowchart of an example of a group generation process. As shown in Figure 37, the information processing device 100C acquires object information indicating multiple objects that are the target of data retrieval (step S601). For example, the information processing device 100C acquires object information indicating multiple objects from the object information storage unit 121.
[0421] The information processing device 100C generates first group information that indicates multiple first groups for classifying multiple objects based on object information (step S602). For example, the information processing device 100C generates blob information that indicates multiple blobs for classifying multiple objects based on object information.
[0422] The information processing device 100C generates second group information indicating multiple second groups, which are groups that have a different classification from multiple first groups, and which classify objects belonging to the same first group in multiple first groups into the same group (step S603). For example, the information processing device 100C generates cluster information indicating multiple clusters in which at least one cluster contains two or more blobs. The information processing device 100C may also generate cluster information indicating clusters that correspond one-to-one with blobs.
[0423] Next, Figure 38 will be explained. For example, Figure 38 is a flowchart showing an example of processing using groups. As shown in Figure 38, the information processing device 100C acquires processing information indicating multiple first groups that classify multiple objects that are the target of data retrieval, and multiple second groups that are classified differently from the multiple first groups, and which classify objects belonging to the same first group in the multiple first groups into the same group (step S701). For example, the information processing device 100C acquires blob information stored in the blob information storage unit 125 and cluster information stored in the cluster information storage unit 127 as processing information. For example, the information processing device 100C acquires the inverted index IND1 or the inverted index IND2.
[0424] The information processing device 100C uses processing information to perform a first process on objects belonging to the first group and a second process on objects belonging to the second group (step S702). For example, the information processing device 100C performs batch processing on objects belonging to a blob and performs quantization on objects belonging to a cluster.
[0425] [5. Fifth Embodiment] Furthermore, when performing search processing using a blob-connected graph (also called a "blob graph") which connects blobs as described above, multiple search ranges from different perspectives may be used. This point will be explained below in the fifth embodiment. In the fifth embodiment, the information processing system 1 has an information processing device 100D instead of information processing device 100, information processing device 100A, information processing device 100B, or information processing device 100C. Note that explanations of points similar to those described above in the first, second, third, and fourth embodiments will be omitted as appropriate. For example, the blob BL* (* is any number) in the following explanation is a group of objects grouped together in the same way as the blob BL* (* is any number) described above, so a detailed explanation will be omitted. Also, the centroid C* (* is any number) in the following explanation is the same as the centroid C* (* is any number) described above, so a detailed explanation will be omitted. In the following explanation, cluster CL* (where * is any number) is a cluster that classifies blobs, similar to the cluster CL* (where * is any number) described above, so a detailed explanation will be omitted.
[0426] Note that in Figures 39 and 40, the representation is the same as in Figure 32, except for the use of thick solid lines to indicate cluster distinctions and dotted lines connecting centroids C of blob BL to indicate connections between blob BLs. For example, among the solid lines, thick lines indicate the correspondence between cluster and blob partitioning, while thin lines indicate the correspondence between blob partitioning. For example, in Figures 39 and 40, the thin solid lines indicate the correspondence between blob partitioning within a cluster. Also, in Figures 39 and 40, centroids CN11 to CN14 represent the respective centroids of clusters CL11 to CL14. Cluster CL11 contains three blob BLs: BL11, BL12, and BL13. Blob BL11 contains nodes N31, N205, etc. Cluster CL12 contains two blob BLs: BL14 and BL15. Cluster CL13 contains three blob BLs: BL16, BL17, and BL18. Cluster CL14 contains two blob BLs: BL19 and BL20. Note that the cluster CLs, blob BLs, and node Ns shown in Figures 39 and 40 are only a portion; the cluster CLs, blob BLs, and node Ns being processed include many more than those shown. Also, the codes for blob BLs and cluster CLs are shown only in Figure 39 and omitted in Figure 40, but Figures 39 and 40 represent the same spatial information (spatial information SP61).
[0427] The information processing device 100D performs a search using a blob graph, which is a group-connected graph in which multiple groups of objects to be searched are linked by edges. The information processing device 100D also performs the search using a range corresponding to the search of a blob group (hereinafter also referred to as the "first search range") and a range corresponding to the search of multiple objects (hereinafter also referred to as the "second search range"). As a result, the information processing device 100D extracts neighboring objects corresponding to the search query from among multiple objects.
[0428] [5-1. Information Processing] First, an overview of the information processing according to the fifth embodiment will be explained using Figures 39 and 40. Figures 39 and 40 are diagrams illustrating an example of information processing according to the fifth embodiment. Specifically, Figure 39 shows an overall overview of the search process executed by the information processing device 100D. Figure 40 shows the changes in the first search range and the second search range in the search process executed by the information processing device 100D. Thus, Figures 39 and 40 explain the changes in the first search range and the second search range and the overview of the search process, while the specific flow of the search process is shown in Figure 43.
[0429] Graph GR61 shown in spatial information SP61 in Figure 39 is a blob graph that connects blobs, similar to graph GR41 shown in spatial information SP51 in Figure 32. For example, the dotted line (edge) connecting the centroids C of blob BL in graph GR61 is equivalent to the arrow line (edge) connecting blob BLs to each other in graph GR41. Graph GR61 in Figure 39 illustrates the portion of blob BL11 to BL20. For example, the information processing device 100D may generate information indicating blob BL11 to BL10 etc. using the method described above, or it may acquire information indicating blob BL11 to BL10 etc. from another device. Note that spatial information SP61-1 to SP61-4 in Figure 40 are denoted by different symbols for the convenience of explaining the changes in the first and second search ranges, but they are the same as spatial information SP61 in Figure 39, and when not distinguished, they may be referred to as spatial information SP61.
[0430] Note that graph GR61 may also be a representation of the portion of graph GR41 corresponding to each blob BL11 to BL20. In this case, spatial information SP51 and spatial information SP61 target the same multiple objects, and graph GR61 and graph GR41 represent different portions of the same graph. For example, graph GR41 in Figure 32 may represent the portion of blobs BL1 to BL10, and graph GR61 may represent the portion of blobs BL11 to BL20. Alternatively, graph GR61 may be a graph in which the centroids of each blob are connected, as shown in graph GR42 in Figure 29. In other words, graph GR61 may be a graph in which the blobs are connected in any manner, as long as the search process described later can be executed.
[0431] Figure 39 shows an example of the process when the query is search query QE11 and the number of objects to extract as neighboring objects (also called the "number of neighboring objects extracted") is "5". Of the five circles centered on search query QE11, the dotted circle represents the blob search range SB, which is an example of the first search range. In Figure 39, only the circle with the largest radius among the three dotted circles representing the blob search range SB is labeled "SB", but the three dotted circles represent the blob search range SB that changes during the search process.
[0432] For example, the blob search range SB is the first search range defined by the radius (blob search radius) centered on the search query QE11. As shown by the dashed arrow in Figure 39, the blob search range SB expands as the search process progresses, a point that will be explained later using Figure 40. Note that the blob search ranges SB1, SB2, and SB3 shown in Figure 40 correspond to the three blob search ranges SB in Figure 39. When explaining blob search ranges SB1, SB2, and SB3 without distinguishing between them, they may be simply referred to as blob search range SB.
[0433] Furthermore, in Figure 39, among the five circles centered on the search query QE11, the dotted circle represents an example of the first search range, which is the object search range SO. Note that in Figure 39, only the outer of the two dotted circles representing the object search range SO is labeled "SO," but both dotted circles represent the object search range SO that changes during the search process.
[0434] For example, the object search range SO is a second search range defined by the radius (object search radius) centered on the search query QE11. As shown by the dashed arrow in Figure 39, the object search range SO decreases in size (radius) as the search process progresses. This point will be explained later using Figure 40. Note that the object search ranges SO1 and SO2 shown in Figure 40 correspond to the two object search ranges SO in Figure 39, respectively. When explaining without distinguishing between object search ranges SO1 and SO2, they may be referred to simply as object search range SO.
[0435] First, the information processing device 100D starts the search process from the state shown in spatial information SP61-1. Then, as shown in spatial information SP61-2, the information processing device 100D determines the blob search range SB to blob search range SB1 and the object search range SO to infinity "∞". That is, in Figure 40, the information processing device 100D determines the blob search range SB to blob search range SB1, which is defined by a radius based on the distance between the nearest blob BL20 and the search query QE11. Also in Figure 40, the information processing device 100D extracts three objects (corresponding to each of the nodes N2, N6, and N8 to which blob BL20 belongs) as candidates for nearby objects corresponding to the search query QE11 and adds them to the candidates for nearby objects. Since there are three candidates for nearby objects, and the number of extracted nearby objects is "5" or less, the object search range SO is maintained at infinity "∞".
[0436] Then, as shown in spatial information SP61-3, the information processing device 100D determines the blob search range SB to be blob search range SB2 and the object search range SO to be object search range SO1. That is, in Figure 40, the information processing device 100D determines the blob search range SB to be blob search range SB2, which is defined by a radius based on the distance between blob BL11, the next nearest blob to blob BL20, and the search query QE11. Also in Figure 40, the information processing device 100D extracts three objects (corresponding to each) of nodes N31, N205, and N312, to which blob BL11 belongs, as candidates for nearby objects corresponding to the search query QE11, and adds them to the list of nearby object candidates. As a result, the list of nearby object candidates includes six nodes: N2, N6, N8, N31, N205, and N312, which exceeds the number of nearby objects extracted, "5". Therefore, the information processing device 100D keeps the five nearest neighbor object candidates from the search query QE11 and discards the rest. In this case, the information processing device 100D discards node N312, which is the furthest from the search query QE11 among the six nodes, leaving five neighbor object candidates. That is, the information processing device 100D leaves five neighbor object candidates: nodes N2, N6, N8, N31, and N205. Since the number of neighbor object candidates has reached the neighbor object extraction number "5", the information processing device 100D determines the object search range SO to be the object search range SO1, which is defined by the radius based on the distance between node N205, the furthest from the search query QE11 among the neighbor object candidates, and the search query QE11.
[0437] Then, as shown in spatial information SP61-4, the information processing device 100D determines the blob search range SB to be blob search range SB3 and the object search range SO to be object search range SO2. That is, in Figure 40, the information processing device 100D determines the blob search range SB to be blob search range SB3, which is defined by a radius based on the distance between blob BL19, the next neighbor to blob BL11, and the search query QE11. Also in Figure 40, the information processing device 100D extracts three objects (corresponding to each) of nodes N3, N11, and N114, to which blob BL19 belongs, as candidates for neighboring objects corresponding to the search query QE11, and adds them to the neighboring object candidates. As a result, the neighboring object candidates include eight nodes: N2, N6, N8, N31, N205, N3, N11, and N114, which exceeds the number of neighboring objects extracted, "5". Therefore, the information processing device 100D keeps the five nearest neighbor object candidates from the search query QE11 and discards the rest. In this case, the information processing device 100D discards the three nodes N205, N3, and N114 that are furthest from the search query QE11, leaving five neighbor object candidates. That is, the information processing device 100D leaves five neighbor object candidates: nodes N2, N6, N8, N11, and N31. Then, the information processing device 100D determines the object search range SO to be the object search range SO2, which is defined by a radius based on the distance between the node N11, which is the furthest from the search query QE11 among the neighbor object candidates, and the search query QE11.
[0438] The information processing device 100D repeats the above-described process until it meets one of the termination criteria #1 to #3 described later. For example, if it meets one of the termination criteria #1 to #3, the information processing device 100D determines that the objects included in the candidate neighbor objects at that point are neighbor objects of the search query QE11. The information processing device 100D then provides information indicating the objects determined to be neighbor objects of the search query QE11 to the search requester or other party specifying the search query QE11.
[0439] As described above, the information processing device 100D performs search processing using search ranges from multiple perspectives, such as a first search range and a second search range, and by changing each search range according to the status of the search processing, it can perform flexible search processing using a graph.
[0440] For example, in blob graphs, large blob graphs can experience slower search speeds due to the referencing of unnecessary blobs. Furthermore, while traditional product quantization (PQ) involves searching a graph to identify neighboring blobs and then calculating approximate distances between them, this method also suffers from slower search speeds due to the referencing of unnecessary blobs.
[0441] On the other hand, the information processing device 100D can reduce unnecessary object searches by searching the blob graph, calculating the approximate distance of objects within each blob whose ranking can be determined in order of proximity, and terminating the search when a predetermined termination criterion is met.
[0442] [5-2. Configuration of Information Processing Equipment] Next, the configuration of the information processing device 100D according to the fifth embodiment will be described using Figure 41. Figure 41 is a diagram showing an example of the configuration of the information processing device according to the fifth embodiment. As shown in Figure 41, the information processing device 100D has a communication unit 110, a storage unit 120C, and a control unit 130B. Note that in the information processing device 100D, the same points as those in the information processing device 100, information processing device 100A, information processing device 100B, or information processing device 100C will be omitted from the explanation as appropriate.
[0443] (Storage unit 120C) The storage unit 120C is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120C according to the fifth embodiment, as shown in Figure 41, includes an object information storage unit 121, a graph information storage unit 122, a quantization information storage unit 123, a codebook information storage unit 124, a blob information storage unit 125, a blob-linked graph information storage unit 126, and a cluster information storage unit 127.
[0444] (Control Unit 130D) The control unit 130D is a controller, and is implemented, for example, by a CPU, MPU, GPU, etc., executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the information processing device 100D using RAM as the working area. Alternatively, the control unit 130D is a controller and can be implemented, for example, by an integrated circuit such as an ASIC or FPGA.
[0445] As shown in Figure 41, the control unit 130D includes an acquisition unit 131, a generation unit 132C, a search processing unit 133D, and a provisioning unit 134, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130D is not limited to the configuration shown in Figure 41, and other configurations are also acceptable as long as they perform the information processing described later.
[0446] The acquisition unit 131 according to the fifth embodiment acquires various information from the storage unit 120C. For example, the acquisition unit 131 according to the fifth embodiment acquires various information used by the generation unit 132C, similar to the acquisition unit 131 according to the fourth embodiment.
[0447] The acquisition unit 131 acquires a group linkage graph in which multiple groups, each containing multiple objects to be searched, are linked by edges. The acquisition unit 131 also acquires search queries for the multiple objects.
[0448] (Search processing unit 133D) The search processing unit 133D executes the search process shown in Figure 43. For example, the search processing unit 133D performs various search-related processes in the same way as the search processing unit 133, search processing unit 133A, search processing unit 133B, or search processing unit 133BC. The search processing unit 133D performs processing using the information stored in the storage unit 120C. The search processing unit 133D performs processing using the information generated by the generation unit 132C.
[0449] The search processing unit 133D performs a search process to extract neighboring objects corresponding to the search query from multiple objects by exploring the group linkage graph using a first search range that indicates a range corresponding to the search of multiple groups and a second search range that indicates a range corresponding to the search of multiple objects. The search processing unit 133D performs the search process using the first search range and the second search range which has a wider range than the first search range.
[0450] The search processing unit 133D executes the search using a first search range that expands as the search process progresses. The search processing unit 133D executes the search using a first search range determined based on one group among several groups that is the target of the search process. The search processing unit 133D executes the search using a first search range determined based on one group during the search process. The search processing unit 133D executes the search using a first search range based on the distance between the search query and the object closest to the search query among the objects included in one group, or the centroid of one group.
[0451] The search processing unit 133D executes the search using a second search range that narrows as the search process progresses. The search processing unit 133D executes the search using a second search range determined based on one object among multiple objects that is the target of the search process. During the search process, the search processing unit 133D executes the search using a second search range determined based on one object that is the furthest from the search query among the objects extracted as candidate neighboring objects corresponding to the search query. The search processing unit 133D executes the search using a second search range based on the distance between the search query and one object.
[0452] The search processing unit 133D terminates the search process if a predetermined termination condition is met. The search processing unit 133D terminates the search process if the number of groups targeted by the search process meets the condition. The search processing unit 133D terminates the search process if the number of groups targeted by the search process exceeds a predetermined threshold.
[0453] The search processing unit 133D terminates the search process if the distance to a predetermined group exceeds the second search range. The search processing unit 133D terminates the search process if the distance to a predetermined group during the search process exceeds the second search range. The search processing unit 133D terminates the search process if the distance between the object closest to the search query among the objects included in the predetermined group and the search query exceeds the second search range.
[0454] The search processing unit 133D terminates the search process if the relative size relationship between the first search range and the second search range satisfies the condition. The search processing unit 133D terminates the search process if the first search range exceeds the second search range.
[0455] [5-3. Information Processing Flow] Next, the procedure for information processing according to the fifth embodiment will be described using Figure 42. Figure 42 is a flowchart showing an example of information processing according to the fifth embodiment.
[0456] First, let's explain Figure 42. For example, Figure 42 is a flowchart showing an example of the group generation process. As shown in Figure 42, the information processing device 100D obtains a group-connected graph in which multiple groups, each containing multiple objects to be searched for, are linked by edges (step S801). For example, the information processing device 100D obtains a blob-connected graph from the blob-connected graph information storage unit 126 as a group-connected graph.
[0457] The information processing device 100D obtains search queries for multiple objects (step S802). For example, the information processing device 100D obtains search queries from a terminal device 10 used by a user (see Figure 4).
[0458] The information processing device 100D performs a search process to extract neighboring objects corresponding to a search query from multiple objects by searching a group linkage graph using a first search range that indicates a range corresponding to the search of multiple groups and a second search range that indicates a range corresponding to the search of multiple objects (step S803). For example, the information processing device 100D performs the search process shown in Figure 43 using the first search range, which is the blob search range, and the second search range, which is the object search range.
[0459] [5-4. Search Processing Example] Here, an example of the search process according to the fifth embodiment will be explained using Figure 43 as an example. Figure 43 is a flowchart of an example of the search process according to the fifth embodiment. The search process described below is performed by the search processing unit 133D of the information processing device 100D. Note that explanations of points similar to those described in the first, second, third, and fourth embodiments will be omitted as appropriate.
[0460] The information processing device 100D obtains a set of search blob origins and calculates the distance between each object and the query (step S901). For example, the information processing device 100D obtains a set of search blob origins that represent the blob that will be the starting point of the search, and calculates the distance between each object contained in the blob of the search blob origin set and the query. For example, the information processing device 100D may randomly select a set of search blob origins, or it may select a set of search blob origins using a predetermined index, such as an index of a tree structure with blobs as leaves. For example, the information processing device 100D calculates the approximate distance between each object contained in the blob of the search blob origin set and the query by calculating the approximate distance as described above.
[0461] The information processing device 100D designates the nearest blob of the query in the search blob origin set as the current nearest neighbor blob B (hereinafter also simply referred to as "blob B") (step S902). For example, the information processing device 100D determines that the blob closest to the query in the search blob origin set is the current nearest neighbor blob B.
[0462] The information processing device 100D sets various parameters (step S903). In Figure 43, the information processing device 100D sets the blob search count to "1". The information processing device 100D also sets the object search radius R, which indicates the second search range (object search range), to "∞ (infinity)". For example, the object search radius R is the radius centered on the query q that defines the second search range. The information processing device 100D also sets the object search radius E, which is an example of the second search range, to "∞ (infinity)".
[0463] Furthermore, the information processing device 100D sets the blob search radius Rb, which is an example of the first search range (blob search range), to "d(q,B)". For example, the blob search radius Rb is the radius centered on the query q that defines the first search range. For example, the information processing device 100D sets the blob search radius Rb to the distance d(q,B) between the query q and the current nearest neighbor blob B. For example, the information processing device 100D sets the blob search radius Rb to the distance d(q,B) between the query q and the object closest to the query q among the objects contained in the current nearest neighbor blob B. Note that the blob search radius Rb may also be the distance between the query q and the centroid of the current nearest neighbor blob B. Furthermore, the information processing device 100D sets the blob search radius Eb, which indicates the first search range (blob search range), to "Rb × (1 + eb)". Here, the parameter eb is the blob search radius expansion factor (also called the "first search range coefficient"), and any value such as 0.1 can be set. For example, the information processing device 100D sets the blob search radius Eb to a value obtained by multiplying the blob search radius Rb by 1 + eb.
[0464] The information processing device 100D adds the search blob origin set other than blob B to the excluded blob set (step S904). For example, the information processing device 100D adds the blobs from the search blob origin set, excluding the current nearest neighbor blob B, to the excluded blob set.
[0465] The information processing device 100D adds the search blob origin set to the unexplored blob set and the distance-calculated set (step S905). For example, the information processing device 100D adds all blobs from the search blob origin set, including the current nearest neighbor blob B, to the unexplored blob set, indicating that they have not yet been processed for exploration. For example, the information processing device 100D adds all blobs from the search blob origin set, including the current nearest neighbor blob B, to the distance-calculated set, indicating that distance calculation processing has been performed for the objects within those blobs.
[0466] The information processing device 100D determines whether the unexplored blob set is empty (step S906). If the unexplored blob set is not empty (step S906: No), the information processing device 100D obtains the blob T that is closest to the query from the unexplored blob set (step S907). This removes the unexplored blob set from the unexplored blob set. For example, the information processing device 100D obtains the blob with the shortest distance to query q from the unexplored blob set as blob T. For example, the distance to query q is the distance between query q and the object in the blob that is closest to query q. Note that the distance to query q may also be the distance between query q and the centroid of the blob.
[0467] The information processing device 100D determines whether blob T is within the blob search range (step S908). For example, the information processing device 100D determines whether the distance between blob T and query q is less than or equal to the blob search radius Eb.
[0468] If blob T is within the blob search range (step S908: Yes), the information processing device 100D determines whether all blobs connected to blob T by its edges have been processed (step S909). For example, the information processing device 100D uses information indicating whether or not blobs connected to blob T by edges (also called "connected blobs") have been processed ("processing status information") to determine whether or not all connected blobs of blob T have been processed. For example, the information processing device 100D may manage the processing status of connected blobs of blob T using information such as flags. For example, the information processing device 100D may update the processing status information stored in the storage unit 120C according to the processing status, and then refer to the processing status information in the storage unit 120C to determine whether or not all connected blobs of blob T have been processed.
[0469] If the information processing device 100D has not processed all blobs connected to blob T by edges (step S909: No), it obtains blobs J that are connected to blob T by edges (step S910). For example, the information processing device 100D obtains blobs J that are connected to blob T by edges and have not yet been processed as blobs connected to blob T by edges (also called "unprocessed connected blobs"). For example, the information processing device 100D refers to the processing status information, obtains unprocessed connected blobs as blobs J, and updates the processing status information for the blobs obtained as blobs J to processed.
[0470] The information processing device 100D determines whether blob J exists in the set of distance calculations (step S911). If blob J exists in the set of distance calculations (step S911: Yes), the information processing device 100D returns to step S909 and processes the data.
[0471] Furthermore, if blob J does not exist in the set of distance-calculated blobs (step S911: No), the information processing device 100D adds blob J to the set of distance-calculated blobs (step S912) and adds blob J to the set of unexplored blobs (step S913). Then, the information processing device 100D calculates the distance De between the centroid of blob J and the query (step S914).
[0472] The information processing device 100D determines whether the distance De is less than the blob search radius Rb (step S915). If the distance De is not less than the blob search radius Rb (step S915: No), the information processing device 100D adds blob J to the set of excluded blobs (step S916) and returns to step S909 to process.
[0473] Furthermore, if the distance De is smaller than the blob search radius Rb (step S915: Yes), the information processing device 100D adds the current nearest neighbor blob B to the set of excluded blobs, and sets blob J as the current nearest neighbor blob B (step S917). In this way, the information processing device 100D updates the current nearest neighbor blob B to blob J when the distance De is smaller than the blob search radius Rb.
[0474] Then, the information processing device 100D updates the parameters related to the blob based on the updated nearest neighbor blob B in blob J (step S918). In Figure 43, the information processing device 100D updates the blob search radius Rb to "d(q,B)" based on the updated nearest neighbor blob B in blob J. The information processing device 100D also updates the blob search radius Eb to "Rb×(1+eb)" based on the updated blob search radius Rb. Then, the information processing device 100D returns to step S909 to perform the processing.
[0475] If the information processing device 100D has processed all blobs connected by the edges of blob T (step S909: Yes), it returns to step S906 and processes again. If the set of unexplored blobs is empty (step S906: Yes), the information processing device 100D performs the process in step S920. Also, if blob T is not within the blob search range (step S908: No), the information processing device 100D returns blob T to the unexplored blob (step S919) and performs the process in step S920.
[0476] In step S920, the information processing device 100D adds 1 to the blob search count (step S920). Then, the information processing device 100D calculates the approximate distance between all objects in blob B and the shortest distance, and sets the blob distance of blob B as the blob distance of blob B (step S921). For example, the information processing device 100D calculates the approximate distance between all objects in blob B and query q, and sets the distance between query q and the object closest to query q among the objects included in the current nearest neighbor blob B as the blob distance of blob B. The blob distance of blob B is used in the termination determination process in step S924.
[0477] The information processing device 100D adds all objects in blob B to the search results set, sorts them, and sets only the top-ranking objects by the specified search count as the search results set (step S922). For example, the information processing device 100D adds all objects in blob B to the search results set which shows candidate objects in the vicinity of query q. Then, the information processing device 100D updates the search results set by sorting the objects in the search results set after adding the objects in blob B in order from the shortest distance from query q, and removes objects from the updated search results set that are lower than the specified search count (e.g., 5, 10, etc.). For example, the information processing device 100D keeps the objects in the search results set after adding the objects in blob B, from the shortest distance from query q up to the search count (e.g., 5, 10, etc.) (i.e., objects equal to the search count), and removes the rest.
[0478] Then, the information processing device 100D updates the parameters related to the object based on the updated search result set (step S923). In Figure 43, the information processing device 100D updates the object search radius R to the maximum distance of objects in the search result set based on the updated search result set. For example, the information processing device 100D updates the object search radius R to the distance of the object furthest from query q in the search result set. The information processing device 100D also sets the object search radius E to "R × (1 + eo)" based on the updated search result set. Here, the parameter eo is the object search radius expansion factor (also called the "second search range coefficient"), and can be set to any value such as 0.05. For example, the parameter eo corresponds to the search range coefficient "ε" mentioned above. For example, the information processing device 100D sets the object search radius E to the value obtained by multiplying the object search radius R by 1 + eo. The parameters eo and eb may have the same value.
[0479] Then, the information processing device 100D determines whether the processing status meets any of the termination criteria #1 to #3 (step S924). The information processing device 100D determines whether the processing status meets any of the termination criteria #1 to #3 based on each termination condition. Termination criterion #1 is that the process terminates when the number of blob searches exceeds the upper limit. For example, termination criterion #1 is a criterion that is conditional on the number of blob searches, which is a parameter, exceeding the upper limit (for example, any value such as 5 or 10).
[0480] Termination criterion #2 terminates when the blob distance exceeds the object's search radius. For example, termination criterion #2 is a criterion that requires the blob distance to exceed the object's search radius E. Termination criterion #3 terminates when the blob's search radius exceeds the object's search radius. For example, termination criterion #3 is a criterion that requires the blob search radius Eb to exceed the object's search radius E. Note that termination criteria #1 to #3 above are merely examples, and various termination criteria may be used. Furthermore, multiple termination criteria may be used in combination.
[0481] If the information processing device 100D does not meet termination criteria #1 to #3 (step S924: No), it retrieves the blob with the shortest distance from the set of excluded blobs and sets it as the current nearest neighbor blob B (step S925). For example, the information processing device 100D retrieves the blob with the shortest distance from query q from the set of excluded blobs as the current nearest neighbor blob B. As a result, the blob with the shortest distance at that time is removed from the set of excluded blobs. For example, the information processing device 100D updates the current nearest neighbor blob B to the blob with the shortest distance from query q from the set of excluded blobs.
[0482] Then, the information processing device 100D updates the parameters related to the blob based on the current nearest neighbor blob B, which has been updated to the blob at the shortest distance (step S926). In Figure 43, the information processing device 100D updates the blob search radius Rb to "d(q,B)" based on the current nearest neighbor blob B, which has been updated to the blob at the shortest distance. The information processing device 100D also updates the blob search radius Eb to "Rb×(1+eb)" based on the updated blob search radius Rb. Then, the information processing device 100D returns to step S906 and processes the data.
[0483] Furthermore, the information processing device 100D terminates processing if it meets termination criteria #1 to #3 (step S924: Yes). For example, if it meets termination criteria #1 to #3, the information processing device 100D determines that the objects included in the search result set at that point are neighboring objects to query q. The information processing device 100D then provides information indicating the objects determined to be neighboring objects to query q to the requester of the search specifying query q.
[0484] [6. Sixth Embodiment] The above examples show a process in which objects are clustered (classified) into multiple blobs, and each blob is further clustered (classified) into multiple quantization clusters (also simply called "clusters"). For example, in the fourth and fifth embodiments, the number of first groups (blobs) (first number) is shown to be greater than the number of second groups (clusters) (second number), which are the units for quantizing objects. However, the number of second groups (clusters) (second number) may also be greater than the number of first groups (blobs). For example, the information processing device 100E according to the sixth embodiment may generate the first and second groups by hierarchical clustering (also called "hierarchical clustering"), which further classifies the classification of the second groups (clusters) into first groups (blobs). That is, the second group (cluster) may be at a higher hierarchical level than the first group (blob). As shown in the fourth and fifth embodiments, the first group (blob) may be at a higher hierarchical level than the second group (cluster).
[0485] This point will be explained below in the form of a sixth embodiment. In the sixth embodiment, the information processing system 1 has an information processing device 100E instead of information processing devices 100, 100A, 100B, 100C, or 100D. Note that explanations of points similar to those described above in the first, second, third, fourth, and fifth embodiments will be omitted as appropriate.
[0486] The information processing device 100E generates blob information (first group information) that indicates multiple blobs for classifying multiple objects, and cluster information (second group information) that indicates clusters, which are groups classified differently from blobs. In the following, the classification process that classifies multiple objects into the first group (blobs) may be referred to as the first classification process, and the classification process that classifies multiple objects into the second group (clusters) may be referred to as the second classification process.
[0487] [6-1. Information Processing] First, an overview of the information processing according to the sixth embodiment will be explained using Figure 44. Figure 44 is a diagram showing an example of the information processing according to the sixth embodiment. Specifically, Figure 44 is a diagram showing a first generation example of the first group and the second group in the sixth embodiment. In the following, we will show the case where the information processing device 100E performs the second classification processing and then the first classification processing, but if the desired first group and second group can be generated, the second classification processing may be performed after the first classification processing.
[0488] Before explaining the information processing by the information processing device 100E, the illustration of Figure 44 will be briefly explained. In Figure 44, blobs are distinguished by thick solid lines, and connections between blobs are shown by dotted lines connecting the centroids C of blob BL. In this way, when explaining without distinguishing the centroids of each blob BL, they are written as "centroid C". For example, among the solid lines, the thick lines indicate partitions corresponding to blobs, and the thin lines indicate partitions corresponding to clusters. For example, in Figure 44, the thin lines among the solid lines indicate partitions corresponding to clusters within a blob.
[0489] Figure 44 shows centroids CN21 to CN30, representing the respective centroids of clusters CL21 to CL30. When describing the centroids of each cluster CL without distinction, they will be referred to as "centroid CN". Figure 44 also shows centroids C21 to C24, representing the respective centroids of blobs BL21 to BL24. Blob BL21 contains three cluster CLs: clusters CL21, CL22, and CL23. Cluster CL21 includes nodes N31, N205, etc. Blob BL22 contains two cluster CLs: clusters CL24 and CL25. Blob BL23 contains three cluster CLs: clusters CL26, CL27, and CL28. Blob BL24 contains two cluster CLs: clusters CL29 and CL30. Note that the blob BL, cluster CL, and node N shown in Figure 44 are only a portion; the blob BL, cluster CL, and node N to be processed include many more than those shown. The specific search process flow using the graph GR71 shown in spatial information SP71 is shown in Figure 48.
[0490] An example of the generation process of the first and second groups by the information processing device 100E will be described. First, the information processing device 100E performs a second classification process and generates blob information that classifies multiple objects into multiple blobs. For example, the information processing device 100E generates blob information that shows the blobs that classify multiple objects by any method. The information processing device 100E performs a second classification process that classifies multiple objects into multiple blobs by any clustering method such as k-means.
[0491] In the example shown in Figure 44, the information processing device 100E classifies multiple objects into multiple blobs such as blobs BL21 to BL24. The information processing device 100E may generate clusters by any method, as long as it can generate multiple objects. For example, the information processing device 100E may generate blob information using an index such as a graph (e.g., graph GR71).
[0492] Furthermore, the information processing device 100E generates cluster information that classifies multiple objects into multiple clusters (step S71). The information processing device 100E generates cluster information that classifies multiple objects into clusters CL21 to CL30, etc., by any method. For example, the information processing device 100E generates cluster information that classifies multiple objects into multiple clusters using the information generated by executing the second classification process. In the example in Figure 44, the information processing device 100E generates cluster information that classifies multiple objects into multiple clusters by executing a first classification process that classifies each of the multiple blobs, such as blobs BL21 to BL24, into multiple clusters. The information processing device 100E generates cluster information that classifies multiple objects into multiple clusters by classifying each of the multiple objects contained in each of the multiple blobs, such as blobs BL21 to BL24, into multiple clusters.
[0493] The above is merely an example, and the information processing device 100E may generate clusters by any method as long as it is capable of generating clusters. The information processing device 100E may classify each node (object) into one of the clusters CL21 to CL30 using any clustering method such as k-means. For example, the information processing device 100E may generate clusters using an index such as a graph.
[0494] In the example shown in Figure 44, the information processing device 100E generates cluster information such as clusters CL21 to CL30, which are obtained by further dividing blobs BL21 to BL24. For example, the information processing device 100E may classify each object into one of the clusters CL21 to CL30 by clustering the objects belonging to each blob BL21 to BL24.
[0495] In the example shown in Figure 44, the information processing device 100E divides blob BL21 into clusters CL21, CL22, and CL23. That is, the information processing device 100E classifies the objects belonging to blob BL21 into objects belonging to cluster CL21, objects belonging to cluster CL22, and objects belonging to cluster CL23. Note that in Figure 44, the cluster boundaries are shown as being on the blob boundaries, but they may not coincide depending on the division method, and depending on the method, the space may not necessarily be divided by the boundaries. Also, Figure 44 conceptually shows objects, centroids, etc., to explain the generation process, and the positions of centroids, etc., do not represent their exact positions.
[0496] Furthermore, the information processing device 100E divides blob BL22 into clusters CL24 and CL25. For example, the information processing device 100E divides blob BL23 into clusters CL26, CL27, and CL28. The information processing device 100E divides blob BL24 into clusters CL29 and CL30. In this way, the information processing device 100E generates information for the second group using the information for the first group. In this way, the information processing device 100E performs the second classification process using the first group information generated by the first classification process.
[0497] In Figure 44, the information processing device 100E generates a blob-connected graph GR71, as shown in the spatial information SP71, but it is not necessary to generate the graph GR71. For example, when the information processing device 100E generates the graph GR71, it does so by processing information similar to that used in Figures 23 and 29, but a detailed explanation is omitted as it is the same as that described in Figures 23 and 29.
[0498] For example, the information processing device 100E performs quantization processing using blob information. The information processing device 100E performs quantization processing on multiple objects using blob information. For example, the information processing device 100E performs quantization for each object belonging to each of the clusters CL21 to CL30, but this will be explained later.
[0499] As described above, the information processing device 100E generates cluster information indicating multiple clusters that classify multiple objects that are the target of data retrieval, and blob information indicating multiple blobs that have a different classification from the multiple clusters. Specifically, the information processing device 100E generates a first group (blob) of a first number and a second group (cluster) of a second number that is greater than the first number.
[0500] As a result, the information processing device 100E can generate two types of groups for classifying multiple objects, and can generate groups for classifying objects. For example, in the case of a data structure in which multiple blobs (first group) are contained within a single quantization cluster (second group), high search performance may not be achieved. Even in such cases, the information processing device 100E can improve search performance by creating a first group with a first number and a second group with a second number greater than the first number, thereby creating an appropriate group relationship.
[0501] [6-1-1. Other processing examples] Note that the process shown in Figure 44 is merely one example, and the information processing device 100E may generate blob information and cluster information by any method of processing, as long as it is capable of generating them.
[0502] For example, the information processing device 100E may generate cluster information for classifying objects into clusters, and then generate blob information for classifying objects into blobs. In this case, the information processing device 100E may generate cluster information for classifying each object into one of the multiple clusters by clustering, which classifies multiple objects into clusters. For example, the information processing device 100E may classify each object into one of the clusters CL21 to CL30 by any clustering method such as k-means.
[0503] The information processing device 100E then generates blob information, classifying clusters CL21 to CL30 into blobs BL21 to BL24, etc. The information processing device 100E may classify clusters CL21 to CL30, etc. into one of the multiple blobs using any clustering method such as k-means. The information processing device 100E may generate blobs by any method that enables blob generation. For example, the information processing device 100E may generate blobs using an index such as a graph.
[0504] For example, the information processing device 100E classifies clusters CL21, CL22, and CL23 into blob BL21. The information processing device 100E also classifies clusters CL24 and CL25 into blob BL22. For example, the information processing device 100E classifies clusters CL26, CL27, and CL28 into blob BL23. The information processing device 100E classifies clusters CL29 and CL30 into blob BL24. As a result, the information processing device 100E generates information for the first group using the information for the second group. In this way, the information processing device 100E may perform the first classification process using the second group information generated by the second classification process.
[0505] [6-2. Configuration of Information Processing Equipment] Next, the configuration of the information processing device 100E according to the sixth embodiment will be described using Figure 45. Figure 45 is a diagram showing an example of the configuration of the information processing device according to the sixth embodiment. As shown in Figure 45, the information processing device 100E has a communication unit 110, a storage unit 120C, and a control unit 130B. Note that in the information processing device 100E, the same points as in the information processing device 100, information processing device 100A, information processing device 100B, information processing device 100C, or information processing device 100D will be omitted from the explanation as appropriate.
[0506] (Storage unit 120C) The storage unit 120C is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120C according to the sixth embodiment, as shown in Figure 45, includes an object information storage unit 121, a graph information storage unit 122, a quantization information storage unit 123, a codebook information storage unit 124, a blob information storage unit 125, a blob-linked graph information storage unit 126, and a cluster information storage unit 127.
[0507] (Control Unit 130E) The control unit 130E is a controller, and is implemented, for example, by a CPU, MPU, GPU, etc., executing various programs (corresponding to an example of an information processing program) stored in the memory device inside the information processing device 100E using RAM as the working area. Alternatively, the control unit 130E is a controller and can be implemented, for example, by an integrated circuit such as an ASIC or FPGA.
[0508] As shown in Figure 45, the control unit 130E includes an acquisition unit 131, a generation unit 132E, a search processing unit 133E, and a provisioning unit 134, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130E is not limited to the configuration shown in Figure 45, and other configurations are also acceptable as long as they perform the information processing described later.
[0509] The acquisition unit 131 according to the sixth embodiment acquires object information indicating a plurality of objects that are the target of data retrieval. The acquisition unit 131 acquires index information indicating an index that targets the plurality of objects for retrieval.
[0510] The acquisition unit 131 according to the sixth embodiment acquires various types of information, similar to the acquisition unit 131 according to the first to fifth embodiments. For example, the acquisition unit 131 acquires object information indicating multiple objects that are the target of data retrieval. For example, the acquisition unit 131 acquires index information indicating an index that targets multiple objects for retrieval.
[0511] (Generation unit 132E) The generation unit 132E generates various types of information in the same manner as the generation unit 132, generation unit 132A, generation unit 132B, or generation unit 132C.
[0512] The generation unit 132E generates, based on the object information acquired by the acquisition unit 131, first group information indicating multiple first groups, which are first number groups that classify multiple objects, and second group information indicating multiple second groups, which are second number groups that classify multiple objects differently from the multiple first groups and are greater than the first number of groups.
[0513] The generation unit 132E generates first group information, which is a group of objects that belong to the same second group in multiple second groups from among multiple objects, and classify objects belonging to the same second group in the same first group. The generation unit 132E generates first group information that shows multiple first groups in which groups of objects belonging to two or more second groups are classified into one first group.
[0514] The generation unit 132E generates first group information indicating multiple first groups used for batch processing in a search process targeting multiple objects. The generation unit 132E generates first group information indicating multiple first groups used for batch calculation of distances between multiple objects. The generation unit 132E generates first group information indicating multiple first groups used for parallel processing of distance calculations between search queries and objects used in the search process.
[0515] The generation unit 132E generates second group information indicating multiple second groups used for processing quantization of multiple objects. The generation unit 132E generates second group information indicating multiple second groups that will serve as units for quantization. The generation unit 132E generates second group information indicating multiple second groups used for quantizing the vectors corresponding to each of the multiple objects.
[0516] The generation unit 132E performs classification of multiple second groups, followed by classification of multiple first groups. The generation unit 132E generates second group information indicating multiple second groups by clustering multiple objects. The generation unit 132E generates second group information indicating multiple second groups using index information. The generation unit 132E generates first group information indicating multiple first groups using the second group information indicating second groups.
[0517] The generation unit 132E generates first group information representing multiple first groups by clustering multiple second groups. After classifying multiple first groups, the generation unit 132E classifies multiple second groups. The generation unit 132E generates first group information representing multiple first groups by clustering multiple objects.
[0518] The generation unit 132E generates second group information indicating multiple second groups using first group information indicating a first group. The generation unit 132E generates second group information indicating multiple second groups by dividing at least one first group from the multiple first groups and generating two or more second groups. The generation unit 132E generates second group information indicating multiple second groups by clustering objects belonging to each first group and dividing each first group into two or more second groups.
[0519] The generation unit 132E, based on the object information acquired by the acquisition unit 131, executes either a first classification process, which classifies multiple objects into multiple first groups, or a second classification process, which classifies multiple objects into multiple second groups, which are groups that are different from the multiple first groups. After executing one classification process, the other classification process is executed using the result of the first classification process, thereby generating first group information indicating multiple first groups that classify multiple objects, and second group information indicating multiple second groups that classify multiple objects.
[0520] The generation unit 132E generates first group information, which is a group of objects that belong to the same second group in multiple second groups, and classifies objects belonging to the same second group in the same first group. The generation unit 132E generates first group information that shows multiple first groups in which objects belonging to at least one second group are classified into other first groups different from the first group to which one object belongs.
[0521] The generation unit 132E generates first group information and second group information by performing the first classification process, which is the other classification process, after the second classification process, which is the other classification process. The generation unit 132E generates second group information indicating multiple second groups by the second classification process, which clusters multiple objects. The generation unit 132E generates second group information by the second classification process using index information.
[0522] The generation unit 132E generates first group information through a first classification process using the second group information generated by the second classification process. The generation unit 132E generates first group information indicating multiple first groups through a first classification process using the clustering results obtained by clustering multiple objects.
[0523] The generation unit 132E generates first group information by performing a first classification process using a second centroid corresponding to each of the multiple second groups. The generation unit 132E generates first group information by performing a first classification process that assigns each of the second groups to a candidate first group, which is the first group with the largest number of objects to which it belongs.
[0524] The generation unit 132E generates first group information by performing a first classification process using partial centroids corresponding to subgroups obtained by dividing each of the second group into multiple subgroups. The generation unit 132E generates first group information by performing a first classification process that assigns each of the second group to a candidate first group, which is the first group to which the partial centroid belongs has the largest number.
[0525] If there are multiple candidate first groups in one second group, the generation unit 132E generates first group information by performing a first classification process in which, among the multiple candidate first groups, the first second group is assigned to the first group whose centroid is closest to the second centroid corresponding to the first second group.
[0526] (Search processing unit 133E) The search processing unit 133E executes the search process shown in Figure 48. For example, the search processing unit 133E performs various search-related processes in the same way as the search processing unit 133, search processing unit 133A, search processing unit 133B, search processing unit 133C, or search processing unit 133D. The search processing unit 133E performs processing using the information stored in the storage unit 120C. The search processing unit 133E performs processing using the information generated by the generation unit 132E.
[0527] For example, the search processing unit 133E performs a search process to extract neighboring objects corresponding to the search query from multiple objects by exploring a group linkage graph using a first search range that indicates a range corresponding to the search of multiple groups and a second search range that indicates a range corresponding to the search of multiple objects. The search processing unit 133E performs the search process using the first search range and the second search range which has a wider range than the first search range. Note that the search process performed by the search processing unit 133E is the same as that of the search processing unit 133D, except that it targets information in which the second number of the second group is greater than the first number of the first group, so a detailed explanation is omitted.
[0528] [6-3. Examples of inverted indexes] From here, an example of an inverted index in the sixth embodiment will be described using Figure 46. Figure 46 is a diagram showing an example of an inverted index according to the sixth embodiment. Note that explanations of the same points as those for the inverted index shown in Figures 46 and 36 will be omitted as appropriate.
[0529] First, let's explain the inverted index IND3 shown in Figure 46. The inverted index IND3 shown in Figure 46 includes items such as "Blob ID," "Number of Clusters," "Cluster #1," and "Cluster #2." Although Figure 46 shows two clusters, "Cluster #1" and "Cluster #2," the inverted index IND3 includes a number of items corresponding to the number of clusters (second group) belonging to each blob (first group). Hereafter, "Cluster #1," "Cluster #2," etc., may be referred to collectively as "cluster."
[0530] The "clusters" such as "Cluster #1" and "Cluster #2" include items such as "Cluster ID," "Number of Quantization Objects," and "Quantization Object." In Figure 46, information about multiple quantization objects is stored in the "Quantization Object" field in the form of an array (list), but individual items such as "Quantization Object #1," "Quantization Object #2," "Quantization Object #3," and "Quantization Object #4" as shown in Figure 35 may also be included.
[0531] "Blob ID" indicates identification information for identifying the blob in the first group. "Number of Clusters" indicates the number of clusters belonging to the corresponding blob. "Cluster" indicates information about the clusters belonging to the corresponding blob.
[0532] The "Cluster ID" indicates identification information for identifying the cluster, which is the second group. The "Number of Quantized Objects" indicates the number of objects belonging to the cluster. The "Quantized Object" indicates the quantized object. For example, the "Quantized Object" stores a list of information indicating the codebooks corresponding to each subvector after the object has been quantized as a Cartesian product.
[0533] In the example in Figure 46, it is shown that the blob with blob ID "1" belongs to four clusters. This indicates that the blob with blob ID "1" belongs to the cluster with cluster ID "2", the cluster with cluster ID "5", and so on.
[0534] It also indicates that there are four objects belonging to the cluster with cluster ID "2". Information indicating the objects belonging to the cluster with cluster ID "2" is associated with cluster ID "2". For example, information showing a list of objects belonging to the cluster with cluster ID "2" arranged in order is associated with cluster ID "2".
[0535] Furthermore, for the cluster with cluster ID "2", the "quantization object" stores "4, 2, 8, 10", indicating that the object is composed of four subvectors, each quantized using codebooks "4", "2", "8", and "10". For example, for the first object belonging to the cluster with cluster ID "2", it is shown that of the four divided subvectors, the first subvector is quantized using codebook "4", the second subvector is quantized using codebook "2", the third subvector is quantized using codebook "8", and the last subvector is quantized using codebook "10".
[0536] Furthermore, for the cluster with cluster ID "2", the "quantization object" stores "5, 6, 2, 0", indicating that the object is composed of four subvectors, each quantized using codebook values "5", "6", "2", and "0". For example, for the second object belonging to the cluster with cluster ID "2", it is shown that of the four divided subvectors, the first subvector is quantized using codebook value "5", the second subvector is quantized using codebook value "6", the third subvector is quantized using codebook value "2", and the last subvector is quantized using codebook value "0".
[0537] Furthermore, the inverted index IND3 may store various types of information depending on the purpose, not limited to those mentioned above. For example, the inverted index IND3 may store information indicating each quantization object (such as an object ID) associated with each quantization object.
[0538] As described above, the inverted index is assigned information that identifies the cluster (cluster ID), and the information processing device 100E calculates the residual vector using the inverted index. For example, the residual vector is used when generating and searching the index, but since this is the same as the processing in the prior art described in Patent Document 3, a detailed explanation will be omitted as appropriate.
[0539] [6-4. Information Processing Flow] Next, the information processing procedure according to the sixth embodiment will be described using Figure 47. Figure 47 is a flowchart showing an example of information processing according to the sixth embodiment.
[0540] First, let's explain Figure 47. For example, Figure 47 is a flowchart of an example of a group generation process. As shown in Figure 47, the information processing device 100E acquires object information indicating multiple objects that are the target of data retrieval (step S1001). For example, the information processing device 100E acquires object information indicating multiple objects from the object information storage unit 121.
[0541] The information processing device 100E generates first group information that indicates a plurality of first groups, which are a first number of groups that classify a plurality of objects, based on the object information (step S1002). For example, the information processing device 100E generates blob information that indicates a plurality of blobs that classify a plurality of objects, based on the object information.
[0542] The information processing device 100E generates second group information indicating multiple second groups, which are groups that have a different classification from multiple first groups and are groups with a second number greater than the first number used to classify multiple objects (step S1003). For example, the information processing device 100E generates second groups with a second number greater than the first number of first groups.
[0543] [6-5. Search Processing Example] Here, an example of the search process according to the sixth embodiment will be explained using Figure 48 as an example. Figure 48 is a flowchart of an example of the search process according to the sixth embodiment. The search process described below is performed by the search processing unit 133E of the information processing device 100E. Each of the steps S1101 to S1128 in Figure 48 corresponds to each of the steps S901 to S928 in Figure 43. Therefore, in each of the steps S1101 to S1128, explanations of the same points as in the steps S901 to S928 will be omitted as appropriate. For example, the process in Figure 48 and the process in Figure 43 are the same except that step S1121 is a part of the process in step S921.
[0544] The information processing device 100E obtains the search blob origin set and calculates the distance between each object and the query (step S1101). The information processing device 100E sets the nearest neighbor blob of the query in the search blob origin set as the current nearest neighbor blob B (blob B) (step S1102). The information processing device 100E sets various parameters (step S1103).
[0545] The information processing device 100E adds the set of search blob origins other than blob B to the set of excluded blobs (step S1104). The information processing device 100E adds the set of search blob origins to the set of unexplored blobs and the set of distances already calculated (step S1105).
[0546] The information processing device 100E determines whether the unexplored blob set is empty (step S1106). If the unexplored blob set is not empty (step S1106: No), the information processing device 100E retrieves the blob T that is closest to the query from the unexplored blob set (step S1107).
[0547] The information processing device 100E determines whether blob T is within the blob search range (step S1108). If blob T is within the blob search range (step S1108: Yes), the information processing device 100E determines whether all blobs connected to blob T by edges have been processed (step S1109). For example, the information processing device 100E uses information indicating whether or not blobs connected to blob T by edges (connected blobs) have been processed (processing status information) to determine whether or not all connected blobs of blob T have been processed. For example, the information processing device 100E may manage the processing status of connected blobs of blob T using information such as flags. For example, the information processing device 100E may update the processing status information stored in the storage unit 120C according to the processing status, and refer to the processing status information in the storage unit 120C to determine whether or not all connected blobs of blob T have been processed.
[0548] If the information processing device 100E has not processed all the blobs connected to the edge of blob T (step S1109: No), it retrieves blob J that is connected to blob T by an edge (step S1110).
[0549] The information processing device 100E determines whether blob J exists in the set of distance calculations (step S1111). If blob J exists in the set of distance calculations (step S1111: Yes), the information processing device 100E returns to step S1109 and processes the data.
[0550] Furthermore, if blob J does not exist in the set of distance-calculated blobs (step S1111: No), the information processing device 100E adds blob J to the set of distance-calculated blobs (step S1112) and adds blob J to the set of unexplored blobs (step S1113). Then, the information processing device 100E calculates the distance De between the centroid of blob J and the query (step S1114).
[0551] The information processing device 100E determines whether the distance De is less than the blob search radius Rb (step S1115). If the distance De is not less than the blob search radius Rb (step S1115: No), the information processing device 100E adds blob J to the set of excluded blobs (step S1116) and returns to step S1109 to process.
[0552] Furthermore, if the distance De is smaller than the blob search radius Rb (step S1115: Yes), the information processing device 100E adds the current nearest neighbor blob B to the set of excluded blobs, and sets blob J as the current nearest neighbor blob B (step S1117). In this way, the information processing device 100E updates the current nearest neighbor blob B to blob J when the distance De is smaller than the blob search radius Rb.
[0553] Then, the information processing device 100E updates the parameters related to the blob based on the updated nearest neighbor blob B in blob J (step S1118). In Figure 48, the information processing device 100E updates the blob search radius Rb to "d(q,B)" based on the updated nearest neighbor blob B in blob J. The information processing device 100E also updat...
Claims
1. An acquisition unit that acquires object information indicating multiple objects that are the target of data retrieval, Based on the object information acquired by the acquisition unit, a generation unit generates first group information indicating a plurality of first groups which are a first number of groups that classify the plurality of objects, and second group information indicating a plurality of second groups which are groups that have a different classification from the plurality of first groups and are a second number of groups which are greater than the first number that classify the plurality of objects. Equipped with, The generating unit is By clustering the plurality of objects, one group information is generated that indicates one of the plurality of first groups and the plurality of second groups; after generating the one group information, the one group indicated by the one group information is clustered to generate the other group information that indicates the other group from the plurality of first groups and the plurality of second groups that is different from the one group; thereby generating the first group information that indicates the plurality of first groups used for batch processing in a search process targeting the plurality of objects; and generating the second group information that indicates the plurality of second groups used for quantization processing targeting the plurality of objects; thereby generating the first group information and the second group information in such a way that the group to be processed in the search process and the group to be quantized are different. An information processing device characterized by the following:
2. The generating unit is The first group information is generated, which is a group that classifies objects belonging to the same second group in the plurality of second groups into the same first group. The information processing apparatus according to feature 1.
3. The generating unit is The process generates the first group information indicating the plurality of first groups, in which the object groups belonging to each of two or more of the plurality of second groups are classified into one first group. The information processing apparatus according to feature 1.
4. The generating unit is To generate the first group information indicating the plurality of first groups used to calculate the distances between the plurality of objects in a single operation, the distances between the plurality of objects are to be calculated. The information processing apparatus according to feature 1.
5. The generating unit is This generates the first group information, which indicates the plurality of first groups, used to parallel process the calculation of the distance between the search query and the object used in the search process. The information processing apparatus according to feature 1.
6. The generating unit is The second group information is generated, which indicates the plurality of second groups that serve as the units of quantization. The information processing apparatus according to feature 1.
7. The generating unit is This generates the second group information, which indicates the plurality of second groups used to quantize the vectors corresponding to each of the plurality of objects. The information processing apparatus according to feature 1.
8. The generating unit is After classifying the plurality of second groups, classifying the plurality of first groups is performed. The information processing apparatus according to feature 1.
9. The generating unit is The second group information, which represents the plurality of second groups, is generated by clustering the plurality of objects. The information processing apparatus according to feature 8.
10. The acquisition unit is, Obtain index information indicating the index that will be used to search the aforementioned multiple objects, The generating unit is Using the aforementioned index information, the second group information indicating the plurality of second groups is generated. The information processing apparatus according to feature 8.
11. The generating unit is Using the second group information that represents the second group, the first group information that represents the plurality of first groups is generated. The information processing apparatus according to feature 8.
12. The generating unit is The first group information representing the plurality of first groups is generated by clustering the plurality of second groups. The information processing apparatus according to feature 8.
13. The generating unit is After classifying the plurality of first groups, classifying the plurality of second groups is performed. The information processing apparatus according to feature 8.
14. The generating unit is The first group information, which represents the plurality of first groups, is generated by clustering the plurality of objects. The information processing apparatus according to feature 13.
15. The generating unit is Using the first group information that represents the first group, the second group information that represents the plurality of second groups is generated. The information processing apparatus according to feature 13.
16. The generating unit is The second group information is generated by dividing at least one of the plurality of first groups and generating two or more second groups. The information processing apparatus according to feature 13.
17. The generating unit is The objects belonging to each first group are clustered, and each first group is divided into two or more second groups to generate the second group information indicating the plurality of second groups. The information processing apparatus according to feature 13.
18. A method of information processing performed by a computer, The acquisition process involves obtaining object information that indicates multiple objects that are the target of the data search, A generation step that generates, based on the object information obtained in the acquisition step, first group information indicating a plurality of first groups which are a first number of groups that classify the plurality of objects, and second group information indicating a plurality of second groups which are groups that are different from the plurality of first groups and are a second number of groups which are greater than the first number that classify the plurality of objects, Includes, The aforementioned generation step is By clustering the plurality of objects, one group information is generated that indicates one of the plurality of first groups and the plurality of second groups; after generating the one group information, the one group indicated by the one group information is clustered to generate the other group information that indicates the other group from the plurality of first groups and the plurality of second groups that is different from the one group; thereby generating the first group information that indicates the plurality of first groups used for batch processing in a search process targeting the plurality of objects; and generating the second group information that indicates the plurality of second groups used for quantization processing targeting the plurality of objects; thereby generating the first group information and the second group information in such a way that the group to be processed in the search process and the group to be quantized are different. An information processing method characterized by the following:
19. The procedure for obtaining object information that indicates multiple objects to be searched for, A generation procedure that generates, based on the object information obtained by the acquisition procedure, first group information indicating a plurality of first groups which are a first number of groups that classify the plurality of objects, and second group information indicating a plurality of second groups which are groups that have a different classification from the plurality of first groups and are a second number of groups which are greater than the first number that classify the plurality of objects, Have the computer run it, The aforementioned generation procedure is: By clustering the plurality of objects, one group information is generated that indicates one of the plurality of first groups and the plurality of second groups; after generating the one group information, the one group indicated by the one group information is clustered to generate the other group information that indicates the other group from the plurality of first groups and the plurality of second groups that is different from the one group; thereby generating the first group information that indicates the plurality of first groups used for batch processing in a search process targeting the plurality of objects; and generating the second group information that indicates the plurality of second groups used for quantization processing targeting the plurality of objects; thereby generating the first group information and the second group information in such a way that the group to be processed in the search process and the group to be quantized are different. An information processing program characterized by the following features.
Citation Information
Patent Citations
Conductor for connection of semiconductor device
JP1987093335A
Power source plug slip alarm
JP1988000982A
Ultrasonic transformer
JP1988011000A
Information processing apparatus, program, computer-readable recording medium, and information processing method
JP2017010575A
Generation device, generation method and generation program
JP2017219929A
