Information processing apparatus, information processing system, and information processing method

Abstract

To prevent deterioration in search accuracy and reduce latency.SOLUTION: An information processing apparatus according to an embodiment includes: a circuit that receives a query vector; and a first processor that is configured to determine a first number of selection posting lists based on a distance between the query vector received via the circuit and each of a plurality of representative vectors associated with each of a plurality of posting lists, receive a first number of pieces of selection data associated with each of the first number of selection posting lists from among a plurality of pieces of search data, decompress, if the first number of pieces of selection data include a compressed posting list, the compressed posting list, and output at least one vector among a plurality of vectors included in the first number of selection posting lists as an answer based on a distance between the query vector and each of the plurality of vectors included in the first number of selection posting lists corresponding to the first number of pieces of selection data.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The embodiments relate to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] The importance of unstructured data is increasing for the purpose of improving business efficiency using IT technology. The amount of unstructured data tends to be large compared to the amount of structured data. A technology called approximate nearest neighbor search is known as a technology for searching for desired data from such a large amount of unstructured data. In the approximate nearest neighbor search, data close to a query, which is input data, is searched from multiple search data stored in a memory based on vector information associated with each of the multiple search data. In the approximate nearest neighbor search, the exact neighbor point for the query is not obtained, but the distance between a query vector based on the query and each of the vector information of the multiple search data is calculated to search for vector information in the vicinity of the query vector. In the above-mentioned approximate nearest neighbor search, a technology has been proposed in which, due to the increase in the amount of search data and memory costs, part of the search data is executed in a state where it is stored in a storage device with a larger capacity than the memory. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2015 / 049732 Brochure [Patent Document 2] International Publication No. 2021 / 044519 Brochure [Non-patent literature]

[0004] [Non-Patent Document 1] Qi Chen, 7 others, “SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search”, [Online], December 8, 2021, NeurIPS2021, [Retrieved July 6, 2020], Internet<URL:https: / / www.microsoft.com / en-us / research / uploads / prod / 2021 / 11 / SPANN_finalversion1.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] An information processing device, an information processing system, and an information processing method are provided that suppress deterioration of search accuracy and reduce latency. [Means for solving the problem]

[0006] An information processing device of an embodiment is an information processing device configured to output an answer to a query vector using a storage device configured to store a plurality of search data, each of the plurality of search data being associated with a posting list each storing a plurality of vectors, and each of the plurality of search data being an uncompressed posting list or a compressed posting list, and the information processing device includes a circuit for receiving the query vector, and a first processor configured to: determine a first number of selected posting lists from the plurality of posting lists based on a distance between the query vector received via the circuit and each of a plurality of representative vectors associated with each of the plurality of posting lists, receive a first number of selection data among the plurality of search data associated with each of the first number of selected posting lists, and if the first number of selection data includes the compressed posting list, decompress the compressed posting list, and output at least one vector among the plurality of vectors included in the first number of selected posting lists as the answer based on a distance between the query vector and each of a plurality of vectors included in the first number of selected posting lists corresponding to the first number of selection data. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing an outline of an approximate nearest neighbor search assuming the use of a storage device. [Diagram 2] FIG. 1 is a block diagram showing an example of a configuration of an information processing system according to an embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device included in the information processing system according to the embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of the hardware configuration of a storage device included in the information processing system according to the embodiment. [Diagram 5] FIG. 1 is a block diagram illustrating an example of a functional configuration of an information processing system according to an embodiment. [Figure 6]FIG. 11 is a block diagram illustrating another example of the functional configuration of the information processing system according to the embodiment. [Figure 7] 1 is a flowchart for explaining an overall flow of an approximate nearest neighbor search in the information processing system according to the embodiment. [Figure 8] 11 is a flowchart showing an example of an index generation process in an approximate nearest neighbor search in the information processing system according to the embodiment. [Figure 9] 11 is a diagram for explaining an example of compression of a posting list in an index generation process in an approximate nearest neighbor search of the information processing system according to the embodiment. [Figure 10] 13 is a diagram for explaining another example of compression of a posting list in an index generation process in an approximate nearest neighbor search of the information processing system according to the embodiment. [Figure 11] 13 is a diagram for explaining yet another example of compression of a posting list in an index generation process in an approximate nearest neighbor search in the information processing system according to the embodiment. FIG. [Figure 12] FIG. 11 is a sequence diagram showing an example of a processing sequence of a search process in an approximate nearest neighbor search in the information processing system according to the embodiment. [Figure 13] 11 is a flowchart showing an example of a search process in an approximate nearest neighbor search of the information processing system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described with reference to the drawings. The drawings are schematic or conceptual. The same reference numerals are attached to components having substantially the same functions and configurations. Numbers and letters attached to the reference numerals are used to distinguish between similar elements referred to by the same reference numerals.

[0009] 1. Embodiment An information processing device, an information processing system, and an information processing method according to the embodiments are configured to be able to preferably execute an approximate nearest neighbor search premised on the use of a storage device.

[0010] 1.1 Overview of Approximate Nearest Neighbor Search Based on Use of Storage Device in the Embodiment First, an overview of an approximate nearest neighbor search premised on the use of a storage device in an embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an overview of an approximate nearest neighbor search premised on the use of a storage device.

[0011] Highly-efficient Billion-scale Approximate Nearest Neighbor Search (SPANN) has been proposed as a type of approximate nearest neighbor search that assumes the use of storage devices such as SSDs (Solid State Drives) and HDDs (Hard Disk Drives). Below, an overview of SPANN is provided as an example of approximate nearest neighbor search that assumes the use of storage devices.

[0012] The search data in this specification corresponds to data to be searched in an approximate nearest neighbor search executed based on a query. Data is, for example, classified into structured data, which is structured, and unstructured data, which is not structured. Structured data is data having the concept of columns and rows. Since structured data is structured, it is easy to search, aggregate, compare, etc., and is suitable for data analysis. Unstructured data is, for example, office document data such as e-mails, proposals, plans, quotations, purchase orders, and contracts, image data, video data, audio data, and sensor logs. Unstructured data has a wide range of uses. The search data in this specification is unstructured data. The unstructured data is, for example, associated with vector information that characterizes the unstructured data. Hereinafter, the vector information is also simply referred to as a vector. In the approximate nearest neighbor search, a correct vector is searched for using, for example, vector information stored in a storage device. The correct vector in the approximate nearest neighbor search is vector information in the vicinity of the query vector. The correct vector is searched for, for example, based on the distance between the query vector and the vector information.

[0013] The approximate nearest neighbor search includes, for example, an index generation process and a search process.

[0014] In the index generation process, a plurality of pieces of vector information VI are classified into a plurality of posting lists PL1 to PLn (n is an integer of 2 or more). Each of the plurality of pieces of vector information VI is associated with one of a plurality of pieces of unstructured data. In the following description, when the plurality of posting lists PL1 to PLn are not distinguished from one another, each of the plurality of posting lists PL1 to PLn is simply called a posting list PL. Each posting list PL stores a plurality of pieces of vector information VI. The number of pieces of vector information VI included in each posting list PL may be the same or different. In FIG. 1, as an example, a case is illustrated in which each of the posting lists PL1, PL2, ..., and PLn includes k pieces of vector information VI. The plurality of posting lists PL1 to PLn can be used to search for a correct vector in a search process.

[0015] The plurality of posting lists PL includes, for example, a plurality of representative vectors RV associated with each of the plurality of posting lists PL. The representative vector RV is, for example, a centroid vector of a plurality of vector information VI included in the posting list PL associated with the representative vector RV, or a vector information VI that is closest to the centroid vector among the vector information VI included in the posting list PL. The plurality of representative vectors RV form, for example, a graph or a tree structure (hereinafter, simply referred to as a graph). The graph functions, for example, as an index used in a search process.

[0016] The multiple posting lists PL1 to PLn will be described in more detail. The posting list PL1 has a representative vector RV1 and multiple pieces of vector information VI1. The representative vector RV1 is, for example, a center of gravity vector of the multiple pieces of vector information VI1. The posting list PL2 has a representative vector RV2 and multiple pieces of vector information VI2. The representative vector RV2 is, for example, a center of gravity vector of the multiple pieces of vector information VI2. The posting list PLn includes a representative vector RVn and multiple pieces of vector information VIn. The representative vector RVn is, for example, a center of gravity vector of the multiple pieces of vector information VIn.

[0017] Each posting list PL can be stored in the storage device as compressed data or uncompressed data. In the following, when the posting list PL and the compressed posting list cPL are not distinguished from each other, each of the posting list PL and the compressed posting list cPL is simply referred to as search data.

[0018] In the search process, when a query to be searched is input to an information processing device, a first stage search is executed using the above-mentioned graph functioning as an index. Hereinafter, the first stage search in the search process is referred to as a first search process. The first search process is a process of searching for a representative vector RV close to a query vector QV based on the input query from among a plurality of representative vectors RV. More specifically, when a query to be searched is input, for example, a distance D between a query vector QV based on the input query and each representative vector RV is calculated. In FIG. 1, distances D(1), D(2), ..., and D(n) between the query vector QV and each of representative vectors RV1, RV2, and RVn are illustrated. Then, for example, j representative vectors RV (j is an integer equal to or greater than 1) are selected from the plurality of representative vectors RV in order of closeness to the query vector QV.

[0019] Then, search data for j posting lists PL corresponding to the selected j representative vectors RV is transferred from the storage device to the memory of the information processing device. That is, based on the result of the first search process, j search data required for the search process is transferred from the storage device to the memory. Each of the j posting lists PL is also called a selected posting list PL. Also, the search data for the selected posting list PL is also called selected search data.

[0020] Then, based on the j selected search data transferred to the memory, a second stage search is performed on the j selected posting lists PL. Hereinafter, the second stage search in the search process is referred to as a second search process. The second search process is a process of searching for vector information VI close to the query vector QV from among the vector information VI in the j selected posting lists PL. More specifically, based on the j selected search data, a calculation is performed of a distance D between the query vector QV and the multiple vector information VI included in each of the j selected posting lists PL. Then, for example, the vector information VI closest to the query vector QV is selected as a neighboring vector from among the multiple vector information VI in the j selected posting lists PL.

[0021] 1.2 Configuration The details of the information processing system according to the embodiment will be described below with reference to FIG. 2. FIG. 2 is a block diagram showing an example of the configuration of the information processing system according to the embodiment. As shown in FIG. 2, the information processing system 1 includes, for example, information processing devices 10A and 10B, and a storage device 20. In the following description, when the information processing devices 10A and 10B are not distinguished from each other, the information processing devices 10A and 10B are simply called the information processing device 10. In addition, in the information processing system 1, the information processing devices 10A and 10B and the storage device 20 are configured as different devices, but this is not limited to this. Two or all of the configurations of the information processing devices 10A and 10B, and the storage device 20 may be configured as one device. In this case, the one device can realize the functions of the information processing devices 10A and 10B, and the storage device 20, which will be described later.

[0022] Each information processing device 10 is, for example, a computer. Each information processing device 10 may be, for example, a server connected to a network.

[0023] The information processing device 10A is configured to execute an index generation process based on, for example, a vector data set VD including a plurality of pieces of vector information VI. More specifically, in the index generation process, the information processing device 10A classifies a plurality of pieces of vector information VI included in the vector data set VD into a plurality of posting lists PL. In addition, the information processing device 10A receives, for example, a compression algorithm from the outside. Then, the information processing device 10A compresses data for each posting list PL based on the received compression algorithm. Then, the information processing device 10A transmits the compressed posting list cPL (search data) to the storage device 20.

[0024] In addition, in FIG. 2, the information processing device 10A receives a compression algorithm from the outside, but is not limited to this. The information processing device 10A does not need to receive a compression algorithm from the outside. In this case, the information processing device 10A is configured to set each posting list PL as a target posting list PL that is a target of compression, or a non-target posting list that is not a target of compression. In addition, the information processing device 10A is configured to determine a compression algorithm to be used for compressing each target posting list PL, for example. The information processing device 10A compresses the target posting list PL using the compression algorithm determined for each target posting list PL. The non-target posting list PL is not compressed. Then, the information processing device 10A transmits the compressed target posting list cPL and the non-target posting list PL (search data) to the storage device 20.

[0025] The information processing device 10B is configured to execute a search process based on a query vector QV received from outside. The information processing device 10B may be configured to generate the query vector QV based on a query received from outside. The information processing device 10B executes a first search process, a transfer request for search data to the storage device 20, and a second search process using the transferred search data. Then, the information processing device 10B outputs vector information VI determined by the search process to the outside as an answer. In this way, the information processing device 10B uses information stored in the storage device 20 when searching for an answer to the query vector QV.

[0026] The storage device 20 is a storage medium capable of storing data in a non-volatile manner. The storage device 20 is, for example, an HDD or an SSD. The storage device 20 may be a single NAND type flash memory instead of an HDD or an SSD. The storage device 20 may store a plurality of pieces of search data. The storage device 20 may be directly or indirectly connected to each information processing device 10. Communication between each information processing device 10 and the storage device 20 may be either wired communication or wireless communication, or may be via a network. The information processing system 1 may include a plurality of storage devices 20.

[0027] 1.2.1 Hardware Configuration The hardware configurations of the information processing devices 10A and 10B and the storage device 20 included in the information processing system 1 according to the embodiment will be described.

[0028] 1.2.1.1 Hardware configuration of information processing device The hardware configuration of the information processing devices 10A and 10B included in the information processing system 1 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the hardware configuration of the information processing device provided in the information processing system according to the embodiment. In the following, the hardware configuration that may be similar in the information processing devices 10A and 10B, as well as different hardware configurations, will be described.

[0029] The information processing device 10 includes, for example, a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output circuit 14, and a communication interface (communication I / F) 15. The CPU 11, the ROM 12, the RAM 13, the input / output circuit 14, and the communication interface 15 are connected via a bus.

[0030] The CPU 11 is a processor configured to execute various programs related to the control of the information processing device 10. A plurality of CPUs 11 may be provided according to the programs to be executed.

[0031] The ROM 12 is a non-volatile storage device configured to store a control program for the information processing device 10 .

[0032] The RAM 13 is a volatile storage device used as a working area for the CPU 11. The RAM 13 may be a non-volatile storage device as long as it is used as a working area for the CPU 11. The RAM 13 is, for example, an SRAM or a DRAM. When the information processing device 10A executes the index generation process, the data stored (expanded) in the RAM 13 is used. When the information processing device 10B executes each of the first search process and the second search process, the data stored (expanded) in the RAM 13 is used.

[0033] The input / output circuit 14 is a circuit that handles signals input to the information processing device 10 and signals output from the information processing device 10. For example, a vector data set VD and a compression algorithm are input to the input / output circuit 14 in the information processing device 10A from the outside. For example, a query vector QV is input to the input / output circuit 14 in the information processing device 10B. Then, a response to the query vector QV is output from the input / output circuit 14 in the information processing device 10B. The input / output circuit 14 in the information processing device 10B may have an encoder that generates the query vector QV based on a query received from the outside.

[0034] The communication interface 15 is an interface circuit used for communication with the storage device 20. For example, search data is output from the communication interface 15 in the information processing device 10A. For example, search data selected by the information processing device 10B is input to the communication interface 15 in the information processing device 10B.

[0035] In addition, when the hardware configurations included in information processing devices 10A and 10B are included in a single device, the CPU, ROM, RAM, input / output circuitry, and communication interface in that single device can execute the processing executed by the CPU 11, ROM 12, RAM 13, input / output circuitry 14, and communication interface 15 in information processing devices 10A and 10B, respectively.

[0036] 1.2.1.2 Storage device hardware configuration The hardware configuration of the storage device 20 included in the information processing system 1 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the hardware configuration of the storage device provided in the information processing system according to the embodiment.

[0037] The storage device 20 includes a controller 30 and a non-volatile memory device 40 .

[0038] The controller 30 is configured to be able to control the nonvolatile storage device 40 in response to a request from the information processing device 10. The controller 30 includes, for example, a CPU 31, a ROM 32, a RAM 33, a communication interface (communication I / F) 34, and a storage interface (storage I / F) 35. The CPU 31, the ROM 32, the RAM 33, the communication interface 34, and the storage interface 35 are connected via a bus.

[0039] The CPU 31 is a processor configured to execute various programs related to the control of the controller 30 .

[0040] The ROM 32 is a non-volatile storage device configured to store a control program for the controller 30 .

[0041] The RAM 33 is a volatile storage device used as a work area for the CPU 31. The RAM 33 may be a non-volatile storage device as long as it is used as a work area for the CPU 31.

[0042] The communication interface 34 is an interface circuit used for communication with the information processing device 10 .

[0043] The storage interface 35 is an interface circuit used for communication with the non-volatile storage device 40 .

[0044] The nonvolatile storage device 40 stores data in a nonvolatile manner.

[0045] When the information processing device 10 and the storage device 20 are configured as one device, the one device includes, for example, a storage interface and a non-volatile storage device in addition to the configuration of the above-mentioned information processing device 10. Also, for example, the CPU 11, ROM 12, RAM 13, and communication interface 15 of the above-mentioned information processing device 10 can execute the processes executed by the above-mentioned CPU 31, ROM 32, RAM 33, and communication interface 34, respectively.

[0046] 1.2.2 Functional Configuration 1.2.2.1 Functional configuration of the information processing device 10A The functional configuration of the information processing device 10A will be described with reference to Fig. 5. Fig. 5 is a block diagram for explaining an example of the functional configuration of the information processing system according to the embodiment. Fig. 5 shows the functional configuration of the information processing device 10A as well as the functional configuration of the storage device 20.

[0047] The information processing device 10A includes a representative vector generation unit 101, a search graph generation unit 102, a search graph transmission unit 103, a posting list generation unit 104, a compression target determination unit 105, a posting list compression unit 106, and a posting list transmission unit 107. The storage device 20 also includes a storage unit 200 that stores various data.

[0048] The representative vector generation unit 101 receives a vector data set VD from, for example, an external device. The representative vector generation unit 101 executes a clustering process on the received vector data set VD. As a result, the representative vector generation unit 101 generates a plurality of representative vectors RV. The representative vector generation unit 101 transmits the generated plurality of representative vectors RV to the search graph generation unit 102. In addition, the representative vector generation unit 101 transmits the plurality of representative vectors RV and the vector data set VD to the posting list generation unit 104.

[0049] The search graph generation unit 102 generates a search graph GPH based on the multiple representative vectors RV received from the representative vector generation unit 101. The search graph generation unit 102 transmits the search graph GPH to the search graph transmission unit 103. The search graph GPH configures a data structure in which multiple representative vectors RV are stored. The search graph GPH is used by the information processing device 10B during the first search process.

[0050] The search graph transmission unit 103 transmits the search graph GPH to the storage unit 200 .

[0051] The posting list generating unit 104 generates a plurality of posting lists PL based on the plurality of representative vectors RV and the vector data set VD received from the representative vector generating unit 101. That is, the posting list generating unit 104 classifies a plurality of pieces of vector information VI included in the vector data set VD into a plurality of posting lists PL using the plurality of representative vectors RV. The posting list generating unit 104 transmits the generated plurality of posting lists PL to the compression target determining unit 105.

[0052] The compression target determination unit 105 receives a plurality of posting lists PL from the posting list generation unit 104. The compression target determination unit 105 may also receive a compression algorithm from outside. When receiving a compression algorithm from outside, the compression target determination unit 105 sets all posting lists PL as target posting lists PL for compression. The compression target determination unit 105 also transmits the received compression algorithm and a plurality of posting lists PL to the posting list compression unit 106. When not receiving a compression algorithm from outside, the compression target determination unit 105 determines whether or not to compress data for each of the plurality of posting lists PL. The compression target determination unit 105 sets the posting list PL for which it has determined that data compression is to be performed as target posting lists PL for compression, and sets the posting list PL for which it has determined that data compression is not to be performed as non-target posting lists PL for compression. The compression target determination unit 105 also determines a compression algorithm corresponding to each of the target posting lists PL. Then, the compression target determination unit 105 transmits the target posting list PL and the compression algorithm determined for the target posting list PL to the posting list compression unit 106. In addition, the compression target determination unit 105 transmits the non-target posting list PL to the posting list transmission unit 107.

[0053] The compression algorithm received from outside and the compression algorithm determined by the compression target determination unit 105 are, for example, compression algorithms using run-length compression (RLE: Run Length Encoding), variable length coding, delta coding, etc. The compression algorithm may also be, for example, a compression algorithm using a combination of two or more of the methods of run-length compression, variable length coding, delta coding, etc. Run-length compression, variable length coding, and delta coding will be described later.

[0054] The posting list compression unit 106 compresses the posting list PL based on the target posting list PL received from the compression target determination unit 105 and the compression algorithm.

[0055] More specifically, when the compression target determination unit 105 receives a compression algorithm from the outside, the posting list compression unit 106 receives the compression algorithm and a plurality of posting lists PL from the compression target determination unit 105. The posting list compression unit 106 compresses each of all the posting lists PL using the compression algorithm. In addition, the posting list compression unit 106 transmits a plurality of compressed posting lists cPL and the compression algorithm to the posting list transmission unit 107.

[0056] When the compression target determination unit 105 does not receive a compression algorithm from the outside, the posting list compression unit 106 receives the target posting list PL and the compression algorithm determined for the target posting list PL from the compression target determination unit 105. The posting list compression unit 106 compresses the target posting list PL based on the compression algorithm determined for each target posting list PL. In addition, the posting list compression unit 106 transmits the compressed target posting list cPL and the compression algorithm used to generate the compressed target posting list cPL to the posting list transmission unit 107.

[0057] FIG. 5 shows an example in which all posting lists PL1 to PLn are compressed based on a compression algorithm received from outside.

[0058] The posting list transmitting unit 107 transmits a plurality of pieces of search data to the storage unit 200. In addition, the posting list transmitting unit 107 can transmit a compression algorithm to the storage unit 200.

[0059] More specifically, when the compression target determination unit 105 receives a compression algorithm from outside, the posting list transmission unit 107 transmits all compressed posting lists cPL (search data) received from the posting list compression unit 106 to the storage unit 200. In addition, the posting list transmission unit 107 transmits the compression algorithm received from outside to the storage unit 200 as the compression algorithm used to compress all posting lists PL.

[0060] Furthermore, when the compression target determination unit 105 does not receive a compression algorithm from the outside, the posting list transmission unit 107 may transmit the compressed target posting list cPL (search data) received from the posting list compression unit 106 and the non-target posting list PL (search data) received from the compression target determination unit 105 to the storage unit 200. Furthermore, the posting list transmission unit 107 may transmit, for example, the compression algorithm determined by the compression target determination unit 105 to the storage unit 200 in association with the compressed target posting list cPL.

[0061] The compression algorithm transmitted to the storage unit 200 as described above is stored in, for example, the compression algorithm information ALG in the storage unit 200. In the compression algorithm information ALG, for example, the compression algorithm used in generating each compressed posting list cPL is stored in association with the compressed posting list cPL. Note that when the compression target determination unit 105 receives a compression algorithm from the outside, the compression algorithm information ALG may be configured to be able to confirm that the compression algorithm has been used in compressing all the posting lists PL.

[0062] 1.2.2.2 Functional configuration of the information processing device 10B The functional configuration of the information processing device 10B will be described with reference to Fig. 6. Fig. 6 is a block diagram for explaining another example of the functional configuration of the information processing system according to the embodiment. The information processing device 10B includes a representative vector management unit 111, a first distance calculation unit 112, a posting list determination unit 113, a posting list reception unit 114, a compressed information management unit 115, a data decompression unit 116, a second distance calculation unit 117, and a neighborhood vector determination unit 118.

[0063] The representative vector management unit 111 receives the search graph GPH in which multiple representative vectors RV are stored from the storage unit 200. In addition, the representative vector management unit 111 transmits the received search graph GPH to the first distance calculation unit 112. The representative vector management unit 111 may transmit to the first distance calculation unit 112 information related to multiple representative vectors RV based on the search graph GPH, rather than transmitting the received search graph GPH as is.

[0064] The first distance calculation unit 112 receives a query vector QV from an external source. Having received the query vector QV, the first distance calculation unit 112 acquires a search graph GPH including a plurality of representative vectors RV of each of a plurality of posting lists PL from the representative vector management unit 111. Then, the first distance calculation unit 112 calculates the distance between the query vector QV and the plurality of representative vectors RV, and transmits the calculation result to the posting list determination unit 113. For example, the first distance calculation unit 112 traces the graph while calculating the distance between each node (each representative vector RV) of the search graph GPH and the query vector QV.

[0065] In the first search process, the posting list determination unit 113 selects j selected posting lists PL to be targeted in the second search process based on the calculation result of the distance between the query vector QV and the representative vector RV received from the first distance calculation unit 112. The number of selected posting lists PL may be predetermined or may be determined according to the characteristics of the query vector QV and the plurality of vector information VI. The posting list determination unit 113 transmits a transfer request of search data for the j selected selected posting lists PL (j selected search data) to the storage unit 200. The transfer request includes address information of the j selected search data. In addition, the posting list determination unit 113 transmits transfer information including a list of the j selected search data (or posting list PL) to the posting list receiving unit 114, for example.

[0066] The posting list receiving unit 114 receives the j selected pieces of search data from the storage unit 200 based on the transfer information received from the posting list determining unit 113. Then, the posting list receiving unit 114 transmits the received search data to the data decompressing unit 116.

[0067] In response to a transfer request from the posting list determination unit 113, the storage unit 200 transmits the j selected pieces of search data to the posting list receiving unit 114. More specifically, in the storage device 20, the controller 30 transmits a read command for the search data stored at the address based on the transfer request to the nonvolatile storage device 40. The nonvolatile storage device 40 then reads the search data from the specified address and transmits the read search data to the controller 30. Then, the controller 30 transmits the search data received from the nonvolatile storage device 40 to the posting list receiving unit 114.

[0068] When the selected search data is a compressed posting list cPL, the compression information management unit 115 receives the compression algorithm used in generating the compressed posting list cPL from the storage unit 200 based on the compression algorithm information ALG. Then, the compression information management unit 115 transmits the received compression algorithm to the data decompression unit 116. That is, when the selected search data is a compressed posting list cPL, the storage unit 200 transmits the compression algorithm used in generating the compressed posting list cPL to the compression information management unit 115 based on the compression algorithm information ALG.

[0069] The data decompression unit 116 determines whether each of the j selected pieces of search data is a posting list PL or a compressed posting list cPL. If the selected search data is a posting list PL, the data decompression unit 116 transmits the posting list PL to the second distance calculation unit 117. If the selected search data is a compressed posting list cPL, the data decompression unit 116 decompresses the compressed posting list cPL based on the compression algorithm used to generate the compressed posting list cPL received from the compression information management unit 115. Then, the data decompression unit 116 transmits the decompressed posting list PL to the second distance calculation unit 117. In this manner, the data decompression unit 116 transmits the j selected posting lists PL to the second distance calculation unit 117.

[0070] The second distance calculation unit 117 receives the query vector QV, similarly to the first distance calculation unit 112. The second distance calculation unit 117 calculates the distance between the query vector QV and each piece of vector information VIN in the j selected posting lists PL received from the data decompression unit 116. The second distance calculation unit 117 transmits the result of the calculation to the neighborhood vector determination unit 118.

[0071] The neighborhood vector determination unit 118 determines at least one neighborhood vector NV based on the result of the calculation received from the second distance calculation unit 117. Then, the neighborhood vector determination unit 118 outputs the determined neighborhood vector NV to the outside. Note that the number of neighborhood vectors NV determined and output by the neighborhood vector determination unit 118 may be changed according to the setting of the approximate nearest neighbor search, and may be one or two or more.

[0072] 1.3 Operation The approximate nearest neighbor search in the information processing system 1 according to the embodiment will be described.

[0073] The overall flow of the approximate nearest neighbor search in the information processing system 1 according to the embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart for explaining the overall flow of the approximate nearest neighbor search in the information processing system according to the embodiment.

[0074] In the approximate nearest neighbor search in the information processing system 1 according to the embodiment, first, an index generation process is executed (Sidx).

[0075] Then, the search process is performed (Ssrc).

[0076] The index generation process and the search process will be described below in turn.

[0077] 1.3.1 Index generation process The index generation process in the information processing system 1 according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the index generation process in the approximate nearest neighbor search in the information processing system according to the embodiment.

[0078] When the index generation process starts, in (S0), the information processing device 10A initializes a value m1 (m1=1).

[0079] Next, the representative vector generation unit 101 receives the vector data set VD (S1).

[0080] Then, the representative vector generation unit 101 generates a plurality of representative vectors RV using the vector data set VD (S2).

[0081] Then, the posting list generating unit 104 generates a plurality of posting lists PL using the vector data set VD and a plurality of representative vectors RV (S3). That is, the information processing device 10A classifies a plurality of vector information VI included in the vector data set VD into a plurality of posting lists PL. Also, in (S3), the search graph generating unit 102 generates a search graph GPH using, for example, a plurality of representative vectors RV.

[0082] Next, the compression target determination unit 105 determines whether or not a compression algorithm has been input from the outside (S4). If a compression algorithm has been input from the outside (S4; YES), the process proceeds to (S10). If a compression algorithm has not been input from the outside (S4; NO), the process proceeds to (S5).

[0083] When no compression algorithm is input from the outside (S4; NO), the compression target determination unit 105 judges whether or not to compress the m1-th posting list PLm1 among the multiple posting lists PL (S5). More specifically, the compression target determination unit 105 judges, for example, whether or not there is a valid compression algorithm for the posting list PLm1. When there is a compression algorithm judged to be valid, the compression target determination unit 105 sets the posting list PLm1 to a posting list PL to be compressed. When there is no compression algorithm judged to be valid, the compression target determination unit 105 sets the posting list PLm1 to a posting list PL not to be compressed. When the posting list PLm1 is set to a posting list PL to be compressed (S5; YES), the process proceeds to (S6). When the posting list PLm1 is set to a posting list PL not to be compressed (S5; NO), the compression target determination unit 105 transmits the posting list PLm1 to the posting list transmission unit 107 as search data. Then, the process proceeds to (S8).

[0084] When the posting list PLm1 is set as the target posting list PL (S5; YES), the compression target determination unit 105 determines the compression algorithm to be used for compressing the posting list PLm1 based on the compression algorithm determined to be valid (S6). An example of determining that a compression algorithm is valid will be described below.

[0085] As examples of compression algorithms, cases where run-length compression, variable-length coding, and delta coding are used will be described with reference to Figs. 9, 10, and 11, respectively. Figs. 9, 10, and 11 are diagrams for explaining an example of compression of a posting list in an index generation process in an approximate nearest neighbor search of an information processing system according to an embodiment. In Figs. 9, 10, and 11, for the posting list PLm1, each component included in a vector is shown to be included in the same row.

[0086] In SPANN, since multiple posting lists PL are generated based on the clustering process as described above, for example, the correlation coefficient between different vector information VIm1 included in the posting list PLm1 becomes high. As a result, the same value may be consecutive in the same component in multiple vector information VIm1, such as the component surrounded by the dotted line in Fig. 9. Fig. 9 shows an example of a representative vector RVm1 and multiple vector information VIm1. In such a case, the compression target determination unit 105 determines that run-length compression is effective, for example.

[0087] Also, for the same reason as the correlation coefficient between different vector information VIm1, the correlation coefficient between the vector information VIm1 and the representative vector RVm1 included in the posting list PLm1 may also be high. In such a case, the compression target determination unit 105 may determine that variable-length coding is effective, for example. The above will be supplemented with reference to FIG. 10. FIG. 10 illustrates an example in which the representative vector RVm1 is equivalent to the example in FIG. 9, and there are a plurality of pieces of vector information VIm1. When the correlation coefficient between the vector information VIm1 and the representative vector RVm1 is high, as illustrated in the area surrounded by the dotted line in FIG. 10, each component of the representative difference vector Drm1 (Drm1=VIm1-RVm1) obtained by subtracting the representative vector RVm1 from each of the plurality of pieces of vector information VIm1 indicates, for example, a value close to 0. In this way, when there is a bias in the frequency of occurrence of the values ​​of the components of the representative difference vector Drm1, the compression target determination unit 105 may determine that compression using variable-length coding is effective.

[0088] Furthermore, when the values ​​of the components of adjacent vector information VIm1 are close to each other, the compression target determination unit 105 determines that a compression algorithm using, for example, delta coding is effective. FIG. 11 (A) shows an example of a representative vector RVm1 and a plurality of pieces of vector information VIm1 different from the example of FIG. 9. FIG. 11 (B) shows an example of a representative vector RVm1 and a plurality of adjacent difference vectors Dam1 generated by delta coding. The adjacent difference vector Dam1 is, for example, a difference vector between two adjacent pieces of vector information VIm1 when a plurality of pieces of vector information VIm1(0) to VIm1(k-1) are arranged in a line in this order. More specifically, the adjacent difference vector Dam1(p) (p is a natural number equal to or less than (k-1)) is a vector obtained by subtracting vector information VIm1(p-1) from vector information VIm1(p). Furthermore, the adjacent difference vector Dam1(0) is, for example, a vector obtained by subtracting the representative vector RVm1 from vector information VIm1(0). As described above, when the values ​​of the components of adjacent vector information VIm1 are close to each other, the values ​​of the adjacent difference vectors Dam1 for each component may form a mountain-shaped distribution centered on 0. This allows high compression efficiency to be achieved by applying a compression method using a variable-length code such as a Huffman code to the adjacent difference vectors Dam1. Therefore, the compression target determination unit 105 may determine that a compression algorithm using delta coding is effective.

[0089] As described above, the compression target determining unit 105 may apply a compression algorithm that combines two or more techniques of run-length compression, variable-length coding, and delta coding, for example, to compress the posting list PLm1.

[0090] Returning to Fig. 8, the index generation process will be continued. The posting list compression unit 106 compresses the posting list PLm1 using the compression algorithm determined for the posting list PLm1 by the compression target determination unit 105 (S7). The posting list compression unit 106 also transmits the compressed posting list cPLm1 to the posting list transmission unit 107 as search data. The posting list compression unit 106 also transmits the compression algorithm used to generate the compressed posting list cPLm1 to the posting list transmission unit 107. Then, the process proceeds to (S8).

[0091] In (S8), the information processing device 10A determines whether the value m1 is less than the value n. If the value m1 is less than the value n (S8; YES), the process proceeds to (S9). If the value m1 is equal to or greater than the value n (S8; NO), the process proceeds to (S11).

[0092] Then, in (S9), the information processing device 10A increments the value m1 (m1=m1+1), and the process proceeds to (S5).

[0093] When a compression algorithm is input from the outside (S4; YES), the posting list compression unit 106 sets all posting lists PL as target posting lists PL. Then, the posting list compression unit 106 compresses each of the posting lists PL1 to PLn using the compression algorithm input from the outside (S10). Furthermore, the posting list compression unit 106 transmits the compressed posting lists cPL for all posting lists PL to the posting list transmission unit 107 as search data. Furthermore, the posting list compression unit 106 transmits the compression algorithm input from the outside to the posting list transmission unit 107. Then, the process proceeds to (S11).

[0094] Then, the posting list transmission unit 107 transmits a plurality of pieces of search data relating to each of the posting lists PL1 to PLn to the storage unit 200 of the storage device 20 (S11). In addition, the posting list transmission unit 107 transmits to the storage unit 200 of the storage device 20 a compression algorithm received from outside or a compression algorithm determined by the compression target determination unit 105 and used to generate each of the compressed posting lists cPL.

[0095] In this manner, the index generation process is executed.

[0096] 1.3.2 Search process Next, a search process in the information processing system 1 according to the embodiment will be described.

[0097] 1.3.2.1 Overview An overview of the search process in the information processing system 1 according to the embodiment will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of a processing sequence of the search process in the approximate nearest neighbor search in the information processing system according to the embodiment.

[0098] When the information processing device 10B receives the query vector QV, it executes a first search process. Then, based on the result of the first search process, the information processing device 10B requests the storage device 20 to transfer search data for the j selected posting lists PL (transfer request).

[0099] Next, in the second search process, the storage device 20 sequentially transfers the j selected pieces of search data to the information processing device 10B based on the transfer request received from the information processing device 10B. Note that Fig. 12 shows an example in which the j selected pieces of search data are all stored in the storage device 20 as compressed posting lists cPL_1st to cPL_jst.

[0100] The order in which the storage device 20 transfers the search data is based on, for example, the contents of the transfer request. For example, the posting list determination unit 113 of the information processing device 10B transmits a transfer request for the search data to the storage device 20 in the order of the posting list PL having the shortest distance between the query vector QV and the representative vector RV. Then, in response to this transfer request, the storage device 20 transfers the search data to the posting list receiving unit 114 of the information processing device 10B in the order of the posting list PL having the shortest distance.

[0101] When the search data received from the storage device 20 is a compressed posting list cPL, the information processing device 10B executes a decompression process of the compressed posting list cPL. When the search data received from the storage device 20 is an uncompressed posting list PL, the information processing device 10B does not execute a decompression process. Then, the information processing device 10B calculates the distance between the multiple pieces of vector information VI included in each posting list PL and the query vector QV (executes a distance calculation process), for example, every time search data is transferred.

[0102] Although not shown in FIG. 12, in order to perform decompression processing of the compressed posting list cPL, the information processing device 10B receives the compression algorithm used to generate the compressed posting list cPL from the storage unit 200 based on the compression algorithm information ALG, for example, each time the compressed posting list cPL is transferred as search data.

[0103] Then, the information processing device 10B determines a neighborhood vector NV based on the result of the distance calculation process for each of the j posting lists PL. Furthermore, the information processing device 10B outputs the determined neighborhood vector NV as a response to the query.

[0104] 1.3.2.2 Flowchart A flowchart of the search process in the information processing system 1 according to the embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the search process in the approximate nearest neighbor search in the information processing system according to the embodiment.

[0105] When the search process starts, in (S20), the information processing device 10B initializes the value m2 (m2=1).

[0106] Next, the first distance calculation unit 112 receives the query vector QV (S21). Furthermore, the representative vector management unit 111 receives the search graph GPH including a plurality of representative vectors RV from the storage unit 200 of the storage device 20.

[0107] Then, the first distance calculation unit 112 executes a first search process (S22). As a result, the posting list determination unit 113 selects j representative vectors RV from among the multiple representative vectors RV, and j posting lists PL corresponding to the j representative vectors RV. More specifically, the first distance calculation unit 112 calculates the distance between the query vector QV and the representative vector RV, for example, while tracing the search graph GPH. Furthermore, the posting list determination unit 113 selects j representative vectors RV based on the result of the calculation.

[0108] Next, the posting list determination unit 113 transmits a transfer request for the search data to the storage device 20 (S23). This transfer request includes address information of the search data determined by the process of S22. Then, the controller 30 of the storage device 20 stores in a queue a read command targeted at the address information included in the transfer request. Then, the controller 30 reads the search data from the non-volatile storage device 40 in response to the read command stored in the queue.

[0109] Then, the posting list receiving unit 114 receives from the storage device 20 search data (m2-th search data) for the m2-th posting list PL among the j selected posting lists PL in order of the distance between the query vector QV and the representative vector RV (S24).

[0110] Then, the data decompression unit 116 determines whether the m2-th search data is compressed (S25).

[0111] If the m2th search data is a compressed posting list cPL (S25; YES), the compression information management unit 115 receives the compression algorithm used to generate the compressed posting list cPL from the storage unit 200 (S26).

[0112] Next, the data decompression unit 116 decompresses the m2-th search data (compressed posting list cPL) based on the compression algorithm received in the process of (S26) (S27).

[0113] After (S27), or if the m2-th search data is an uncompressed posting list PL (S25; NO), the second distance calculation unit 117 calculates the distance between the query vector QV and each of the multiple vector information VI included in the m2-th posting list PL (S28).

[0114] Then, the information processing device 10B determines whether the value m2 is less than the value j (S29).

[0115] If the value m2 is less than the value j (S29; YES), the information processing device 10B increments the value m2 (m2=m2+1) in (S30), and the process proceeds to (S24).

[0116] If the value m2 is equal to or greater than the value j (S29; NO), the neighborhood vector determination unit 118 determines at least one neighborhood vector NV based on the calculation results of the distances between the query vector QV for the j posting lists PL and each of the vector information VI (S31).Then, the neighborhood vector determination unit 118 outputs the determined neighborhood vector NV to the outside.

[0117] As described above, the information processing device 10B executes the search process.

[0118] 1.4 Effects of the embodiment According to the information processing system 1 according to the embodiment, it is possible to suppress a decrease in search accuracy and reduce latency. The effects of the embodiment will be described in detail below.

[0119] The approximate nearest neighbor search includes a first search process targeting the representative vector and a second search process executed using data transferred from the storage unit based on the result of the first search process. In such an approximate nearest neighbor search method, it is preferable to suppress an increase in latency due to the transfer of data from the storage unit while suppressing a decrease in search accuracy.

[0120] The information processing device 10A of the information processing system 1 according to the embodiment is configured to be capable of compressing the posting list PL to be transmitted to the storage unit 200. This makes it possible to reduce the amount of data stored in the storage unit 200 and to reduce the amount of search data transmitted from the storage unit 200 for the second search process.

[0121] Moreover, the information processing device 10B of the information processing system 1 according to the embodiment is configured to be able to decompress the compressed posting list cPL received from the storage unit 200 in the search process. As a result, even if the search data stored in the storage unit 200 is a compressed posting list cPL, the information processing device 10B can execute the second search process using the search data.

[0122] With the above-described configuration, according to the embodiment, it is possible to suppress a decrease in search accuracy and reduce latency.

[0123] To supplement, in the approximate nearest neighbor search, when all posting lists are stored in the storage unit in an uncompressed state, there is a trade-off between search accuracy and latency. More specifically, the search accuracy can be improved by increasing the number of selected posting lists in the second search process, but the latency increases due to an increase in the amount of data transferred from the storage unit to the information processing device. In addition, the amount of data transferred from the storage unit to the information processing device can be reduced by reducing the number of selected posting lists in the second search process, and the latency can be reduced, but the search accuracy deteriorates. For this reason, when all posting lists are stored in the storage unit in an uncompressed state, it was difficult to reduce the latency while suppressing a decrease in search accuracy.

[0124] According to the information processing system 1 according to the embodiment, the information processing device 10A is configured to be able to transmit the compressed posting list cPL to the storage unit 200, and the information processing device 10B is configured to be able to decompress the compressed posting list cPL transferred from the storage unit 200. Therefore, it is possible to reduce latency without reducing the number of posting lists PL used in the second search process.

[0125] 2. Other In the embodiment, the flowcharts used to explain the operations are merely examples. The order of the operations explained using the flowcharts may be changed to the extent possible, other processes may be added, or some processes may be omitted.

[0126] In the embodiment, the case where each posting list PL includes a representative vector RV is shown, but is not limited to this. For example, when the representative vector RV associated with each posting list PL is a centroid vector of a plurality of vector information VI included in the posting list PL, the representative vector RV may not be included in each posting list PL and may be stored in the search graph GPH. In this case, when compressing each posting list PL, the posting list compression unit 106 of the information processing device 10A does not compress, for example, the representative vector RV associated with the posting list PL. Then, the posting list transmission unit 107 of the information processing device 10A transmits the compressed posting list cPL, which does not include data related to the representative vector RV, to the storage unit 200.

[0127] In the embodiment, the hardware configurations of the information processing devices 10A and 10B and the controller 30 are not limited to those in the embodiment. For example, instead of the CPUs 11 and 31, an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (field-programmable gate array) may be used. The functions of the information processing devices 10A and 10B and the controller 30 may be partially or entirely realized by dedicated hardware, or may be configured as a program executed by a processor such as a CPU. That is, the functions of the information processing devices 10A and 10B and the controller 30 may be realized by a computer and a program. The program may be recorded in a storage medium, or the program may be provided via a network.

[0128] In addition, some or all of the above-described embodiments can be described as follows, but are not limited to the following.

[0129] <Appendix 1> An information processing device configured to output an answer to a query vector by utilizing a storage configured to store a plurality of search data, Each of the plurality of search data is associated with a posting list, each of the posting lists storing a plurality of vectors, and each of the plurality of search data is an uncompressed posting list or a compressed posting list; a circuit for receiving the query vector; determining a first number of selected posting lists from the plurality of posting lists based on a distance between the query vector received via the circuit and each of a plurality of representative vectors associated with each of the plurality of posting lists; receiving a first number of selected data items associated with each of the first number of selected posting lists from among the plurality of search data items; performing decompression of the compressed posting list if the first number of selection data includes the compressed posting list; a first processor configured to output at least one vector among the plurality of vectors included in the first number of selection posting lists corresponding to the first number of selection data, as the answer, based on a distance between the query vector and each of the plurality of vectors included in the first number of selection posting lists; Equipped with Information processing device.

[0130] <Appendix 2> The storage is configured to further store compression algorithm information related to the compressed posting list among the plurality of search data stored therein; the data decompression unit decompresses the compressed selection posting list based on the compression algorithm information; 2. The information processing device according to claim 1.

[0131] <Appendix 3> The first processor determines a plurality of selected posting lists from the plurality of posting lists based on a search graph storing the plurality of representative vectors. 2. The information processing device according to claim 1.

[0132] <Appendix 4> The representative vector is a vector based on a center of gravity of the plurality of vectors included in an associated posting list among the plurality of posting lists, a vector included in an associated posting list among the plurality of posting lists, or a vector not included in an associated posting list among the plurality of posting lists. 2. The information processing device according to claim 1.

[0133] <Appendix 5> A second processor is further provided to generate the plurality of posting lists using the plurality of vectors, and output the plurality of search data generated based on the plurality of posting lists to the storage. 5. An information processing device according to claim 1,

[0134] <Appendix 6> The second processor, generating the plurality of representative vectors based on the plurality of vectors; Based on the generated representative vectors, the vectors are classified into the posting lists; Each of the plurality of posting lists is set to a target posting list that is a target of compression or a non-target posting list that is not a target of compression; The above target posting list can be compressed, Sending the compressed target posting list and the non-target posting list to the storage; It is configured as follows: 6. The information processing device according to claim 5.

[0135] <Appendix 7> the second processor is configured to receive a first compression algorithm; When the second processor receives the first compression algorithm, All of the plurality of posting lists are the target posting lists, performing compression based on said first compression algorithm; Sending the first compression algorithm to the storage; When the second processor does not receive the first compression algorithm, For each of the plurality of posting lists, determine whether to set the posting list as the target posting list or the non-target posting list, and determine a second compression algorithm corresponding to each of the set target posting lists; Compress the corresponding target posting list according to the second compression algorithm; sending the second compression algorithm to the storage; 7. The information processing device according to claim 6.

[0136] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0137] 1...information processing system, 10A, 10B...information processing device, 11...CPU, 12...ROM, 13...RAM, 14...input / output circuit, 15...communication interface, 20...storage device, 30...controller, 31...CPU, 32...ROM, 33...RAM, 34...communication interface, 35...storage interface, 40...non-volatile storage device, 101...representative vector generation unit, 102...search graph generation unit, 103...search graph transmission unit, 104...posting list generation unit, 1 05... compression target determination unit, 106... posting list compression unit, 107... posting list transmission unit, 111... representative vector management unit, 112... first distance calculation unit, 113... posting list determination unit, 114... posting list reception unit, 115... compression information management unit, 116... data decompression unit, 117... second distance calculation unit, 118... neighborhood vector determination unit, 200... storage unit, PL... posting list, cPL... compressed posting list, RV... representative vector, VI... vector information

Claims

1. An information processing device configured to output an answer to a query vector by utilizing a storage configured to store a plurality of search data, Each of the plurality of search data is associated with a posting list, each of the posting lists storing a plurality of vectors, and each of the plurality of search data is an uncompressed posting list or a compressed posting list; a circuit for receiving the query vector; determining a first number of selected posting lists from the plurality of posting lists based on a distance between the query vector received via the circuit and each of a plurality of representative vectors associated with each of the plurality of posting lists; receiving a first number of selected data items associated with each of the first number of selected posting lists from among the plurality of search data items; performing decompression of the compressed posting list when the first number of selection data includes the compressed posting list; a first processor configured to output at least one vector among the plurality of vectors included in the first number of selection posting lists corresponding to the first number of selection data, as the answer, based on a distance between the query vector and each of the plurality of vectors included in the first number of selection posting lists; Equipped with Information processing device.

2. The storage is configured to further store compression algorithm information related to the compressed posting list among the plurality of search data stored therein; the first processor decompresses the compressed selection posting list based on the compression algorithm information; 2. The information processing device according to claim 1.

3. the first processor determines a plurality of selected posting lists from the plurality of posting lists based on a search graph storing the plurality of representative vectors; 2. The information processing device according to claim 1.

4. The representative vector is a vector based on a center of gravity of the plurality of vectors included in an associated posting list among the plurality of posting lists, a vector included in an associated posting list among the plurality of posting lists, or a vector not included in an associated posting list among the plurality of posting lists.

2. The information processing device according to claim 1.

5. A second processor is further configured to generate the plurality of posting lists using the plurality of vectors, and to output the plurality of search data generated based on the plurality of posting lists to the storage.

5. The information processing apparatus according to claim 1.

6. The storage device further includes:

5. The information processing apparatus according to claim 1.

7. The information processing device according to any one of claims 1 to 4, The storage; a device that generates the plurality of posting lists using the plurality of vectors, and outputs the plurality of search data generated based on the plurality of posting lists to the storage; Equipped with Information processing system.

8. 1. An information processing device configured to output data to a storage device for use in searching for an answer to a query using the storage device, the information processing device comprising: a circuit configured to communicate with the storage; generating a plurality of representative vectors based on the plurality of vectors; Based on the generated representative vectors, the vectors are classified into a plurality of posting lists corresponding to the representative vectors, respectively; Setting each of the plurality of posting lists to a target posting list that is a target of compression or a non-target posting list that is not a target of compression; Compression of the target posting list is possible; a processor configured to transmit a plurality of search data including the compressed target posting list to the storage via the circuit; Equipped with Information processing device.

9. An information processing method for outputting an answer to a query vector by using a storage configured to store a plurality of search data, comprising: Each of the plurality of search data is associated with a posting list, each of the posting lists storing a plurality of vectors, and each of the plurality of search data is an uncompressed posting list or a compressed posting list; receiving the query vector; determining a first number of selected posting lists from the plurality of posting lists based on a distance between the received query vector and each of a plurality of representative vectors associated with each of the plurality of posting lists; receiving a first number of selected data items associated with each of the first number of selected posting lists from among the plurality of search data items; performing decompression of the compressed posting list when the first number of selection data includes the compressed posting list; outputting at least one vector among the plurality of vectors included in the first number of selection posting lists corresponding to the first number of selection data, as the answer, based on a distance between the query vector and each of the plurality of vectors included in the first number of selection posting lists; Equipped with Information processing methods.

10. 1. An information processing method for outputting data to a storage device, the data being used to search for an answer to a query using the storage device, the method comprising: generating a plurality of representative vectors based on the plurality of vectors; Based on the generated representative vectors, the vectors are classified into a plurality of posting lists corresponding to the representative vectors, respectively; Setting each of the plurality of posting lists to a target posting list that is a target of compression or a non-target posting list that is not a target of compression; When the target posting list is set, performing at least one compression of the target posting list; Sending a plurality of search data including the compressed target posting list to the storage; Information processing methods.

Images