Pair data generation system and pair data generation method

Abstract

To generate pair data necessary for learning a retrieval model capable of performing matching from a specific viewpoint.SOLUTION: A pair data generation system 100 includes a storage device 101 that holds a document, a process of giving viewpoint information indicating a viewpoint of information search and the document to a predetermined algorithm as an input and outputting summary information of the document based on the viewpoint, a process of giving the summary information and the viewpoint information to a predetermined algorithm as an input and generating an instruction document for query generation based on the summary information, and a processor 105 that executes a process of generating a query and a process of associating the query with the document to generate pair data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention generally relates to a pair data generation system and a pair data generation method, and more particularly to a technology that enables generation of pair data necessary for training a search model capable of matching from a specific perspective. [Background technology]

[0002] With the recent advances in machine learning technology, vector search technologies, which differ from traditional keyword search, have been proposed and are now widely used. Vector search is a search method that recognizes unstructured data such as text and images as numerical vectors and identifies and outputs data with concepts similar to the query based on the similarity (mainly cosine similarity) between those numerical vectors. This type of vector search is an extremely useful technique for efficiently and accurately retrieving documents containing the information required for business from a huge population.

[0003] Note that matching in the above vector search reflects a general perspective, such as "documents that match the topic specified by the query." For example, in Non-Patent Document 1, a large number of "pairs of web articles and their titles" are used to train a search model that reflects this perspective. Since the title of an article is considered to be a condensed representation of the topic of that article, topic matching is learned using this pair data.

[0004] Furthermore, Patent Document 1 discloses a technique for obtaining accurate search results for text pairs in response to an input query.

[0005] The above technology relates to a model creation device that includes: a model training unit that uses, as input, a group of training text pairs consisting of a pair of a first training text and a second training text that serves as an answer when the first training text is posed as a question, and trains a query expansion model that creates a text that becomes an expanded query for a text that becomes a query; and a word filtering unit that uses a search target text word list consisting of each word included in a group of search target text pairs consisting of a pair of a first text to be searched and a second text to be searched that serves as an answer when the first text to be searched is posed as a question, and the model training unit trains the query expansion model based on the group of training text pairs from which the words not included in the search target text word list have been removed by the word filtering unit. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Text Embeddings by Weakly-Supervised Contrastive Pre-training, Wang, Liang, et al. [Patent documents]

[0007] [Patent Document 1] Patent No. 6813591 Summary of the Invention [Problem to be solved by the invention]

[0008] However, when applying the above vector search to the search needs associated with actual business operations, matching from a perspective specific to that operation is often required. Here, we will explain matching from the above specific perspective using the example of a system integration (SI) solution proposal task to a customer. In this task, it is necessary to select an in-house solution that can provide the optimal solution to the issues the customer is facing. Therefore, when performing a vector search in this task, matching is required from the perspective of searching for a "solution that solves the issue" using the "customer's issue" as the input query.

[0009] To train a vector search model that can perform matching from this perspective, a large amount of paired data, consisting of "customer issues and solutions to resolve them," is required. However, manually creating such a large amount of paired data is difficult in terms of time and cost. On the other hand, even if automatic generation is performed using AI, in this case, the query and search results are not similar as vectors, so paired data that leads to suitable search results cannot be generated.

[0010] Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that makes it possible to generate pair data necessary to train a search model that can perform matching from a specific perspective. [Means for solving the problem]

[0011] The present application includes multiple means for solving the above-mentioned problems, examples of which are as follows: To solve the above-mentioned problems, a paired data generation system according to one aspect of the present invention comprises: a storage device for storing documents; a process for applying viewpoint information indicating a viewpoint of information retrieval and the document as input to a predetermined algorithm and outputting summary information of the document based on the viewpoint; a process for applying the summary information and the viewpoint information as input to the predetermined algorithm and generating an instruction document for generating a query based on the summary information; a process for applying the document and the instruction document as input to the predetermined algorithm and generating a query; and a processor for executing processes for linking the query and the document to generate paired data.

[0012] In order to solve the above-mentioned problem, a paired data generation method according to one aspect of the present invention is characterized in that a paired data generation system performs the following steps: storing a document in a storage device; applying viewpoint information indicating a viewpoint for information retrieval and the document as input to a predetermined algorithm; outputting summary information of the document based on the viewpoint; applying the summary information and the viewpoint information as input to the predetermined algorithm; generating an instruction document for generating a query based on the summary information; applying the document and the instruction document as input to the predetermined algorithm; generating a query; and linking the query and the document to form paired data. [Effects of the Invention]

[0013] According to the present invention, it is possible to generate pair data necessary for training a search model that can perform matching from a specific perspective. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a network configuration including a pair data generation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a pair data generation system according to the present embodiment. [Figure 3]FIG. 2 is a diagram illustrating an example of the configuration of a document DB according to the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a pair data DB according to the present embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of the flow of a pair data generation method according to the present embodiment. [Figure 6] FIG. 2 is an explanatory diagram showing a specific example of the functional configuration and processing of the pair data generation system according to the present embodiment. [Figure 7] FIG. 2 is an explanatory diagram showing a specific example of the functional configuration and processing of the pair data generation system according to the present embodiment. [Figure 8] FIG. 2 is an explanatory diagram showing a specific example of the functional configuration and processing of the pair data generation system according to the present embodiment. [Figure 9] FIG. 2 is an explanatory diagram showing a specific example of the functional configuration and processing of the pair data generation system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] In the following description, a communication device may be one or more communication interface devices, which may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., a NIC and an HBA (Host Bus Adapter)).

[0016] In the following description, a "memory" refers to one or more memory devices, which are an example of one or more storage devices. At least one of the memory devices may be a volatile memory device or a non-volatile memory device.

[0017] In the following description, a "storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device, specifically, for example, a hard disk drive (HDD), a solid state drive (SSD), or a non-volatile memory express (NVMe) drive.

[0018] In the following description, a "CPU" refers to a computing device and may be one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a hardware circuit that performs part or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0019] In the following description, information that provides an output in response to an input may be described using expressions such as "xxx table" or "xxx database." However, this information may be data of any structure (for example, structured data or unstructured data), or may be a learning model such as a neural network, genetic algorithm, or random forest that generates an output in response to an input. Therefore, "xxx table" or "xxx database" may be referred to as "xxx information." In the following description, the structure of each database or table is an example, and one database or table may be divided into two or more databases or tables, or all or part of two or more databases or tables may be one database or table.

[0020] In the following description, processing may be described using a "program" as the subject. However, since a program is executed by a CPU to perform a predetermined process using a storage device and / or an interface device, etc., as appropriate, the subject of the process may also be the CPU (or a device such as a controller having a processor). A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0021] In addition, in the following description, when describing elements of the same type without distinguishing between them, common parts of the reference symbols may be used, and when describing elements of the same type with distinction between them, reference symbols or element identifiers may be used.

[0022] <Network configuration including the pair data generation system> FIG. 1 is a diagram showing an example of a network configuration including a paired data generation system 100 according to this embodiment. The paired data generation system 100 according to this embodiment is a system capable of generating paired data necessary for training a search model capable of performing matching from a specific perspective. Unlike conventional techniques (searches based on simple vector similarities between keywords), this paired data generation system 100 generates paired data, i.e., training data for a machine learning model, for a user who wishes to perform vector searches from various perspectives, such as solutions to business problems, to efficiently and accurately search for and provide information that the user originally desired, such as solutions to the perspectives.

[0023] The pair data generation system 100 is communicably connected to a user terminal 200 and a document DB system 300 via an appropriate network N. The user terminal 200 is connected to the pair data generation system 100 and serves as a terminal that executes, for example, a process of notifying the results of user operations such as the designation and selection of a viewpoint, and a process of acquiring and displaying pair data generated by the pair data generation system 100 with respect to the viewpoint. The user of the user terminal 200 may include, for example, a person in charge of generating and learning a vector search model adapted to information searches from various viewpoints, or a person in charge of needing information regarding solutions to business problems.

[0024] The document DB system 300 is a database system that manages, in the form of a database, various documents that are paired with queries when generating pair data. Therefore, the document DB system 300 is implemented in the form of a so-called DB server, and holds and manages a document DB 350. The document DB system 300 also receives requests via the network N from the pair data generation system 100 or the user terminal 200, retrieves documents from the document DB 350, and responds to the pair data generation system 100 or the user terminal 200. The configurations, functions, data, etc. of the document DB system 300 and the document DB 350 may be integrally held and used by the pair data generation system 100.

[0025] The document DB 350 has, for example, the data structure shown in Fig. 3. The document DB 350 is a collection of records in which information such as document name, bibliographical information, article title, and content is linked using a document ID that uniquely identifies a document as a key. The content of a document may be in any format, including text, tables, graphs, illustrations, icons, images, video, audio, and the like.

[0026] The pair data generation system 100 shown in Figures 1 and 2 includes a storage device 101 consisting of a non-volatile storage device such as a hard disk drive or an embedded multimedia card, an I / O 102 for connecting to an input device such as a keyboard and an output device such as a display, a memory 103 consisting of a volatile storage device such as a RAM (Random Access Memory), a communication device 104 connected to a network N and communicating with at least a user terminal 200 and a document DB system 300, and a CPU 105 that calls a program 1011 stored in the storage device 101 into the memory 103 to perform overall control of the system itself and to perform various judgments, calculations, and control processes.

[0027] The functions implemented in the storage device 101 in the paired data generation system 100 include a viewpoint-conditioned document information generation unit 1013, a viewpoint-conditioned instruction generation unit 1014, a viewpoint-conditioned query generation unit 1015, a paired data display unit 1016, a viewpoint generation unit 1017, and a query duplication elimination unit 1018 (see FIG. 2). Details of each of these functions will be described later.

[0028] The storage device 101 also stores at least an LLM 1012 and a pair data DB 1020. Of these, the LLM 1012 is a large-scale language model that generates necessary information for inputs to each of the above-mentioned functional units. Therefore, it is assumed that machine learning corresponding to the function has been performed in advance using predetermined training data.

[0029] For example, as the engine of the perspective-conditioning document information generator 1013, the LLM 1012 has already learned a function to extract corresponding information (problem information in the above case) from a document in response to a prompt (e.g., a prompt to output problems that can be solved by a solution described in a document based on a document stored in the document DB 350) and generate and output text. The LLM 1012 also has a function to generate appropriate information for the perspective-conditioning instruction generator 1014, perspective-conditioning query generator 1015, and perspective generator 1017. The function of the LLM 1012 in each of these functional units will be described later.

[0030] The pair data DB 1020 is a database that stores pair data generated by the pair data generation system 100. The pair data DB 1020 is a collection of records that link together information such as a viewpoint in the pair data and a query and a document / solution that constitute the pair data, using an ID that uniquely identifies the pair data as a key.

[0031] Among these, "perspective" is a concept that indicates the essence of a user's search intent, so to speak, as indicated by a pseudo-query that a user may enter when performing a vector search, i.e., a pseudo-query useful for vector search model training. Examples include "resolve customer issue," "technical interest," "department," and "budget constraint." A "query" is text that indicates a pseudo-query that a user may enter when performing a vector search and that is useful for model training. A "document / solution" is information about documents that should be hit and extracted when a vector search is performed in the document DB 350 using the "perspective" for the "query," and the solutions that are the subject of the hits. Therefore, a set of "query" and "document / solution" in the pair data DB 1020 becomes pair data.

[0032] Note that part of the processing performed by CPU 105 when it executes program 1011 may be executed by another arithmetic device (for example, hardware such as ASIC or FPGA). Also, memory 103 may be a ROM (Read Only Memory), which is a non-volatile storage element. ROM stores unchanging programs (for example, BIOS). RAM is a high-speed, volatile storage element such as DRAM (Dynamic Random Access Memory), and temporarily stores program 1011 executed by CPU 105 and data used when program 1011 is executed.

[0033] Furthermore, the above-mentioned input device and output device may be provided by a user terminal 200 connected to the pair data generation system 100 via a network N. The program 1011 executed by the CPU 105 is provided to the pair data generation system 100 via a removable medium (CD-ROM, flash memory, etc.) or the network N, and is stored in a non-volatile storage device 101, which is a non-transitory storage medium.

[0034] Therefore, the pair data generation system 100 reads data from the removable media via the I / O 102. The pair data generation system 100 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer built on multiple physical computer resources. The pair data generation system 100 is preferably configured on the cloud, but may also be on-premise configured on a specific computer (hardware).

[0035] The network N connecting the pair data generation system 100 with the user terminal 200 and the document DB system 300 may be the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or a mobile phone network, but is not limited to these.

[0036] The user terminal 200 may be implemented in various forms, such as a personal computer (PC), a tablet terminal, a smartphone, or XR (cross reality) goggles. The user terminal 200 may also include a document DB 350, which may be used by the pair data generation system 100 via the network N.

[0037] Note that data exchange between the pair data generation system 100 and the user terminal 200 or the document DB system 300 may be performed according to, for example, an API (Application Programming Interface) protocol. In this case, it is assumed that each device is pre-implemented with functions and configurations for executing each process of requests and responses by the API.

[0038] <How to generate pair data: Main flow> Next, a processing flow in the pair data generation method of this embodiment will be described together with each functional unit in the pair data generation system 100. Fig. 5 is a diagram showing an example of the flow of the pair data generation method of this embodiment, and Figs. 6 to 9 are diagrams showing the functional configuration and processing flow in the pair data generation system 100.

[0039] First, the viewpoint-conditioned document information generation unit 1013 of the pair data generation system 100 reads the document 351 from the document DB 350 of the document DB system 300, and assigns the document 351 and viewpoint information indicating viewpoint K1 for information retrieval as input to the LLM 1012, outputting summary information of the document 351 based on viewpoint K1 (step S10). The example of FIG. 6 shows an example in which viewpoint information K2 "Please list the problems that the solution can solve" indicating viewpoint K1 "Problem solving" is input to the LLM 1012. Note that viewpoint K1 is either assigned in advance by the user via the user terminal 200, or is generated by the LLM 1012 itself by reading the document 351 (perspective generation will be described later).

[0040] In step S10, the perspective-conditioned document information generation unit 1013 obtains summary information K3 from the LLM 1012. The example in Fig. 6 shows a situation in which texts such as "information sharing" and "digitization of paper documents" are generated as summary information K3. The perspective-conditioned document information generation unit 1013 provides this summary information K3 to the perspective-conditioning instruction generation unit 1014.

[0041] Meanwhile, the viewpoint-conditioning instruction generation unit 1014 obtains the summary information K3, and uses this and the viewpoint K1 information as inputs, sets them in the template K0, or assigns them to the LLM 1012, to generate an instruction document K4 for query generation based on the summary information K3 (step S11). The example in Figure 6 shows a situation in which the instruction document K4 has been generated with text such as "Please think of a query that a user who wants to realize information sharing would use when considering this solution" and "A user who wants to realize digitization of paper documents..." The viewpoint-conditioning instruction generation unit 1014 assigns this instruction document K4 to the viewpoint-conditioning query generation unit 1015.

[0042] The viewpoint-conditioning query generation unit 1015 obtains the instruction document K4 and inputs it as a prompt to the LLM 1012 to generate a query K5 (step S12). The example in Fig. 6 shows a situation in which text such as "We want to activate communication between departments" and "We want to utilize the knowledge accumulated within the company" is generated as query K5.

[0043] The perspective-conditioning query generation unit 1015 also associates the query K5 generated up to this point with information K10 indicated by the document 351 (e.g., the name of the document itself, and information on the solution that solves the problem of the query K5 among the information on the solutions described in the document) to generate paired data, and stores this in the paired data DB 1020 (step S13). Note that the perspective-conditioning query generation unit 1015 queries the LLM 1012 to extract from the document 351 information indicated by the document 351 that corresponds to the query K5. In this extraction, when summary information K3 for the perspective information K2 is obtained, the LLM 1012 has naturally identified from the document 351 information on the solution to the problem (e.g., information sharing) indicated by the summary information K3.

[0044] After generating and storing the paired data as described above, the paired data display unit 1016 may output the paired data to, for example, the user terminal 200 and display it on the screen. In the example shown in Fig. 7, the paired data display unit 1016 reads paired data K11 consisting of query K5 and information K10 indicated by document 351 from the paired data DB 1020, and delivers it to the user terminal 200 as a predetermined screen G10 to be displayed. The user can view this screen G10 on the user terminal 200 and check the validity of the paired data, etc.

[0045] The paired data generation system 100 according to this embodiment can also generate the viewpoint K1. In this case, as shown in FIG. 8 , the viewpoint generation unit 1017 inputs feature information K20 of the document 351 extracted from the document 351 and the document 351 itself to the LLM 1012, thereby outputting text representing the viewpoint K1 for information retrieval. The viewpoint generation unit 1017 inputs the value and content of the article title in the document 351 and the extracted viewpoint K21, "solution," to the LLM 1012, thereby identifying, for example, a solution name as element information of the feature information of the document 351. The feature information K20 can be generated by setting this solution name as a template for the feature information K20.

[0046] In the example of FIG. 8, values ​​such as "solution to customer problem," "technology used," and "budget constraint" are generated as perspective K1. This perspective K1 is assigned to the perspective-conditioning document information generation unit 1013. When extracting such characteristic information K20, in addition to using the LLM 1012, known information extraction techniques such as characteristic word extraction may also be appropriately adopted. Furthermore, when the document 351 is structured data such as a table, information such as the solution name, offered price, and target department may be extracted directly according to the corresponding tags, etc.

[0047] As shown in FIG. 9, the query duplication elimination unit 1018 of the pair data generation system 100 preferably eliminates one of the queries K5 generated as described above that are similar to each other, thereby eliminating duplication between queries. Similar queries K5 are, for example, queries K5 whose vector cosine similarity is equal to or exceeds a certain standard. Of course, methods such as the probability of occurrence of the same character or word can also be used to determine similarity, and are not limited to these. This duplication elimination also leads to the elimination of duplication in pair data K11, contributing to improved accuracy in vector search model learning.

[0048] As described above, the pair data generation system 100 according to this embodiment makes it possible to generate pair data necessary for training a search model capable of performing matching from a specific perspective.

[0049] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0050] Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program 1011 that realizes each function.

[0051] The program 1011 that realizes each function, information such as tables and files can be stored in a storage device 101 such as memory 103, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0052] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected.

[0053] The above various explanations can be summarized as follows. The following summary may include supplementary explanations and explanations of variations of the above explanations. In the pair data generation system 100 of this embodiment, the processor may further execute a process of outputting the pair data to a predetermined device.

[0054] This makes it possible to clearly show users (such as the vector search model administrator or solution sales representative) the entity of the pair data used for vector search model training, helping them understand the vector search algorithm and allowing knowledgeable individuals to manually remove inappropriate pair data. Ultimately, it becomes possible to generate with greater accuracy the pair data required to train a search model that can perform matching from a specific perspective.

[0055] In addition, in the pair data generation system 100 of this embodiment, the processor may further execute a process of applying the feature information of the document extracted from the document and the document itself as input to a predetermined algorithm and outputting text representing a viewpoint of information retrieval.

[0056] This eliminates the need for a person managing vector search model learning to specify and select the viewpoints, improving the efficiency of model learning. This in turn makes it possible to more efficiently generate pair data necessary for learning a search model capable of matching from a specific viewpoint.

[0057] In addition, in the pair data generation system 100 of this embodiment, the processor may further execute a process of eliminating overlapping queries by eliminating one of the generated queries that are similar to each other.

[0058] This allows us to proceed with model learning after eliminating queries that reduce the accuracy of vector search model learning in advance, which in turn allows us to generate more accurate pair data necessary for learning a search model that can perform matching from a specific perspective. [Explanation of symbols]

[0059] N Network 100 Pair Data Generation System 101 Storage device 1011 Program 1012 LLM 1013 Viewpoint Conditioned Document Information Generation Unit 1014 viewpoint conditioning instruction generation unit 1015 Viewpoint Conditioning Query Generation Unit 1016 Pair data display section 1017 Viewpoint Generation Unit 1018 Query Deduplication Unit 1020 Pair Data DB 102 I / O 103 memory 104 Communication equipment 105 CPUs (processors) 200 user terminals 300 Document DB System 350 Document DB

Claims

1. a storage device for holding documents; a processor that executes a process of applying viewpoint information indicating a viewpoint for information retrieval and the document as input to a predetermined algorithm, and outputting summary information of the document based on the viewpoint; a process of applying the summary information and the viewpoint information as input to a predetermined algorithm, and generating an instruction document for generating a query based on the summary information; a process of applying the document and the instruction document as input to a predetermined algorithm, and generating a query; and a process of linking the query and the document to generate paired data; A pair data generation system comprising:

2. 2. The pair data generation system according to claim 1, wherein the processor further executes a process of outputting the pair data to a predetermined device.

3. 2. The pair data generation system according to claim 1, wherein the processor further executes a process of applying the document feature information extracted from the document and the document itself as input to a predetermined algorithm to output text representing a viewpoint of information retrieval.

4. The pair data generation system according to claim 1 , wherein the processor further executes a process of eliminating overlapping queries by eliminating one of the generated queries that are similar to each other.

5. A pair data generation system including at least a processor and a storage device, The document is stored in a storage device; a process of inputting viewpoint information indicating a viewpoint for information retrieval and the document to a predetermined algorithm, and outputting summary information of the document based on the viewpoint; a process of inputting the summary information and the viewpoint information to a predetermined algorithm, and generating an instruction document for generating a query based on the summary information; a process of inputting the document and the instruction document to a predetermined algorithm, and generating a query; and a process of linking the query and the document to generate paired data.

2. A pair data generating method comprising:

Images