Information processing device, vector generation method

Abstract

This technology provides the ability to generate vector representations suitable for searching and classifying documents within a specific domain, without requiring additional training of the language model. [Solution] The search device 1 comprises a transformation unit 12, an adaptation target data storage unit 14, a synthesis unit 15, and a kNN search unit 16. The transformation unit 12 has a BERT model and generates a value vector and a key vector for each domain document, which are stored in the adaptation target data storage unit 14. The value vector is the BERT output, and the key vector is a vector representation generated by the intermediate layer of the BERT model. The synthesis unit 15, in cooperation with the kNN search unit 16, obtains neighboring examples similar to the target document in terms of key vectors from the adaptation target data storage unit 14. Then, by synthesizing the value vectors of the neighboring examples and the value vector of the target document, it generates an adaptation vector, which is the final vector representation of the target document.

Description

[Technical Field]

[0001] This disclosure relates to a technology for generating vector representations of documents that can be used for searching, classifying, or other purposes. [Background technology]

[0002] Patent Document 1 discloses a machine translation system using the k-Nearest Neighbor (kNN) method. The system disclosed in Patent Document 1 searches a search data store using a time-dependent hidden state vector generated based on the features of the input sentence as a query, and outputs words corresponding to neighboring examples (e.g., translated words).

[0003] Non-patent document 1 discloses the BERT (Bidirectional Encoder Representations from Transformers) model used in Natural Language Processing (NLP). The BERT model is a transformer-based language model (LM) that is said to be excellent at accurately capturing the meaning of words by considering the context of the entire document. BERT models are often pre-trained using a large text corpus and then fine-tuned for a specific domain or task. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2024-124959 [Non-patent literature]

[0005] [Non-Patent Document 1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics. [Overview of the project] [Problems that the invention aims to solve]

[0006] The application and utilization of language models to specific domains (e.g., within a company) is being considered. A BERT model pre-trained using text data from the web can generate vector representations that reflect the content and meaning of typical documents. By constructing a vector database that stores these document-specific vector representations linked to document IDs, a vector-based document search system can be realized.

[0007] However, while language models such as BERT are good at vectorizing general documents, they are not good at vectorizing documents that contain domain-specific expressions and words, such as internal company documents. Therefore, when using a language model to accurately vectorize documents of a specific domain, such as internal company documents, additional training using documents of that specific domain (so-called fine tuning) is necessary. Herein lies the challenge that training language models is generally computationally expensive, and training itself may not be possible if there is insufficient data.

[0008] This disclosure is made based on the circumstances described above, and one of its purposes is to provide a technology that can generate vector representations suitable for searching and classifying documents in a specific domain without requiring additional training of the language model. This disclosure makes additional training of the language model an optional element and does not prohibit it. This disclosure may also be applied to information processing devices that use an additionally trained language model. [Means for solving the problem]

[0009] The information processing device disclosed herein is an information processing device that generates a vector representation of a document using a predetermined language model, comprising: a conversion unit (12) that acquires a key vector, which is the internal representation of the language model for the document, and a value vector, which is the output of the language model for the document; an adaptation target data storage unit (14) in which the key vectors and value vectors for each of a plurality of domain documents, which are documents of the domain to be applied to, are stored in association with each other; a document acquisition unit (11) that acquires a target document, which is the document to be used for generating the vector representation; a neighbor example acquisition unit (16) that acquires information on neighbor examples of the target document from the adaptation target data storage unit; and The system includes a vector generation unit (15) that generates a vector representation of a document using the value vectors of neighboring examples acquired by a neighboring example acquisition unit. The transformation unit generates a key vector and a value vector for the target document acquired by the document acquisition unit as the target key vector and target value vector. The neighboring example acquisition unit identifies neighboring examples of the target document from the target data storage unit using a k-nearest neighbor search method with the key vector and provides the value vectors of the neighboring examples to the vector generation unit. The vector generation unit generates an adapted vector, which is a vector representation of the target document, by combining the value vectors of the neighboring examples and the target value vector.

[0010] Furthermore, the vector generation method included in this disclosure is a vector generation method for generating a vector representation of a document using a predetermined language model, and includes: obtaining a target document which is the document for which the vector representation is to be generated; generating a key vector which is the internal representation of the language model for the target document and a value vector which is the output of the language model for the target document as the target key vector and target value vector; identifying neighboring cases of the target document from an adaptation target data storage unit (14) in which the key vectors and value vectors for each of a plurality of domain documents which are documents of the domain to be adapted are stored in association with each other, using a k-nearest neighbor search method with key vectors, and obtaining the value vectors of the neighboring cases; and generating an adaptation vector which is a vector representation of the target document by combining the value vectors of the neighboring cases and the target value vector.

[0011] According to the above technology, the vector representation of the target document is generated by combining the value vector of the target document with the value vector of a domain document that is similar in terms of key vectors. The vector representation generated in this way is a representation that includes domain-specific representations, which are included in domain documents that are similar to the target document. Therefore, it becomes easier to generate a vector representation suitable for searching and classifying documents of a specific domain without performing additional training on the language model.

[0012] The reference numerals in parentheses in the claims indicate the correspondence with the specific means described later in the embodiments, and do not limit the technical scope of this disclosure. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing an example of the overall structure of a search system. [Figure 2] This diagram shows the structure of the language model in the conversion unit. [Figure 3] This figure shows an example of data stored in the target data storage unit. [Figure 4] It is a diagram showing an example of data stored in the document vector storage unit. [Figure 5] It is a diagram showing the flow of information related to the construction of the destination data storage unit. [Figure 6] It is a diagram showing the flow of information in the similar document search.

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The present disclosure is not limited to the following embodiments. The configurations disclosed below may be variously modified and implemented without departing from the gist. Various modifications may be appropriately combined and implemented within a range where no technical contradiction occurs. The present disclosure includes configurations that are not explicitly stated, which are combinations of a plurality of modifications. In the following description, members having the same function may be denoted by the same reference numerals, and specific descriptions thereof may be omitted. Also, members having the same function may be given the same or similar names, and specific descriptions thereof may be omitted. When only a part of the configuration is mentioned, the descriptions given elsewhere may be applied to other parts.

[0015] <Overall Image> FIG. 1 shows a search system as one embodiment of the present disclosure. The search system of this embodiment includes a search device 1, an input device 2, a document DB 3, and a display device 4. Note that DB is an abbreviation for database. Hereinafter, a case where the search system is applied to a company including, for example, Department A and Department B and is used for in-company similar document search will be exemplified.

[0016] The search device 1 is a device that analyzes a document input by a user and outputs a search result for documents similar thereto. The search device 1 corresponds to an information processing device. The search device 1 is communicatively connected to each of the input device 2, the document DB 3, and the display device 4. Note that the search device 1 may be realized using one or more servers. The search device 1 may be realized in the form of a cloud server.

[0017] Input device 2 is a device for receiving user input. Input device 2 may be a keyboard, mouse, touch panel, or any other device. Input device 2 may be connected to search device 1 by wire or wirelessly. Input device 2 may also be a device spatially separated from search device 1, such as a smartphone, tablet, or laptop. Input device 2 inputs data corresponding to user input into search device 1.

[0018] Document DB3 is a database that stores multiple documents, including documents belonging to the target domain (hereinafter also referred to as the target domain). Document DB3 may be, for example, a database that stores documents from all departments of a company. Documents in Document DB3 may be stored in text format or other formats. Documents stored in Document DB3 may be assigned attribute information, which is information used to determine the domain. Attribute information may include information about the managing department, author information, or keywords. Author information may include information about the author's department.

[0019] Document DB3 may be a database containing only documents within the target domain. Here, "domain" can be understood as the scope of a search in a particular context. A domain may be limited to a specific department / division within a company. A domain may also be defined in terms of a specific technical field, industry, business, operation, job, or task. Target domains may be set according to their intended use / purpose.

[0020] Document DB3 is connected to search device 1 via a communication network. The connection between search device 1 and document DB3 may be wired or wireless. Document DB3 may be located on the cloud. Document DB3 may contain multiple databases.

[0021] Display device 4 is a device that displays the search results from search device 1. Display device 4 may be a display. Display device 4 may be a device spatially separated from search device 1, such as a smartphone, tablet, or laptop. Display device 4 displays an image corresponding to the data input from search device 1. For example, display device 4 displays the search results from search device 1.

[0022] <Search device> The search device 1 is a computer that includes one or more processors, memory, storage, and input / output circuits. The processor may be any arithmetic core, such as a GPU (Graphics Processing Unit), NPU (Neural Processing Unit), or DFP (Data Flow Processor). The memory may be any volatile memory, such as RAM (Random Access Memory). The storage may be flash memory, SSD (solid state drive), HDD (hard disk drive), etc. The storage may also be cloud storage. The input / output circuits may be circuits for the search device 1 to receive data from the input device 2 and document DB 3, and output data to the display device 4. The memory / storage of the search device 1 stores programs and language models for providing the functions of the search device 1 described below.

[0023] The search device 1, as a functional block provided by the processor, includes an input unit 11, a conversion unit 12, an adapted data construction unit 13, an adapted data storage unit 14, a synthesis unit 15, a kNN search unit 16, a search unit 17, and a document vector storage unit 18, as shown in Figure 1. kNN stands for k-Nearest Neighbor. The input unit 11, conversion unit 12, adapted data construction unit 13, adapted data storage unit 14, synthesis unit 15, and kNN search unit 16 function as a domain adaptor 10 for generating an adapted vector, which will be described later. The domain adaptor 10 may be referred to as a vector adaptor or the like.

[0024] The input unit 11 acquires multiple data related to similar document search and database construction from the input device 2. For example, the input unit 11 acquires the k value, the specified value for the key vector extraction layer, and domain information. Database construction here refers to the construction of the target data storage unit 14 and the document vector storage unit 18, which will be described later.

[0025] The k-value is a parameter used in kNN search. The k-value defines the range of the kNN search. The k-value is set to an integer of 1 or greater (more preferably 2 or greater). For example, the k-value is set to 6. The k-value may also be 4, 8, 12, etc. The k-value acquired by the input unit 11 is stored in memory and referenced by the kNN search unit 16. The specified value for the key vector extraction layer is a parameter used by the conversion unit 12, which will be described later. The input unit 11 provides the acquired specified value for the key vector extraction layer to the conversion unit 12. The technical significance of the specified value for the key vector extraction layer will be described separately later.

[0026] Domain information specifies the target domain. For example, the domain can be selected from a predetermined list of options, such as the entire company or a specific department. The input unit 11 provides the acquired domain information to the target data construction unit 13. Hereafter, documents stored in the document DB3 that correspond to the domain information will also be referred to as domain documents.

[0027] If the search device 1 has multiple language models, the input unit 11 may obtain information via the input device 2 to specify which language model to use for processing (hereinafter referred to as the "used model"). The language model corresponds to the conversion unit 12, which will be described later. A configuration that allows the use of multiple language models corresponds to a configuration in which the conversion units 12 exist in parallel. When the input unit 11 receives information on the used model, the search device 1 executes the following processing using the specified language model.

[0028] Furthermore, during similar document searches, the input unit 11 retrieves query documents based on user operations. A query document is the document that serves as the basis (seed) for the search / classification, and may be rephrased as a document of interest, etc. When the input unit 11 receives a query document, it inputs the query document into the conversion unit 12. In addition, the input unit 11 may acquire parameters such as the number of output items M, which defines the number of documents to be output as search results, and reflect these parameters in the operation of the search device 1. The input unit 11 corresponds to an example of a document acquisition unit.

[0029] The conversion unit 12 is a module that generates a vector representation of an input document. During similar document search, the conversion unit 12 receives a query document from the input unit 11. During DB construction / update, the conversion unit 12 may receive a domain document from the target data construction unit 13. During DB construction / update, the domain document received from the target data construction unit 13 corresponds to the target document, and in the similar document search phase, the query document corresponds to the target document. The target document can be understood as the document that will be used to generate the adaptive vector, which will be described later. The input document here can be understood as the document input to the conversion unit 12 from the input unit 11 or the target data construction unit 13. The term "input document" can also be rephrased as "conversion target document". Since the conversion unit 12 converts the input document into a vector (a kind of code) corresponding to its content and outputs it, the conversion unit 12 can also be referred to as the encoding unit. In this disclosure, generating a vector representation corresponding to a document is also referred to as document vectorization or encoding.

[0030] When a document is input to the conversion unit 12, it analyzes the input document and divides it into multiple tokens. Each token corresponds to a word. Tokens may be generated using any tokenizer. The conversion unit 12 is an implementation of a language model and is configured to generate a key vector and a value vector from the input tokens. Both the key vector and the value vector are vectors that represent the characteristics of the input document. The key vector is the internal representation of the language model. The key vector can be the vector representation of the input document in the intermediate layer. The value vector is the vector representation as the output of the language model. The vector representation of a document is a vector (i.e., an array of numbers).

[0031] The transformation unit 12 in this embodiment is realized by implementing a BERT model. The transformation unit 12 may be a language model that has undergone basic pre-training with predetermined training data, but has not undergone additional training (so-called fine-tuning) according to the target application. In the following description, the case in which a BERT-base model having 12 transformer layers is used as the transformation unit 12 will be explained.

[0032] The transformation unit 12, as shown in Figure 2, comprises an embedding layer 121 and a 12-layer transformer layer 122 for processing input tokens. The embedding layer 121 is a layer that converts input tokens into a vector representation. The term "layer" may be replaced with "block" or "processing module," for example. The transformer layer may also be called a transformer block.

[0033] The embedding layer 121 performs token embedding, segment embedding, position embedding, etc. Token embedding may involve converting each token into a fixed-length vector. Position embedding may involve embedding positional information of a token in a sentence (which word it is). Segment embedding may involve embedding positional information in a document, such as whether it is the first or second sentence. The vector representation generated by the embedding layer 121 is input to the first layer, the transformer layer 122.

[0034] Each of the multiple transformer layers 122 is configured to function as an encoder. Each transformer layer 122 performs operations such as multi-head attention, feed-forward, skip-connection, and layer normalization. Details of the BERT model including the transformer layers are disclosed in Non-Patent Document 1 or its references, and therefore a detailed explanation is omitted here.

[0035] Multiple transformer layers 122 are connected in series, and the output of the last transformer layer 122 corresponds to the output of the BERT model (also called the BERT output). The BERT output contains CLS tokens. The BERT output corresponds to a vector that condenses the information of the entire input document. In this way, the BERT model functions like a feature extractor. Note that the number of transformer layers 122 in the language model used by the transformation unit 12 is not limited to 12, but may be 6, 18, 24, etc. Among the multiple transformer layers 122, the layer located in the middle is also called the central layer.

[0036] The key vector may be any one output of the transformer layers 122 from the first to the twelfth layer. For example, the transformation unit 12 takes the output of the transformer layer 122 of the first layer as the key vector. The key vector may also be inside a transformer layer. For example, the key vector may be a vector representation after multi-head attention in a certain layer, or a vector representation after feedforward. The key vector may be a vector representation at any point (also called a phase) inside a transformer layer.

[0037] Of the multiple transformer layers 122, the layer to be used as the source of the key vector may be determined by the specified value of the key vector extraction layer. If the specified value of the key vector extraction layer is φ, the transformation unit 12 obtains the output (or internal vector representation) of the φ-th transformer layer 122 from the bottom as the key vector. When the transformation unit 12 has 12 transformer layers 122, φ may be any value from 1 to 12.

[0038] Furthermore, when the 12th layer is used as a key extraction layer, it is preferable for the transformation unit 12 to obtain the vector representation before layer regularization, multi-head attention, or feedforward in the 12th layer as the key vector in order to ensure that the key vector is a different vector from the value vector. Also, when the number of transformer layers 122 in the transformation unit 12 is 24, φ may be any value from 1 to 24.

[0039] Generally, vector representations in shallower layers (such as the first layer) represent more superficial literal meanings, while vector representations in deeper layers (such as the eleventh layer) can represent more semantic meanings that reflect a deeper understanding of the input sentence. If the language model used by the conversion unit 12 is pre-trained on the domain, a deeper layer is preferable as the key extraction layer because it can obtain a vector representation that takes context into greater consideration. On the other hand, if the language model used by the conversion unit 12 is a publicly available model, in other words, if it has not been pre-trained on the domain (e.g., fine-tuned), the model may convert unfamiliar words (such as detailed equipment names or company jargon) into meanings different from their original meanings. Therefore, in the case of a language model that has not undergone additional training on the target domain, as in this embodiment, a shallower layer than the middle layer is preferable as the key extraction layer.

[0040] The key vector extraction layer in DB construction may be determined based on the model characteristics of the transformation unit 12 (e.g., whether it is tuned or not). The key vector extraction layer may be a shallower layer than the central layer, or a deeper layer than the central layer. In this embodiment, the key vector extraction layer (i.e., the φ value) is set by the user, but in other embodiments, the key vector extraction layer may be fixed.

[0041] The value vector may be a vector representation of the BERT output. Such a value vector corresponds to the hidden state vector of the final layer. In other embodiments, the value vector may be a vector obtained by performing mean pooling. For example, the value vector may be the mean pooling output of the hidden state vectors of the final layer corresponding to the tokens. Note that the value vector may be obtained in a manner similar to the general procedure for obtaining a document vector, not limited to the method illustrated here.

[0042] In this way, the conversion unit 12 uses the language model to create a vector representation e corresponding to the input document d. LM (d) is obtained. If LMEmbed(d) is the function that obtains the value vector which is the BERT output for the input document d, then eLM (d) is represented as LMEmbed(d). When the transformation unit 12 obtains a set of key vectors and value vectors corresponding to the domain document, it inputs them to the target data construction unit 13 and the synthesis unit 15. Also, when the transformation unit 12 obtains a set of key vectors and value vectors of the query document, it inputs them to the synthesis unit 15.

[0043] The target data construction unit 13 is a block for constructing a target data storage unit 14 that corresponds to a domain specified by the user. The target data construction unit 13 receives domain information from the input unit 11.

[0044] The target data construction unit 13 retrieves domain documents, which are documents corresponding to the domain information received from the input unit 11, from the document DB 3. The target data construction unit 13 is also an example of a document acquisition unit. Domain documents are documents that have attribute information belonging to the domain designated as the target domain. If the target domain is Department A, then documents created in Department A are included in the domain documents. Departments A and B may also be registered in the target data construction unit 13 as related departments. In that case, not only documents created in Department A but also documents created in Department B may be extracted as domain documents. As mentioned above, the classification of domains may be defined from perspectives such as medicine, economics, law, sports, electrical engineering, history, etc. In this embodiment, the descriptions of Department A and Department B may be read as Field A and Field B.

[0045] The target data construction unit 13 inputs the domain documents read from the document DB3 into the conversion unit 12. The target data construction unit 13 then obtains key vectors and value vectors for the domain documents from the conversion unit 12, associates them with each other, and stores them in the target data storage unit 14. Each domain document is assigned a unique document ID, and multiple domain documents may be distinguished by their document IDs. The document ID corresponds to an identification number. When inputting domain documents into the conversion unit 12, the target data construction unit 13 may also notify the conversion unit 12 of the document ID.

[0046] Figure 3 shows an example of the data structure stored in the target data storage unit 14, where the key vector and value vector for each document are stored in association with the document ID. The target data storage unit 14 corresponds to a database where two types of features, such as the key vector and value vector of domain documents, are stored.

[0047] The synthesis unit 15 is a block that generates adaptive vectors for input documents in cooperation with the kNN search unit 16. The input documents for the synthesis unit 15 may be the same as the input documents for the transformation unit 12. During the DB construction phase, the synthesis unit 15 receives the key vector, value vector, and document ID of the domain document from the transformation unit 12. During the similar document search phase, the synthesis unit 15 receives the key vector and value vector of the query document from the transformation unit 12. The synthesis unit 15 and the transformation unit 12 may operate in conjunction with each other.

[0048] When the synthesis unit 15 receives a key vector or the like from the transformation unit 12, it inputs the key vector (hereinafter referred to as the target key vector) to the kNN search unit 16. The kNN search unit 16 may be an implementation of the kNN method. Upon receiving the target key vector from the synthesis unit 15, the kNN search unit 16 extracts neighboring examples (also called neighboring documents) of the input document from the application target data storage unit 14. The kNN search in this embodiment is performed from the perspective of the key vector. The search for neighboring documents may be performed using any distance function, such as cosine similarity. Neighboring documents are a set of k domain documents extracted in order of proximity to the input document in the key vector space.

[0049] The kNN search unit 16 obtains the value vectors and vector distances of k domain documents that correspond to neighboring examples for the input document. The vector distance is the distance between the target key vector and the key vector of the document found by the kNN search. The vector distance can be understood as a parameter corresponding to similarity. The number of domain documents to be searched using kNN is determined by the k value obtained by the input unit 11. Once the kNN search unit 16 has obtained the value vectors and vector distances of k neighboring documents, it provides this data to the synthesis unit 15 as search results.

[0050] The synthesis unit 15 obtains the value vectors and vector distances of k neighboring documents from the kNN search unit 16, and then averages the k value vectors by the vector distance to create an averaged neighboring vector e DS (d) is generated. Averaged neighborhood vector e DS (d) is expressed by the following equation 1. In equation 1, "w j This can be determined by the following equation 2.

[0051]

number

[0052]

number

[0053] According to Equation 1, the value vector comprehensively representing the group of neighboring documents considering the distance from the input document in terms of the key vector is the averaged neighboring vector e DS (d) obtained as. When the synthesis unit 15 generates the averaged neighboring vector e DS (d), the adaptation vector is generated by synthesizing the averaged neighboring vector e DS (d) and the value vector of the input document obtained from the conversion unit 12 with the weight λ.

[0054] Let the output of the conversion unit 12 for document d be e LM (d), the averaged neighboring vector for document d be e DS (d), and the adaptation vector be e combined (d). Then, e combined (d) may be expressed by the following Equation 3 using the weight λ. The weight λ used in Equation 3 may be a variable parameter acquired by the input unit 11 via the input device 2, or may be a fixed value. The weight λ can take a value from 0 to 1.

[0055]

Equation

[0056] The adaptive vector calculated in this manner is a vector that reflects domain-specific words and semantic expressions not present in the language model's output. In other words, it is a vector representation that brings the language model's output closer to the document features to be used in the target domain. Therefore, even if the language model as the transformation unit 12 has not been further trained, the final output vector representation (i.e., the adaptive vector) will be a vector representation that is more adapted to the target domain than the raw output of the language model. Here, raw output refers to the unprocessed, unadjusted output.

[0057] In this disclosure, the process of combining the vector representation (i.e., value vector) output by a language model for a given document with the vector representations of neighboring documents of that document is also referred to as adaptation. The composition unit 15 can be understood as a block that performs the adaptation of the value vector of the input document, and the composition unit may also be referred to as the adaptation unit. The composition unit 15 corresponds to the vector generation unit, and the kNN search unit 16 corresponds to the neighbor example acquisition unit.

[0058] The division of roles between the kNN search unit 16 and the synthesis unit 15 may be changed as appropriate. The kNN search unit 16 may perform tasks ranging from extracting neighboring examples to generating averaged neighbor vectors. In that case, the synthesis unit 15 may calculate an adaptive vector using the averaged neighbor vectors input from the kNN search unit 16. Furthermore, the kNN search unit 16 may be included within the synthesis unit 15. In other words, the kNN search unit 16 may be a sub-function of the synthesis unit 15.

[0059] The synthesis unit 15 generates adaptive vectors for domain documents and stores these adaptive vectors in the document vector storage unit 18 along with the document ID and text indicating the document content. The processor, acting as the domain adaptor 10, performs registration processing for the corresponding adaptive vectors for each of the multiple domain documents stored in the document database. This constructs the document vector storage unit 18 as a search database adapted to the target domain.

[0060] The document vector storage unit 18 can be referred to as a data store or the like. As shown in Figure 4, the document vector storage unit 18 stores the ID and adaptive vector for each domain document as document vectors. Note that "AAAAAA" and "BBBBBB" shown in Figure 4 represent the content of the corresponding document. The document content is the text contained in the document and corresponds to the source of the key vector and value vector.

[0061] Figure 5 schematically shows the data flow related to the construction of the document vector storage unit 18 corresponding to domain documents. When the input unit 11 acquires domain information indicating the target domain based on user operation, the target data construction unit 13, in cooperation with the conversion unit 12, constructs the target data storage unit 14 in which the key vector and value vector for each domain document are stored. In addition, the synthesis unit 15 generates the target vector for each domain document in cooperation with the kNN search unit 16 and stores it in the document vector storage unit 18.

[0062] On the other hand, the search device 1 of this embodiment has a function to search for domain documents similar to the query document entered by the user. Figure 6 schematically shows the data flow related to the provision of the search function. When the input unit 11 obtains the query document data from the input device 2, it inputs the query document to the conversion unit 12. The conversion unit 12 generates a key vector and a value vector of the query document as an input document, similar to when a domain document is input, and provides this data to the synthesis unit 15. The key vector of the query document corresponds to the target key vector mentioned above. The value vector of the query document may also be called the target value vector or query vector.

[0063] When the synthesis unit 15 receives the key vector and value vector of the query document from the conversion unit 12, it inputs the key vector of the query document as the target key vector to the kNN search unit 16. Based on the target key vector input from the synthesis unit 15, the kNN search unit 16 extracts neighboring documents of the query document from the perspective of the key vector and returns their value vectors and vector distance data to the synthesis unit 15. The value vectors of neighboring documents may be expressed as neighbor value vectors or neighboring document vectors, etc.

[0064] The synthesis unit 15 obtains the value vectors and vector distances of k neighboring documents related to the query document from the kNN search unit 16, and then averages the k value vectors by vector distance to obtain an averaged neighboring vector e DS (d) is generated. Averaged neighborhood vector e DS The method for generating (d) may be as described above. Also, the synthesis unit 15 generates the averaged neighborhood vector e DS (d) When generating the value vector of the query document and the averaged nearest neighbor vector e DS (d) and are combined using formula 3 to generate an adaptive vector. When the combination unit 15 obtains an adaptive vector corresponding to the query document, it inputs it to the search unit 17.

[0065] The search unit 17 is a block that performs a search for domain documents similar to the query document, i.e., a similar document search. Upon receiving the adaptive vector of the query document from the synthesis unit 15, the search unit 17 retrieves M documents similar to the adaptive vector from the document vector storage unit 18. M is a parameter that defines the upper limit of the number of search results to be retrieved. The value of M may be a fixed value, or it may be obtained by the input unit 11. The similarity between the query document and the domain documents may be evaluated in terms of the adaptive vector. The search unit 17 may be configured to operate according to the kNN method.

[0066] For example, the search unit 17 sorts the documents stored in the document vector storage unit 18 in descending order using a predetermined distance function and obtains the top M documents as the set of search results. The distance function used in the final similar document search may also be a function that calculates any metric, such as cosine similarity. The search unit 17 may be configured to transmit image data representing the search results to the display device 4. The display device 4 displays information corresponding to the input data from the search unit 17.

[0067] Furthermore, during the search phase, the input unit 11 may acquire k-values ​​and specified values ​​for the key vector extraction layer as search parameters. If the document vector storage unit 18 stores data from multiple domains, the input unit 11 may acquire domain information to be searched as search range specification information. The search unit 17 may perform a similar document search using adaptive vectors for the specified domains.

[0068] <Regarding effects, etc.> Language models that have not undergone additional training may experience decreased accuracy when analyzing documents containing industry-specific or technical terms, such as internal company documents. This is because language models are pre-trained using documents with general content, such as those found in online encyclopedias. While such language models can improve their accuracy by being further trained with documents from the target domain, the training process for language models is generally computationally expensive, and training itself may not be possible if there is insufficient data for additional training. Furthermore, assuming additional training raises concerns about maintenance costs for model updates and the difficulty in learning rare facts. Note that additional training here can be understood as fine-tuning or continuous pre-training.

[0069] To address these challenges, the transformation unit 12 of the domain adaptor 10 outputs two types of vector representations for a single document: a key vector and a value vector, and stores them in the target data storage unit 14. Then, the synthesis unit 15 of the domain adaptor 10 generates an adapted vector for the query document using the vector representations of the domain document stored in the target data storage unit 14. Specifically, the adapted vector is generated by synthesizing (also called mixing) the value vectors of neighboring examples within the domain discovered by the kNN method with the value vector of the query document.

[0070] Such adaptive vectors are vector representations that reflect domain features more accurately than the value vectors themselves. For example, adaptive vectors can be vector representations that include specific nouns, named entities, numbers, dates, etc., which are difficult to learn using language models. Adaptive vectors can also be vector representations that take into account rare content (also known as rare facts), for which it is difficult to secure sufficient training data. In other words, according to this embodiment, it is possible to obtain a higher quality vector representation of query documents without additional training.

[0071] Thus, the domain adaptor 10 of this embodiment makes it possible to generate a more accurate (in other words, more domain-fitting) vector representation of a query document that reflects semantic representations of words and documents that are not present in the original language model or the original training data. Therefore, the search device 1 including the domain adaptor 10 can achieve high accuracy in searching for documents related to a domain. Furthermore, when the domain adaptor 10 is applied to a document classifier, the accuracy of document classification can also be improved.

[0072] The above configuration corresponds to a configuration that incorporates the kNN method into document vectorization using a language model in one aspect. By combining a language model and the kNN method, it becomes possible to generate vector representations that cover long-tail phenomena such as rare cases without additional training of the language model.

[0073] Furthermore, the target data storage unit 14 of the above embodiment includes a key vector, which is an intermediate output, in addition to the value vector, which is the final output of the language model. The neighboring examples used as material for generating the adaptive vector are extracted from the perspective of the key vector, not the value vector. The inventors of this disclosure have found that extracting neighboring examples using an intermediate key vector, rather than using the value vector, can result in an adaptive vector that is more domain-adapted and has higher accuracy as a vector representation. In other words, the configuration of this embodiment can achieve higher search accuracy compared to configurations that generate adaptive vectors or construct data stores using only the value vector. One feature of the above search device 1 is that the search for neighboring examples for generating adaptive vectors is performed based on the distance between key vectors, while the final similar document search is performed based on the distance between adaptive vectors.

[0074] Furthermore, the search device 1 and domain adaptor 10 described above are configured to have a variable layer for extracting key vectors, depending on the settings. By setting a key vector extraction layer according to the characteristics of the domain and the user layer, it becomes possible to generate vector representations suitable for the characteristics of the domain and users as adaptive vectors.

[0075] <Variation> When a new domain document is added to the document DB3, the search device 1 may generate a key vector, a value vector, and an adaptation vector for the added domain document and update the adaptation target data storage unit 14 and the document vector storage unit 18. The target document may be a domain document added to the document DB3.

[0076] The search device 1 may be configured to allow selection of the similarity evaluation metric used in the kNN search. In other words, the distance function may be switchable. In that case, the input unit 11 may receive data indicating the distance function / evaluation metric used in the kNN search via the input device 2 and input it to the kNN search unit 16. The kNN search unit 16 may be configured to extract neighboring examples using the distance function specified by the user.

[0077] Similarly, the search device 1 may be configured to allow the selection of a similarity evaluation metric used to search for domain documents similar to the query document. In that case, the input unit 11 may receive data indicating the distance function / evaluation metric to be applied to the search unit 17 via the input device 2 and input it to the search unit 17. The search unit 17 may be configured to extract domain documents similar to the query document using a distance function specified by the user.

[0078] The above describes the case where the conversion unit 12 is configured to operate according to the BERT model, but the conversion unit 12 may also support language models other than the BERT model. For example, the conversion unit 12 may not be limited to the BERT model, but may be an implementation of other language models such as GPT(registered trademark)-4 or Llama 3. Furthermore, the domain adaptor 10 may have conversion units 12 in parallel, each corresponding to a different language model.

[0079] Furthermore, although the conversion unit 12 in the above embodiment uses a language model that has not been further trained, this disclosure may also be applied to search devices, etc., using a language model that has been further trained. The conversion unit 12 may also be a language model that has been further trained. Even if the language model has been further trained, by applying domain adaptation processing of vector representations using neighboring examples to its output, as in this disclosure, the ability to handle rare cases and the like can be further improved.

[0080] The k-value used by the kNN search unit 16 may be determined by cross-validation, error rate, or elbow method. The k-value may also be determined considering the domain characteristics. If the domain includes multiple categories and the categories are closely related, the k-value may be set to a small value. The k-value may also be determined by experimentation and iteration.

[0081] The k value does not necessarily have to be 2 or greater; k=1 is also acceptable. In other words, the kNN search unit 16 may perform nearest neighbor search. The description of k-nearest neighbor search may be read as nearest neighbor search. When k=1, the averaged neighbor vector may be the value vector of the domain document with the most similar key vector, in which case the description of averaged neighbor vector may be replaced with nearest neighbor vector.

[0082] Furthermore, the key vector extraction layer applied during database construction and the layer used to obtain the key vectors of query documents may be basically the same, but they may also be different layers. The processor may be configured to maintain a list of domain terms, which are characteristic nouns that appear in domain documents a predetermined number of times or more. Domain terms may be terms equivalent to technical terms or industry jargon. The processor may also be configured to change the layer from which the key vectors of query documents are extracted depending on whether or not the query documents contain domain terms.

[0083] For example, if a query document contains domain terms that are listed, it is highly likely that the query document itself is expressed in a way that reflects the characteristics / conventions of the domain. In other words, it is highly likely that the user belongs to / is associated with the domain. Therefore, if the query document contains domain terms, the layer from which the key vectors of the query document are extracted can be set to a relatively shallow layer, such as the first or second layer. On the other hand, if the query document does not contain domain terms, it is highly likely that the query document itself is expressed in a general way that is unrelated to the domain. In other words, it is highly likely that the user is someone who is not very familiar with the domain. Therefore, if the query document does not contain domain terms, the layer from which the key vectors of the query document are extracted can be set to a relatively deep layer, such as the ninth or tenth layer.

[0084] The processor may also dynamically change the value of the weight λ depending on whether or not the query document contains domain terms. For example, if the query document contains domain terms, λ may be smaller than if it does not. As can be seen from Equation 3, by making λ smaller, the component derived from the query document (e LM (d)) will have a significant impact.

[0085] The above describes an embodiment in which a domain document is adapted and stored in the document vector storage unit 18 as a data store, but it is not limited to this embodiment. The document vector stored in the document vector storage unit 18 may also be a value vector.

[0086] When the document vector stored in the document vector storage unit 18 is an adaptive vector of a domain document, the vector used for similar document searching in the search unit 17 does not necessarily have to be the adaptive vector of the query document. When the document vector stored in the document vector storage unit 18 is an adaptive vector of a domain document, the vector representation used for similar document searching may be the value vector of the query document as is.

[0087] The above describes an adaptation process performed on a document-by-document basis, but the data to be processed may be in token, sentence, or chunk units. A chunk may be a group of a predetermined number of sentences, or a group of text smaller than a predetermined maximum number of characters. If adaptation is performed on a token-by-token basis, the domain adaptor 10 may generate an adaptation vector corresponding to the document by averaging the adaptation vectors for each token. Similarly, if adaptation is performed on a sentence-by-sentence basis, the domain adaptor 10 may generate an adaptation vector corresponding to the document by averaging the adaptation vectors for each sentence contained in a single document.

[0088] Furthermore, language models used in the transformation unit 12, such as BERT, have an upper limit on the amount of data (actually the number of tokens) they can handle at once. Documents with a number of tokens exceeding this limit (hereinafter also referred to as large documents) may be split into multiple documents for processing. For example, the domain adaptor 10 may process large documents by treating the set of tokens up to the upper limit as one document, and the remainder as a separate document.

[0089] Furthermore, for documents with a token count exceeding the upper limit, the domain adaptor 10 may encode the document sentence by sentence and generate a vector representation of the document by averaging the vectors of each sentence. Also, for large documents, the domain adaptor 10 may be configured to first generate a summary sentence using a predetermined language model, and then treat the vector of this summary sentence as the vector of the document itself. Here, "vector" can refer to both the key vector and the value vector. The processing for large documents described here may also be applied to ordinary documents that are not large in scale.

[0090] Averaged neighborhood vector e DS The term "w" in equation 1 is used when calculating (d). j This may be determined by a more simplified formula, Equation 4, below.

[0091]

number

[0092] <Additional Note> The terms acquisition, determination, detection, generation, and calculation are interchangeable. Acquisition of data by a device includes generation of that data based on signals input from other devices / sensors. A method executed by the processor constituting the search device 1 corresponds to a vector generation method. This disclosure also includes a program containing instructions for causing a normal processor to function as the search device 1 or domain adaptor 10, and a recording medium on which the program is stored. [Explanation of symbols]

[0093] 1 Search device, 2 Input device, 3 Document database, 4 Display device, 10 Domain adapter, 11 Input unit (document acquisition unit), 12 Conversion unit, 13 Adaptation target data construction unit, 14 Adaptation target data storage unit, 15 Synthesis unit (vector generation unit), 16 kNN search unit (neighbor example acquisition unit), 17 Search unit, 18 Document vector storage unit

Claims

1. An information processing device that generates a vector representation of a document using a predetermined language model, A conversion unit (12) that acquires a key vector, which is the internal representation of the language model for the document, and a value vector, which is the output of the language model for the document, A target data storage unit (14) in which the key vector and value vector for each of the multiple domain documents that are documents of the domain to be applied are stored in association with each other, A document acquisition unit (11) acquires a target document which is the document to be used for generating the vector representation, A neighboring example acquisition unit (16) acquires information on neighboring examples of the target document from the aforementioned target data storage unit, The system includes a vector generation unit (15) that generates a vector representation of the target document using the value vectors of the neighboring examples acquired by the neighboring example acquisition unit, The conversion unit generates the key vector and value vector for the target document acquired by the document acquisition unit as the target key vector and target value vector, The neighboring example acquisition unit identifies the neighboring examples of the target document from the target data storage unit using a k-nearest neighbor search method with the key vector, and provides the value vector of the neighboring example to the vector generation unit. The vector generation unit is configured to generate an adaptive vector, which is the vector representation of the target document, by combining the value vector of the neighboring example and the target value vector.

2. A document vector storage unit (18) stores the adaptive vectors generated by the vector generation unit for the domain document, linked to the identification number or content of the domain document. The system includes a search unit (17) that searches for documents similar to the query document to be searched from among the domain documents stored in the document vector storage unit, The document acquisition unit is configured to acquire the query document, The vector generation unit generates an adaptive vector for the query document acquired by the document acquisition unit, The information processing apparatus according to claim 1, wherein the search unit is configured to search for domain documents similar to the query document using the adaptive vector.

3. The neighboring example acquisition unit acquires the value vectors of two or more neighboring examples, The aforementioned vector generation unit, An averaged neighbor vector is generated, which is a vector obtained by averaging the value vectors of two or more of the aforementioned neighboring examples. The information processing apparatus according to claim 2, configured to generate the adaptive vector by combining the target value vector and the averaged neighbor vector with predetermined weights.

4. The information processing apparatus according to claim 1 or 2, wherein the language model is a BERT model.

5. The aforementioned language model includes 12 or more transformer layers, The information processing apparatus according to claim 3, wherein the key vector is the output of the transformer layer located below the central layer of the plurality of transformer layers.

6. The aforementioned language model includes 12 or more transformer layers, The information processing apparatus according to claim 3, wherein the key vector is the output of the transformer layer located above the central layer of the plurality of transformer layers.

7. A vector generation method for generating a vector representation of a document using a predetermined language model, Obtaining the target document which is the document to be used for generating the aforementioned vector representation, The key vector, which is the internal representation of the language model for the target document, and the value vector, which is the output of the language model for the target document, are generated as the target key vector and the target value vector, From the target data storage unit (14), which stores the key vectors and value vectors for each of the multiple domain documents that are documents of the domain to be applied, in association with each other, neighboring examples of the target document are identified by a k-nearest neighbor search method using the key vectors, and the value vectors of the neighboring examples are obtained. A vector generation method comprising generating an adaptive vector, which is the vector representation of the target document, by combining the value vector of the neighboring example and the target value vector.

Images