Vector processing apparatus, method, and program

Abstract

To select a vector to be used together with an inputted vector in vector processing.SOLUTION: A vector processing device according to an embodiment of the present invention includes a vector storage unit in which a vector calculated from a vector sequence is stored, a first calculation unit that calculates a vector from an input vector sequence, a vector selection unit that selects a vector from the vector storage unit based on the vector calculated from the input vector sequence, and a second calculation unit that calculates a vector from a vector sequence including the vector calculated from the input vector sequence and the vector selected from the vector storage unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a vector processing device, a method, and a program. [Background technology]

[0002] Conventionally, there are known techniques for converting text, images, etc. into vectors for processing. For example, Retrieval-Augmented Generation (RAG) uses a combination of input vectors (e.g., consisting of word vectors of words included in a query) and vectors of documents retrieved by a document retrieval means. Summary of the Invention [Problem to be solved by the invention]

[0003] However, in the conventional method, vectors can only be obtained on a document-by-document basis by the document search means, and therefore it is not possible to select vectors to be used in combination with the input vector.

[0004] Therefore, an object of the present invention is to select a vector to be used together with an input vector in vector processing. [Means for solving the problem]

[0005] A vector processing device according to one embodiment of the present invention comprises a vector memory unit in which vectors calculated from a vector sequence are stored, a first calculation unit that calculates a vector from an input vector sequence, a vector selection unit that selects a vector from the vector memory unit based on the vector calculated from the input vector sequence, and a second calculation unit that calculates a vector from a vector sequence consisting of the vector calculated from the input vector sequence and the vector selected from the vector memory unit. [Effects of the Invention]

[0006] According to the present invention, in vector processing, it is possible to select a vector to be used together with an input vector. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an overall configuration according to an embodiment of the present invention. [Figure 2] FIG. 1 is a hardware diagram of a vector processing device according to an embodiment of the present invention. [Figure 3] FIG. 1 is a functional block diagram of a vector processing device according to an embodiment of the present invention. [Figure 4] FIG. 10 is a flow chart diagram illustrating vector processing according to an embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating an input / output buffer storing an input vector sequence according to an embodiment of the present invention. [Figure 6] FIG. 10 illustrates an input / output buffer in which a state vector is added to a sequence of input vectors according to an embodiment of the present invention. [Figure 7] FIG. 4 is a diagram showing state vectors stored in a vector storage unit according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating an input / output buffer in which a state vector selected from a vector storage unit is added after a state vector corresponding to an input vector sequence according to an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating vector processing according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] <Terminology> This specification describes vector processing of any vectors representing text, images, etc. Note that while this specification primarily describes vector processing of vectors representing text, the same applies to images. In this specification, a "vector sequence" is composed of multiple vectors. A vector sequence is arranged in an order based on the text, images, etc. represented by the vectors. For example, a vector sequence is composed of word vectors, which are obtained by converting words contained in text (e.g., a word sequence such as a document or question) into vectors. As used herein, a "state vector" is a vector that depends on the context of a sequence of vectors. As used herein, a "vector group" is a collection of vectors (i.e., one or more vectors).

[0010] <Overall structure> 1 is a diagram showing the overall configuration of an embodiment of the present invention. The vector processing device 10 is composed of one or more computers. For example, the vector processing device 10 is operated by an operator 11.

[0011] For example, when generating an answer to a question using a large-scale language model, the vector processing device 10 performs vector processing by searching for documents related to the question and using the documents in combination with the question (also known as RAG (Retrieval-Augmented Generation)).

[0012] <Hardware> FIG. 2 is a hardware diagram of the vector processing unit 10 according to one embodiment of the present invention.

[0013] As shown in Figure 2, the vector processing device 10 is constructed by a computer, and as shown in Figure 2, it is equipped with a CPU 1001, a ROM 1002, a RAM 1003, an HD 1004, an HDD (Hard Disk Drive) controller 1005, a display 1006, an external device connection I / F (Interface) 1007, a network I / F 1008, a data bus 1009, a keyboard 1010, a pointing device 1011, a DVD-RW (Digital Versatile Disk Rewritable) drive 1013, and a media I / F 1015.

[0014] Of these, the CPU 1001 controls the overall operation of the vector processing device 10. The ROM 1002 stores programs, such as an IPL, used to drive the CPU 1001. The RAM 1003 is used as a work area for the CPU 1001. The HD 1004 stores various data, such as programs. The HDD controller 1005 controls the reading and writing of various data from and to the HD 1004 under the control of the CPU 1001. The display 1006 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 1007 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memories and printers. The network I / F 1008 is an interface for data communication using a communication network. The bus line 1009 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 1001, shown in FIG. 2.

[0015] The keyboard 1010 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 1011 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 1013 controls reading and writing of various data from a DVD-RW 1012, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 1015 controls reading and writing (storing) of data from a recording medium 114, such as a flash memory.

[0016] <Function block> 3 is a functional block diagram of a vector processing device 10 according to one embodiment of the present invention. The vector processing device 10 can include a vector storage unit 100, an input unit 101, a first calculation unit 102, a vector selection unit 103, a second calculation unit 104, and an output unit 105. Furthermore, the vector processing device 10 can function as the input unit 101, the first calculation unit 102, the vector selection unit 103, the second calculation unit 104, and the output unit 105 by executing a program.

[0017] Vectors calculated from a vector sequence (specifically, state vectors of each vector constituting the vector sequence) are stored in the vector storage unit 100. For example, a vector sequence is composed of word vectors obtained by converting words contained in a document into vectors.

[0018] The input unit 101 receives an input vector sequence (hereinafter also referred to as an input vector sequence). For example, the vector sequence is made up of word vectors obtained by converting words included in a question into vectors.

[0019] The first calculation unit 102 calculates a vector from the input vector sequence received by the input unit 101 (specifically, the state vector of each vector constituting the input vector sequence).

[0020] The vector selection unit 103 selects a vector from the vector storage unit 100 based on the vector calculated from the input vector sequence by the first calculation unit 102. Specifically, the vector selection unit 103 selects a vector from the vector storage unit 100 based on the similarity between the vector calculated from the input vector sequence by the first calculation unit 102 and the vector stored in the vector storage unit 100. For example, the similarity is the dot product value between the vector calculated from the input vector sequence by the first calculation unit 102 and the vector stored in the vector storage unit 100.

[0021] [Selection method 1] For example, the vector selection unit 103 selects, from the vectors stored in the vector storage unit 100, a vector whose similarity to the vector calculated from the input vector sequence is greater than a predetermined threshold value.

[0022] [Selection method 2] For example, the vector selection unit 103 selects vectors stored in the vector storage unit 100 in descending order of similarity with the vector calculated from the input vector sequence, until a predetermined upper limit value for the number of vectors is reached.

[0023] The second calculation unit 104 calculates a vector (specifically, a state vector of each vector constituting the vector sequence) from a vector sequence composed of the vector calculated by the first calculation unit 102 from the input vector sequence and the vector selected by the vector selection unit 103 from the vector storage unit 100.

[0024] The output unit 105 outputs the vector calculated by the second calculation unit 104.

[0025] <Processing method> FIG. 4 is a flow chart diagram illustrating vector processing according to one embodiment of the present invention.

[0026] In step 1 (S1), the input unit 101 stores an input vector sequence in an input / output buffer provided in the vector processing device 10.

[0027] Figure 5 shows an input / output buffer storing input vector sequences e(1), e(2), and e(3). Here, each e(i) (i=1, 2, 3) is a vector with the same number of dimensions, and corresponds to an embedding vector assigned to a token (such as a word or a partial image) of text or an image.

[0028] In step 2 (S2), the first calculation unit 102 calculates state vectors h(1), h(2), and h(3) for each of e(1), e(2), and e(3) from the input vector sequence stored in the input / output buffer, and adds them to the input / output buffer.

[0029] 6 shows the input / output buffer to which state vectors h(1), h(2), and h(3) calculated by the first calculation unit 102 have been added. Here, the first calculation unit 102 calculates the state vectors according to, for example, Equation 1 and Equation 2. Note that the calculation of the state vectors is not limited to calculations using Equation 1 and Equation 2, and any method may be used as long as it can calculate a state vector that reflects the context of the vector sequence.

[0030]

number

[0031]

number

[0032] In Equation 1,<e(i),e(j)> is the dot product of the vectors, and exp is the exponential function (note that the dot product can be negative, so by taking the exponent, it is possible to make it always positive while reflecting the absolute value, so the exponential function is used). The value of N in the formula (the number of vectors included in the input vector sequence) is 3 in the above example. α(i,j) is a number between 0 and 1 that represents the relative similarity between vectors e(i) and e(j).

[0033] Using Equation 2, for each vector e(i) (i=1,2,3), the state vector h(i) is calculated by adding up the vectors e(j) (j=1,2,3), including itself, weighted by the relative similarity calculated using Equation 1.

[0034] In step 3 (S3), the vector selection unit 103 selects a group of vectors (that is, one or more vectors) from the vector storage unit 100 using the state vector in the input / output buffer as a key.

[0035] Here, it is assumed that a plurality of vectors calculated from a separately prepared vector sequence are stored in advance in the vector storage unit 100. The separately prepared vector sequence is, for example, a vector sequence in which embedded vectors of tokens in a plurality of separately prepared texts or images are arranged in order of appearance position, and for this vector sequence, it is assumed that state vectors t(i) (i=1, 2, 3, ...) calculated, for example, by the above-mentioned steps S1 and S2, are stored in the vector storage unit 100 as shown in the example of FIG. 7. Note that the calculation of the state vector is not limited to the calculation by the above-mentioned steps S1 and S2, and any method may be used as long as it can calculate a state vector that reflects the context of the vector sequence.

[0036] The vector selection unit 103 uses the first h(1) of the state vectors h(i) (i=1, 2, 3) in the input / output buffer as a key to calculate the inner product of the state vector t(i) (i=1, 2, 3, . . . ) in the vector storage unit 100.<h(1),t(i)> The vector selection unit 103 selects state vectors whose dot product value is greater than a predetermined threshold value S, and sets the selected state vectors as a vector group. Alternatively, the vector selection unit 103 selects state vectors in descending order of dot product value until it reaches a predetermined upper limit value K of the number of vectors, and sets the selected state vectors as a vector group.

[0037] Alternatively, a vector obtained by averaging h(i) (i=1,2,3) may be used as the key. Alternatively, instead of using a single state vector as the key, a group of vectors may be selected using each of h(i) (i=1,2,3) as a key, and the union of these selection results may be used as the selection result.

[0038] In step 4 (S4), the vector selection unit 103 adds the vector group selected in S3 to the input / output buffer, as shown in FIG.

[0039] 8, after the state vector h(i) (i=1, 2, 3) corresponding to the input vector sequence, the state vectors (for example, three state vectors) selected by the vector selection unit 103 from the vector storage unit 100 are added as h(4), h(5), and h(6). Note that h(4), h(5), and h(6) may be added in a predetermined order.

[0040] In step 5 (S5), the second calculation unit 104 calculates a vector O(i) (i=1, 2, ..., 6) (i.e., a state vector for each state vector) from the state vector sequence h(i) (i=1, 2, ..., 6) in the input / output buffer, for example, according to equations 3 and 4. Here, the value of M in the equations (the sum of the number of vectors included in the input vector sequence and the number of state vectors selected by the vector selection unit 103 from the vector storage unit 100) is 6 in the above example. Note that the calculation of the state vector is not limited to the calculation using equations 3 and 4, and any method may be used as long as it can calculate a state vector that reflects the context of the vector sequence.

[0041]

number

[0042]

number

[0043] That is, the relative similarity between state vectors h(i) (i=1, 2, ..., 6) is calculated using equation 3, and for each state vector h(i), the state vectors h(j) (j=1, 2, ..., 6) including the state vector h(i) are weighted by the relative similarity calculated using equation 3 and summed up to obtain vector O(i).

[0044] Thereafter, the output unit 105 outputs the leading vector O(1) of the vectors O(i) (i=1, 2, ..., 6). Alternatively, the output unit 105 outputs a vector obtained by averaging the vectors O(i) (i=1, 2, ..., 6) and combining them into one.

[0045] FIG. 9 is a diagram for explaining vector processing according to one embodiment of the present invention.

[0046] As a concrete example, we will show processing using the question: "Capital of Japan."

[0047] [Input Vector Sequence] The character string of the question "Capital of Japan" is decomposed into a word string "Capital of Japan", which is converted into a vector string shown as [Input Vector String] in Figure 9 and input.

[0048] In this example, the number of dimensions of the vector is 2 for the sake of explanation, but in reality it is 4096, etc. Also, the type of the numbers in the example is integer, but in reality it is floating-point.

[0049] The above vectors (abbreviated as word vectors) are learned in advance for each word and stored in a dictionary or the like, and these vectors can be assigned to an input word string using the word as a key.

[0050] [Step 1] The input unit 101 stores the input vector sequence in an input / output buffer provided in the vector processing device 10. The input vectors e(1), e(2), and e(3) in FIG. 5 correspond to (3, 5), (1, 6), and (2, 4), respectively, in [Step 1] in FIG. 9.

[0051] [Step 2] The first calculation unit 102 calculates state vectors h(1), h(2), and h(3) for each of the input vectors e(1), e(2), and e(3) stored in the input / output buffer using the above-described equations 1 and 2, and adds them to the input / output buffer ([Step 2] in Figure 9).

[0052] For example, the dot product<e(1),e(1)> is, <(3,5),(3,5)>=3*3+5*5=34, so exp(<e(1),e(1)> )=exp(34)=583461742527454.9. Similarly, exp(<e(1),e(2)> )=exp(33)=214643579785916.1 exp(<e(1),e(3)> )=exp(26)=195729609428.8.

[0053] Therefore, α(1,1)=exp(34) / (exp(34)+exp(33)+exp(26)=0.7309 α(1,2)=0.2689 α(1,3)=0.0002.

[0054] In this way, α represents the relative similarity between words. In other words, it represents the percentage of similarity relative to the whole by adding up and dividing the similarities between words (the positively corrected inner products). The sum of α is 1.0 (100%). The relative similarity between "Japan" and "Japan" is 0.7309 The relative similarity between "Nihon" and "no" is 0.2689 The relative similarity between "Japan" and "Capital" is 0.0002. From the above results, we can say that ("Nihon" and "Nihon") are more than twice as similar as ("Nihon" and "no").

[0055] The state vector h(1) corresponds to the sum of the word vectors weighted by α. In other words, it corresponds to a vector that takes into account the characteristics of the surrounding words according to the ratio indicated by α. When actually calculated, h(1)=α(1,1) e(1)+α(1,2) e(2)+α(1,3) e(3) =0.7309 e(1)+0.2689 e(2)+0.0002 e(3) =0.7309 (3,5) + 0.2689 (1,6) + 0.0002 (2,4) =(2.4620,5.2686)

[0056] In other words, the original word vector e(1) = (3, 5) is transformed into a value that reflects the context, the state vector h(1) = (2.4620, 5.2686), by taking into account the surrounding word vectors according to their similarity.

[0057] In this way, the state vector h(1) = (2.4620, 5.2686) reflects not only its own characteristics but also the characteristics (vectors) of the words around it, so as will be explained later, considering the similarity between state vectors is equivalent to considering the similarity between words including their context.

[0058] [Step 3] It is assumed that the vector storage unit 100 stores in advance a plurality of state vectors calculated from a separately prepared input vector sequence (t(1), t(2), . . . in FIG. 7).

[0059] Here, two separately prepared sentences, "Capital Tokyo Japan" and "LDP", are respectively converted into word strings "Capital Tokyo Japan" and "LDP", and the state vectors calculated by converting them into vector strings are stored in the vector storage unit 100 as shown in [Step 3] of Figure 9. Vector processing is performed for each sentence. Note that other sentences are also stored in the same way.

[0060] The vector selection unit 103 selects a group of vectors from the vector storage unit 100 using the state vectors in the input / output buffer as a key.

[0061] Here, the first state vector in the input / output buffer, h(1)=(2.5,5.3), is used as a key, and the inner product of this vector and t(i) (i=1,2,...) is first calculated. <h(1),t(1)> =<(2.5,5.3),(2.9975,4.9975)>=2.5*2.9975+5.3*4.9975=33.9807 <h(1),t(2)> =29.7593 <h(1),t(3)> =33.9973 <h(1),t(4)> =-17.8000 : This becomes:

[0062] Here, the threshold S (lower limit of the inner product) set for vector group selection is set to 10. Then, the vector groups selected are t(1), t(2), and t(3), and t(4) is excluded.

[0063] In other words, the state vector h(1) calculated by the first calculation unit 102 for the first word "Japan" in the question "Capital of Japan" also takes into account the context (capital of ...), so "capital," "Tokyo," and "Japan," which are in similar contexts, have a large similarity (inner product value) with each other, but "LDP" has a small inner product value even when the context (the empty string in this example) is taken into account, and is therefore unlikely to be selected.

[0064] In particular, the original word vector (-1,2) of "Tokyo" is significantly different from the word vector (3,5) of "Japan", but its state vector t(2) takes into account the context (capital...Japan), so its similarity (inner product value) with the state vector of "Japan" is high.

[0065] [Step 4] The vector selection unit 103 adds the vector group selected by the vector selection unit 103 to the input / output buffer as shown in step 4 of FIG. 9 (note that the values ​​are rounded down to two decimal places).

[0066] [Step 5] The second calculation unit 104 calculates a vector O(i) (i=1, 2, . . . , 6) from the state vector sequence h(i) (i=1, 2, . . . , 6) in the input / output buffer according to equations 3 and 4.

[0067] The resulting vector O(i) (i=1,2,...,6) is as follows: O(1)=(2.0371, 5.4661) O(2)=(1.2219, 5.8893) O(3)=(1.7179, 5.6352) O(4)=(2.3114, 5.3131) O(5)=(2.1771, 5.3840) O(6)=(2.3114, 5.3131)

[0068] The output unit 105 outputs the leading vector O(1) of the vectors O(i) (i=1, 2, ..., 6). Alternatively, the output unit 105 outputs a vector obtained by averaging the vectors O(i) (i=1, 2, ..., 6) and combining them into one.

[0069] For example, O(1) is a state vector that takes into account the first word "Japan" in the question "Capital of Japan," the context "Capital of...," and the context "Capital Tokyo Japan" formed by the group of vectors selected from the vector storage unit 100, so this state vector includes the vector component of "Tokyo."

[0070] Then, in the subsequent answer generation process, the vector component of the answer "Tokyo" is extracted from the input O(1), and the corresponding word "Tokyo" is assigned to it and output as the answer.

[0071] In one embodiment of the present invention, vectors can be selected, which reduces the chance of excess or deficiency of information being added compared to conventional cases where adjustments can only be made on a document-by-document basis, thereby improving the performance of subsequent processing.

[0072] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above. [Explanation of symbols]

[0073] 10 Vector Processing Unit 11 Operator 100 Vector storage unit 101 Input section 102 1st calculation section 103 Vector Selection Section 104 2nd calculation section 105 Output section [Prior art documents] [Non-patent literature]

[0074] [Non-Patent Document 1] "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", arXiv:2005.11401

Claims

1. a vector storage unit in which a vector calculated from a vector sequence is stored; a first calculation unit that calculates a vector from an input vector sequence; a vector selection unit that selects a vector from the vector storage unit based on a vector calculated from the input vector sequence; a second calculation unit that calculates a vector from a vector sequence that is composed of a vector calculated from the input vector sequence and a vector selected from the vector storage unit; A vector processing unit comprising:

2. The vector processing device according to claim 1 , wherein the vector is a vector that depends on the context of a vector sequence.

3. The vector processing device according to claim 1 , wherein the vector selection unit selects a vector from the vector storage unit based on a similarity between a vector calculated from the input vector sequence and a vector stored in the vector storage unit.

4. The vector processing device according to claim 3 , wherein the vector selection unit selects, from the vectors stored in the vector storage unit, a vector whose similarity to the vector calculated from the input vector sequence is greater than a predetermined threshold value.

5. 4. The vector processing device according to claim 3, wherein the vector selection unit selects vectors stored in the vector storage unit in descending order of the degree of similarity with the vector calculated from the input vector sequence, until a predetermined upper limit value for the number of vectors is reached.

6. The vector processing device according to claim 3 , wherein the similarity is an inner product value of a vector calculated from the input vector sequence and a vector stored in the vector storage unit.

7. an output unit that outputs a vector calculated from a vector sequence composed of a vector calculated from the input vector sequence and a vector selected from the vector storage unit; The vector processing device of claim 1 further comprising:

8. The vector processing device of claim 1 , wherein the vectors represent text or images.

9. A method executed by a vector processing device having a vector storage unit in which a vector calculated from a vector sequence is stored, comprising: Computing a vector from a sequence of input vectors; selecting a vector from the vector storage unit based on a vector calculated from the input vector sequence; calculating a vector from a vector sequence composed of a vector calculated from the input vector sequence and a vector selected from the vector storage unit; A method comprising:

10. a vector processing device including a vector storage unit in which a vector calculated from a vector sequence is stored, a first calculation unit that calculates a vector from an input vector sequence; a vector selection unit that selects a vector from the vector storage unit based on a vector calculated from the input vector sequence; a second calculation unit that calculates a vector from a vector sequence that is composed of a vector calculated from the input vector sequence and a vector selected from the vector storage unit; A program to function as a

Images