Information processing device, information processing system, information processing method, and program

Abstract

To enable the output of an information vectorization model, which converts non-text information into vectors and outputs it without generating a new adapter, to be converted into a format that can be processed by a new large-scale language model. [Solution] The information processing device includes: a first conversion unit that converts the output of an information vectorization model, which converts non-text information into vectors and outputs it, into a form that can be processed by a first large-scale language model; a second conversion unit that converts the output of the first conversion unit into a form that can be processed by a second large-scale language model; a processing request unit that outputs the output of the second conversion unit to the second large-scale language model when the second conversion unit has converted the output of the first conversion unit; and an output unit that outputs a response based on the processing result of the second large-scale language model.

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and a program. [Background Art]

[0002] The following Non-Patent Document 1 discloses a technique of encoding an image with an image encoder and processing the encoded image with a text encoder.

[0003] Further, it is known that a conventional model for vectorizing non-text information (e.g., a Vision Encoder that vectorizes images) and a large language model (LLM) are connected via an adapter layer, and an LMM (Large Multi-Modal Model) configured such that a vector output from the model for vectorizing non-text information is converted by the adapter layer into a form processable by the large language model. [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] In conventional LMMs, when the combination of the model for vectorizing non-text information and the large language model is changed, it is necessary to newly generate an adapter layer corresponding to the new combination of the model for vectorizing non-text information and the LLM. Generating the adapter layer requires a large amount of training, which is costly.

[0005] In order to solve the above-mentioned problem of the prior art, an object of the present invention is to make an adapter layer of a previous combination usable in a new combination of a model for vectorizing non-text information and a large language model. [Means for Solving the Problem]

[0006] To solve the above-mentioned problems, an information processing device according to one embodiment includes: a first conversion unit that converts the output of an information vectorization model, which converts non-text information into vectors and outputs it, into a form that can be processed by a first large-scale language model; a second conversion unit that converts the output of the first conversion unit into a form that can be processed by a second large-scale language model; a processing request unit that outputs the output of the second conversion unit to the second large-scale language model when the second conversion unit has converted the output of the first conversion unit; and an output unit that outputs a response based on the processing result of the second large-scale language model. [Effects of the Invention]

[0007] According to an information processing device according to one embodiment, the connection layer of the previous combination can be used in a new combination of a model that vectorizes non-text information and a large-scale language model. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the system configuration of an information processing system according to one embodiment. [Figure 2] Hardware configuration diagram of a server and PC according to one embodiment. [Figure 3] Functional block diagram of a server according to one embodiment [Figure 4] A conceptual diagram showing the procedure of the conversion process by the server according to one embodiment. [Figure 5] A diagram showing a method for generating text vectors using the first LLM and second LLM according to one embodiment. [Figure 6] A figure showing an example of a vector space generated by the first LLM and second LLM according to one embodiment. [Figure 7] This figure shows an example of a method for determining the word vector transformation relationship between the first LLM and the second LLM using a server according to one embodiment. [Figure 8] This figure shows the relationship between the vector space of the image vector and token and the transformation layer in this embodiment. [Figure 9]This figure shows an example of a method for generating a conversion layer by the conversion layer generation unit of a server according to one embodiment. [Figure 10] Figure showing a comparative example between the method described herein and conventional methods. [Figure 11] Functional block diagram of a server and PC according to one embodiment. [Figure 12] This figure shows an example of a data table stored in the storage unit of a server according to one embodiment. [Figure 13] A flowchart illustrating an example of the procedure for AI chat generation by a server and PC according to one embodiment. [Figure 14] This figure shows an example of the generated AI chat screen displayed on a PC according to one embodiment. [Figure 15] This figure shows an example of the generated AI chat screen displayed on a PC according to one embodiment. [Figure 16] This figure shows an example of the generated AI chat screen displayed on a PC according to one embodiment. [Modes for carrying out the invention]

[0009] One embodiment of the present invention will be described below with reference to the drawings.

[0010] (An example of the system configuration of information processing system 10) Figure 1 shows an example of the system configuration of an information processing system 10 according to one embodiment.

[0011] The information processing system 10 shown in Figure 1 is an example of an "information processing system." The information processing system 10 includes a server 12 and a PC 14. The server 12 and the PC 14 are connected to each other via a communication network 16 so that they can communicate with each other. The communication network 16 can be, for example, a LAN or the Internet.

[0012] PC14 is an example of a terminal device. PC14 comprises an input device for inputting information other than character strings and character strings. Examples of the input device include a keyboard, a touch panel, and the like. PC14 can transmit information other than character strings and character strings input by a user using the input device to server 12 via communication network 16. Note that PC14 may have a speech recognition function for recognizing speech input by a user from a microphone, and in this case, the character string recognized by the speech recognition function may be transmitted to server 12.

[0013] Server 12 is an example of an "information processing apparatus". Server 12 receives information other than character strings and character strings transmitted from PC14 via communication network 16, and performs predetermined processing by an LLM (for example, character string division processing, prediction processing, etc.), thereby generating text according to the received information other than character strings and the received character strings. At this time, for information other than character strings, server 12 converts the information other than character strings into a form processable by the LLM using a model that vectorizes information other than character strings and an adapter layer. The adapter layer is a layer that converts an output of the model that vectorizes information other than character strings into a form that can be input to the LLM. For example, if the information other than a character string is an image, the information is converted into a form processable by the LLM using a Vision Encoder and the adapter layer. Then, server 12 can output text obtained as a processing result by the LLM to PC14 via communication network 16.

[0014] That is, the server 12 functions as an LMM (Large Multi-Modal Model). The LMM is a model that simultaneously processes information other than character strings such as images and character strings as input to generate text. The LMM is configured to include a model that vectorizes information other than character strings (which may also be expressed as submodels or modules), an LLM, and an adapter layer. The LMM may include a plurality of models for vectorizing information other than character strings. In the embodiments described below, the description is given using images as information transmitted from the PC 14 and information other than character strings processed by the LMM, but the present invention is not limited thereto. For example, the information may be sound or the like. In the embodiments described below, the LMM is configured to include a Vision Encoder, an LLM, and an adapter layer. The Vision Encoder vectorizes an image. The adapter layer converts the image vector output from the Vision Encoder into a form processable by the LLM (that is, a new image vector). Specifically, the adapter layer matches the number of dimensions of the image vector output from the Vision Encoder to the number of dimensions of a text vector processed by the LLM. The LLM generates text corresponding to the input image and character string by performing predetermined processing (for example, character string division processing, prediction processing, etc.) on the new image vector output from the adapter layer.

[0015] The PC 14 can receive text transmitted from the server 12 (that is, text obtained as a processing result by the LLM) via the communication network 16, and perform various processes using the text (for example, screen display, etc.).

[0016] Note that in the example shown in Fig. 1, one terminal device (PC 14) is connected to the server 12 via the communication network 16. In practice, however, in the information processing system 10, a plurality of terminal devices can be connected to the server 12 via the communication network 16, and an image and a character string can be transmitted to the server 12 from each of the plurality of terminal devices.

[0017] Furthermore, while the example shown in Figure 1 uses a PC14 as an example of a terminal device for inputting images and text, the terminal device is not limited to this. Any other device capable of inputting and transmitting images and text (for example, a smartphone, tablet, electronic whiteboard, inkjet printer, MFP, video conferencing terminal, projector, 360-degree camera, etc.) may be used.

[0018] (Hardware configuration of Server 12 and PC 14) Figure 2 is a hardware configuration diagram of server 12 and PC 14 according to one embodiment. Here, we will describe the hardware configuration common to server 12 and PC 14.

[0019] As shown in Figure 2, the server 12 and PC 14 are built by a computer and include a CPU 501, ROM 502, RAM 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, data bus 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0020] The CPU 501 controls the operation of the entire server 12 or PC 14. The ROM 502 stores programs used to drive the CPU 501, such as IPLs. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using the communication network 16. The data bus 510 is an address bus and data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 2.

[0021] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0022] (Functional configuration of Server 12) Figure 3 is a functional block diagram of server 12 according to one embodiment. Figure 3 shows several functions of server 12 that relate to the process of generating text corresponding to input images and strings using the first LLM 111 and the second LLM 112.

[0023] As shown in Figure 3, the server 12 comprises an acquisition unit 101, an image vectorization unit 102 (an example of an "information vectorization unit"), a first conversion unit 103, a processing request unit 104, a second conversion unit 105, an output unit 106, a conversion layer generation unit 107, a first LLM 111 (an example of a "first large-scale language model"), and a second LLM 112 (an example of a "second large-scale language model").

[0024] The acquisition unit 101 acquires images and text from the PC 14 via the communication network 16. The images acquired by the acquisition unit 101 are output to the image vectorization unit 102. The text acquired by the acquisition unit 101 is output to the processing request unit 104.

[0025] The image vectorization unit 102 vectorizes the image acquired by the acquisition unit 101 using the Vision Encoder 102A (an example of an "information vectorization model that converts non-text information into vectors and outputs it" and an "image vectorization model that converts images into vectors and outputs them"). The image vectorization model is a model that vectorizes an input image. In other words, when an image is input to the image vectorization model, it outputs a vector corresponding to the image.

[0026] The first conversion unit 103 uses an adapter 103A (adapter layer) corresponding to the combination of Vision Encoder 102A and the first LLM 111 to convert the output of the image vectorization unit 102 into a format that can be processed by the first LLM 111.

[0027] The processing request unit 104 outputs the string acquired by the acquisition unit 101 and the output of the first conversion unit 103 to the first LLM 111, thereby requesting the first LLM 111 to perform a predetermined process (for example, string splitting, prediction, etc.). The processing request unit 104 then obtains text corresponding to the image and string acquired by the acquisition unit 101 from the first LLM 111 as the processing result of the predetermined process.

[0028] When the output destination of the processing request unit 104 is switched to the second LLM 112, the second conversion unit 105 uses the conversion layer 105A to convert the output of the first conversion unit 103 into a format that can be processed by the second LLM 112. In this case, the processing request unit 104 outputs the string acquired by the acquisition unit 101 and the output of the second conversion unit 105 to the second LLM 112, thereby requesting the second LLM 112 to perform a predetermined process (for example, string splitting, prediction, etc.). The processing request unit 104 then obtains text corresponding to the image and string acquired by the acquisition unit 101 from the second LLM 112 as the processing result of the predetermined process.

[0029] In particular, in this embodiment, the second transformation unit 105 can use the transformation layer 105A to linearly transform the output of the first transformation unit 103 into a form that can be processed by the second LLM 112 (i.e., an even newer image vector).

[0030] The output unit 106 receives the processing result of a predetermined process performed by the first LLM 111 or the second LLM 112 (for example, text corresponding to the image and string acquired by the acquisition unit 101) from the processing request unit 104, and outputs a response based on the processing result to the PC 14 via the communication network 16. The response may be generated by editing and modifying the LLM output, or the LLM output may be used as is as the response. The response is not limited to text; it may also be audio or an image.

[0031] The conversion layer generation unit 107 generates a conversion layer 105A for converting the output of the first conversion unit 103 into a format that can be processed by the second LLM 112.

[0032] The first LLM111 and the second LLM112 perform predetermined processing (for example, string splitting processing, prediction processing, etc.) in response to processing requests from the processing request unit 104.

[0033] For example, the first LLM111 and the second LLM112 divide images and strings into multiple vocabulary (i.e., multiple morphemes) by string splitting processing. Specifically, the first LLM111 and the second LLM112 have a tokenizer. The tokenizer has a vocabulary group. The vocabulary group has multiple vocabulary to which identifiers are associated. The vocabulary group is implemented, for example, by a JSON file. Based on the vocabulary group, the tokenizer divides the images and strings output from the processing request unit 104 into multiple vocabulary (i.e., multiple morphemes) by performing string splitting processing (i.e., tokenization processing) on ​​the images and strings using well-known techniques.

[0034] Furthermore, for example, the first LLM111 and the second LLM112 perform prediction processing on the results of the string splitting process (i.e., multiple vocabulary words) to generate text corresponding to the image and string acquired by the acquisition unit 101. The first LLM111 and the second LLM112 then output the text generated by the prediction processing to the processing request unit 104.

[0035] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0036] (Procedure for conversion processing by Server 12) Figure 4 is a conceptual diagram showing the procedure of the conversion process performed by the server 12 according to one embodiment. As shown in Figure 4, the server 12 executes Step 1, Step 2, and Step 3 in order as the conversion process. Figure 4 shows the conversion process that the server 12 executes after the LLM that performs predetermined processing (e.g., string splitting processing, prediction processing, etc.) is switched to the second LLM 112. Figure 4 also shows the conversion process for an image acquired from the PC 14.

[0037] In Step 1, the acquisition unit 101 acquires an image from the PC 14 (acquisition step). Then, the image vectorization unit 102 uses the Vision Encoder 102A to vectorize the image acquired by the acquisition unit 101 (image vectorization step).

[0038] In Step 2, the first conversion unit 103 uses an adapter 103A corresponding to the combination of the Vision Encoder 102A and the first LLM 111 to convert the output of the image vectorization unit 102 (i.e., the image vector) into a form that can be processed by the first LLM 111 (i.e., a new image vector) (first conversion step).

[0039] In Step 3, the second transformation unit 105 uses the transformation layer 105A to transform the output of the first transformation unit 103 (i.e., a new image vector) into a form that can be processed by the second LLM 112 (i.e., yet another new image vector) (second transformation step). In particular, in this embodiment, the second transformation unit 105 uses the transformation layer 105A to transform the output of the first transformation unit 103 (i.e., a new image vector) into a form that can be processed by the second LLM 112 (i.e., yet another new image vector) by performing a linear transformation.

[0040] Then, the processing request unit 104 outputs the output of the second transformation unit 105 (i.e., yet another new image vector) to the second LLM 112, thereby requesting the second LLM 112 to perform a predetermined process (for example, string splitting, prediction, etc.) (processing request step).

[0041] As described above, in one embodiment, the server 12 uses the second conversion unit 105 and the conversion layer 105A to convert the output of the first conversion unit 103 into a form that can be processed by the second LLM 112.

[0042] As a result, even if the large-scale language model in the LMM is changed to the second LLM112, the server 12 according to one embodiment can convert the image vector output from the Vision Encoder 102A (image vectorization unit) into a form that can be processed by the second LLM112 simply by adding the conversion layer 105A to the adapter 103A, without generating a new adapter.

[0043] (Method for generating text vectors using LLM) Figure 5 shows a method for generating text vectors using the first LLM111 and the second LLM112 according to one embodiment. The method for generating text vectors common to the first LLM111 and the second LLM112 will be described below.

[0044] As shown in Figure 5, the LLM has a Token Embedded layer and a tokenizer. The Token Embedded layer stores vectors corresponding to each of the identifiers that represent the multiple vocabulary items in the LLM.

[0045] The tokenizer breaks down the input string into multiple tokens (vocabulary) and outputs an identifier for each token, representing the vocabulary. The Token Embedded layer outputs a vector corresponding to the identifier output by the tokenizer. Therefore, LLM can obtain a vector representation of the input string by utilizing the Token Embedded layer and the tokenizer.

[0046] For example, in the example shown in Figure 5, as shown in the left column of Figure 5, the input string "red apple" is broken down by the tokenizer into token "red" and token "apple".

[0047] Then, as shown in the right column of Figure 5, the tokenizer obtains the vector for token "red" and the vector for token "apple" from the token-embedded layer.

[0048] Furthermore, as shown in the right column of Figure 5, the tokenizer obtains a vector representation (word vector) for the input string "red apple" by adding the vectors of token "red" and token "apple" and dividing by 2. Note that the vectors before division by 2 are shown in the right column of Figure 5.

[0049] (An example of a vector space generated by LLM) Figure 6 shows an example of a vector space generated by the first LLM111 and the second LLM112 according to one embodiment.

[0050] As shown in Figure 6, the vector spaces generated by the Token embedded layer differ between the first LLM111 and the second LLM112. For example, in the example shown in Figure 6, the directions of the three basis vectors (basis 1, basis 2, basis 3) of the vector space differ between the first LLM111 and the second LLM112. Therefore, in the example shown in Figure 6, the directions of the vector for token "red" and the vector for token "apple" differ between the first LLM111 and the second LLM112. A basis vector is a set of vectors that define the axis of a vector space, and a vector in this vector space can be represented as a linear combination of the basis vectors.

[0051] However, if the LLM is ideally trained, the vector relationship between two tokens can be considered constant, regardless of the combination of the two tokens, and therefore can be expressed by a constant linear transformation relationship.

[0052] From this, the inventors of the present invention have found that word vectors that can be processed by the first LLM 111 before modification (i.e., the output of the first transformation unit 103) can be converted into word vectors that can be processed by the modified second LLM 112 by linear transformation by the transformation layer 105A of the second transformation unit 105.

[0053] (An example of a method for determining the word vector transformation relationship between two LLMs) Figure 7 shows an example of a method used by a server 12 according to one embodiment to determine the transformation relationship between word vectors between the first LLM 111 and the second LLM 112.

[0054] First, as shown in the left column of Figure 7, server 12 inputs a common string into the first LLM111 and the second LLM112, causing the tokenizer and Token Embedded to generate word vectors in each of them.

[0055] Next, as shown in the right column of Figure 7, the server 12 determines the transformation relationship between the word vector generated by the first LLM 111 and the word vector generated by the second LLM 112 through a predetermined calculation process. A linear transformation matrix is ​​used as the transformation relationship to be determined here.

[0056] Next, server 12 repeatedly performs the word vector generation process and the transformation relation calculation process described above, while changing the strings input to the first LLM111 and the second LLM112, until the loss of the transformation relation (linear transformation matrix) converges. Here, server 12 may calculate the convergence of the loss using an existing method such as the steepest descent method.

[0057] Then, when the loss of the transformation relation (linear transformation matrix) converges in the above iterative process, server 12 terminates the series of processes and decides that the last obtained transformation relation (linear transformation matrix) is the formal transformation relation (linear transformation matrix).

[0058] (A diagram showing the relationship between the vector space of the image vector and token, and the transformation layer.) Figure 8 shows the relationship between the image vector, the token vector space, and the transformation layer in this embodiment. As shown in Figure 8 (first column from the left), the image vector output from the Vision Encoder is oriented to the first LLM111 (LLM A in the figure), and after passing through Adapter A (adapter 103A), it is represented in the token vector space generated by the Token Embedded of the first LLM111.

[0059] However, since Adapter A is designed for the combination of Vision Encoder and the first LLM111, if the large-scale language model used is switched from the first LLM111 to the second LLM112 (LLM B in the diagram), the conventional method would require generating a new Adapter B that corresponds to the combination of Vision Encoder and the first LLM111, at enormous cost.

[0060] Therefore, in this embodiment, as shown in Figure 8 (first column from the right), a transformation layer (transformation layer 105A) is provided between Adapter A and the second LLM 112. This transformation layer converts the output of Adapter A into a form that can be processed by the second LLM 112 without generating a new Adapter B. The transformation relationship of this transformation layer (transformation layer 105A) can be determined by the method shown in Figure 7 based on the geometric relationship of the vector space. This transformation layer (transformation layer 105A) performs a linear transformation that represents the basis relationship between Token Embedded A, which is present in the first LLM 111, and Token Embedded B, which is present in the second LLM 112.

[0061] (An example of how to generate conversion layer 105A) Figure 9 shows an example of a method for generating a conversion layer 105A by the conversion layer generation unit 107 of the server 12 according to one embodiment.

[0062] In Figure 9, Token Embedded A and Tokenizer A belong to the first LLM111. Token Embedded B and Tokenizer B belong to the second LLM112.

[0063] Furthermore, Token Embedded A and Tokenizer A are examples of the "first text vectorization model." Similarly, Token Embedded B and Tokenizer B are examples of the "second text vectorization model." A text vectorization model is a model that outputs a vector corresponding to the input text (string).

[0064] First, the conversion layer generation unit 107 prepares several words (Step 1).

[0065] Next, the conversion layer generation unit 107 inputs a common word from the multiple words prepared in step 1 into each of the first LLM 111 and the second LLM 112, causing each of the first LLM 111 and the second LLM 112 to generate token vectors for two vocabulary words obtained by string splitting that single word (step 2).

[0066] In this process, LLM111 1 divides one word into two vocabulary words. Then, Token Embedded B generates token vectors for the two vocabulary words.

[0067] Furthermore, in the second LLM112, Tokenizer A splits one word into two vocabulary words. Then, Token Embedded A generates token vectors for the two vocabulary words.

[0068] Next, the conversion layer generation unit 107 causes each of the first LLM 111 and the second LLM 112 to generate word vectors from the token vectors of the two vocabulary words generated in step 2 (step 3). Here, each of the first LLM 111 and the second LLM 112 generates word vectors by averaging the token vectors of the two vocabulary words.

[0069] Then, the transformation layer generation unit 107 determines the transformation relationship (linear transformation matrix) between the word vector generated by the first LLM 111 (an example of the "first word vector") and the word vector generated by the second LLM 112 (an example of the "second word vector") through a predetermined calculation process (step 4). For example, the transformation layer generation unit 107 finds the linear transformation matrix that minimizes the absolute value (loss) of the difference between the vector obtained by linearly transforming the first word vector and the second word vector.

[0070] Next, the transformation layer generation unit 107 repeatedly executes steps 2 to 4 (step 5) while changing the word input to the first LLM 111 and the second LLM 112 until the loss of the transformation relationship (linear transformation matrix) calculated by step 4 converges.

[0071] Then, when the loss of the transformation relationship (linear transformation matrix) in step 5 above converges, the transformation layer generation unit 107 terminates the series of processes shown in Figure 9 and determines the last obtained transformation relationship (linear transformation matrix) (W and b in the formula shown in Figure 9) as the transformation layer 105A.

[0072] The conversion layer 105A generated by the conversion layer generation unit 107 is used by the second conversion unit 105 to convert the output of the first conversion unit 103 into a format that can be processed by the second LLM 112.

[0073] As described above, in one embodiment, the server 12 converts the output of the first conversion unit 103 into a form that can be processed by the second LLM 112 by linearly transforming it using the conversion layer 105A with the second conversion unit 105.

[0074] As a result, in one embodiment, the server 12 can use the adapter 103A of the first conversion unit 103 without changing it, and by linearly transforming the output of the first conversion unit 103 with the conversion layer 105A, it can convert the output of the first conversion unit 103 into a form that can be processed by the second LLM 112 with relatively simple calculation processing.

[0075] Furthermore, in one embodiment, the server 12 generates a conversion layer 105A using a conversion layer generation unit 107 based on the relationship between the base of a first vector obtained by a first text vectorization model for vectorizing text, which is included in the first LLM 111, and the base of a second vector obtained by a second text vectorization model for vectorizing text, which is included in the second LLM 112.

[0076] As a result, the server 12 according to one embodiment can generate a transformation layer 105A that can accurately absorb the difference between the base of the first vector and the base of the first vector by a linear transformation.

[0077] In particular, in one embodiment, the server 12 generates a transformation layer 105A for linearly transforming the first word vector into the second word vector, based on a first word vector generated by averaging a plurality of vectors obtained by inputting a string into the first LLM 111 and a second word vector generated by averaging a plurality of vectors obtained by inputting a string into the second LLM 112, using a transformation layer generation unit 107.

[0078] As a result, the server 12 according to one embodiment can generate a transformation layer 105A that can absorb the difference between the first word vector and the second word vector with high accuracy by linear transformation.

[0079] The conversion layer generation unit 107 may generate the conversion layer 105A by a method other than the method based on the first word vector and the second word vector.

[0080] Furthermore, the conversion layer generation unit 107 may generate the conversion layer 105A by a method other than the method based on the base of the first vector and the base of the second vector.

[0081] Furthermore, the conversion layer 105A is not limited to linearly converting the output of the first conversion unit 103.

[0082] (Comparative example) Figure 10 shows a comparison between the method according to this embodiment and a conventional method. Figure 10 shows an example in which the LLM that performs a predetermined process (for example, string splitting, prediction, etc.) is changed from "LLM A" to "LLM B".

[0083] As shown in Figure 10, in conventional methods, if the LLM performing a predetermined process is changed from "LLM A" to "LLM B", it is necessary to generate a new adapter "Adapter B" that corresponds to the combination of Vision Encoder and "LLM B", instead of the adapter "Adapter A" that corresponds to the combination of Vision Encoder and "LLM A". In this case, conventional methods require extensive training of the newly generated adapter "Adapter B" so that it can adapt to "LLM B".

[0084] On the other hand, as shown in Figure 10, in the method of this embodiment, if the LLM performing a predetermined process is changed to "LLM B", the adapter "Adapter A" corresponding to "LLM A" is used as is, and only a conversion layer is added to convert the output of "Adapter A" into a format usable by "LLM B". In this case, since "Adapter A" is already trained in the method of this embodiment, only a small amount of training is required for the conversion layer to adapt to "LLM B".

[0085] (Examples) The following describes an embodiment that realizes a generated AI chat using the conversion layer (the conversion layer generated by the conversion layer generation unit 107) described in the above embodiment.

[0086] (Functional configuration of Server 12 and PC 14) Figure 11 is a functional block diagram of server 12 and PC 14 according to one embodiment. Figure 11 differs from Figure 3 in that multiple functions related to AI chat generation processing have been added to server 12 and PC 14.

[0087] In the example shown in Figure 11, the PC 14 includes an input receiving unit 121, a communication unit 122, and a display unit 123.

[0088] The input receiving unit 121 receives input from the user on the UI screen displayed on the display unit 123. Specifically, the input receiving unit 121 receives the selection of a combination of the vision encoder and the LLM model, and the input of images and text to the chat (an example of "input text").

[0089] The communication unit 122 communicates with the server 12. For example, the communication unit 122 transmits user input received by the input reception unit 121 to the server 12. Also, for example, the communication unit 122 receives screen information from the server 12 for displaying the UI screen.

[0090] The display unit 123 displays a UI screen based on the screen information received by the communication unit 122.

[0091] In the example shown in Figure 11, the server 12 includes a storage unit 131, a communication unit 132, a specification unit 133, a response generation unit 134, and a display control unit 135.

[0092] The storage unit 131 stores the vision encoder, LLM, adapter layer, and conversion layer in association, as shown in the data table illustrated in Figure 12. The storage unit 131 may also store an LMM identifier that identifies the combination of the vision encoder, LLM, adapter layer, and conversion layer.

[0093] The communication unit 132 communicates with the PC 14. For example, the communication unit 132 receives user input from the PC 14. Also, for example, the communication unit 132 transmits screen information for displaying the UI screen to the PC 14 under the control of the display control unit 135.

[0094] The identification unit 133 refers to the storage unit 131 and identifies the adapter layer and conversion layer that are associated with the combination of vision encoder and LLM models selected by the user in the PC 14.

[0095] The response generation unit 134 uses the vision encoder and LLM selected by the user on the PC 14, and the adapter layer and conversion layer identified by the identification unit 133, to input the image and text entered by the user into the LLM, and generates a response sentence from the output of the LLM. The response generation unit 134 may edit and modify the output of the LLM to generate the response sentence, or it may use the output of the LLM as is as the response sentence.

[0096] The display control unit 135 controls the screen display on the PC 14. Specifically, the display control unit 135 controls the screen display on the PC 14 by transmitting screen information for displaying the UI screen to the PC 14 via the communication unit 132. For example, the display control unit 135 displays the generated AI chat screen, which will be described later, on the PC 14.

[0097] (An example of a data table stored in the memory unit 131) Figure 12 shows an example of a data table stored in the storage unit 131 of a server 12 according to one embodiment.

[0098] As shown in Figure 12, the storage unit 131 of the server 12 stores the vision encoder, LLM, adapter layer, and conversion layer in association with each other using a data table.

[0099] For example, an LMM called "LMM-1" is an LLM that has a vision encoder "vision-A" and an LLM "LLM-A", and the vision encoder "vision-A" and the LLM "LLM-A" are connected by an adapter layer "VA-LA".

[0100] Furthermore, the LMM referred to as "LMM-2" is an LLM that has a vision encoder "vision-A" and an LLM "LLM-B", and the vision encoder "vision-A" and the LLM "LLM-B" are connected by an adapter layer "VA-LA" and a conversion layer "A-AB".

[0101] Furthermore, an LMM may have multiple conversion layers, such as an LMM named "LMM-3".

[0102] (An example of the procedure for AI chat generation by Server 12 and PC 14) Figure 13 is a flowchart showing an example of the procedure for generating AI chat using server 12 and PC 14 according to one embodiment.

[0103] First, in server 12, the display control unit 135 transmits the screen information of the generated AI chat screen to PC 14 via the communication unit 132, thereby displaying the generated AI chat screen on PC 14 (step S301). At this time, in PC 14, the communication unit 122 receives the screen information of the generated AI chat screen, and the display unit 123 displays the generated AI chat screen.

[0104] Next, in PC14, the input receiving unit 121 accepts the user's selection of a vision encoder and an LLM model on the generated AI chat screen (step S302). The input receiving unit 121 also accepts chat input from the user on the generated AI chat screen (step S303).

[0105] Then, in PC14, the communication unit 122 sends the selection of the vision encoder and LLM model received in step S302, and the chat input received in step S303, to the server 12 (step S304).

[0106] Next, in the server 12, the identification unit 133 refers to the storage unit 131 and identifies the adapter layer and conversion layer corresponding to the combination of vision encoder and LMM selected by the user (step S305).

[0107] Next, in server 12, the response generation unit 134 generates a response statement for the image and text input by the user using an LMM having a vision encoder and LLM selected by the user, and an adapter layer and a conversion layer identified by the identification unit 133 (step S306). Here, the response generation unit 134 may use the output from the LMM as the response statement as is, or it may generate a response statement based on the output from the LMM.

[0108] Next, in server 12, the display control unit 135 transmits the screen information of the generated AI chat screen, including the response sentence generated in step S306, to PC 14 via the communication unit 132, thereby displaying the generated AI chat screen on PC 14 (step S307). At this time, in PC 14, the communication unit 122 receives the screen information of the generated AI chat screen, and the display unit 123 displays the generated AI chat screen.

[0109] Subsequently, Server 12 and PC 14 complete the series of processes shown in Figure 13.

[0110] (Example of a generated AI chat screen) Figures 14 to 16 show an example of a generated AI chat screen displayed on a PC 14 according to one embodiment.

[0111] The generated AI chat screen 210 shown in Figure 14 is displayed on the PC 14 under the control of the display control unit 135 of the server 12, and represents the state after the image and input text have been displayed by the user.

[0112] In the example shown in Figure 14, the upper part of the display area 212 on the right side of the generated AI chat screen 210 displays a model selection field where the user can select the models for the vision encoder and LMM. The user can select the models for the vision encoder and LMM in this model selection field.

[0113] Furthermore, in the example shown in Figure 14, the lower input field in the display area 212 on the right side of the generated AI chat screen 210 displays the image and text entered by the user, "Please create a story based on this image." The user can send the text to the server 12 by pressing the triangular send button displayed in the input field. In addition, buttons for voice input and file attachment are also displayed in the input field, and the user can input the text by voice or attach files by pressing these buttons.

[0114] The generated AI chat screen 220 shown in Figure 15 represents the state after the response text to the input text has been displayed under the control of the display control unit 135 of the server 12. In the example shown in Figure 15, the right-hand display area 222 of the generated AI chat screen 220 displays the input text entered by the user and the response text to the input text (i.e., a story based on the input image) generated by the response generation unit 134 of the server 12 in chronological order.

[0115] The generation AI chat screen 230 shown in Figure 16 is a modified version of the generation AI chat screen 210 shown in Figure 14. In the generation AI chat screen 230 shown in Figure 16, the model selection field displayed at the top of the right-hand display area 232 allows selection of an LMM instead of selecting between a vision encoder and an LLM model. The user can select an LMM in this model selection field. Even in this case, the identification unit 133 of the server 12 can identify the adapter layer and conversion layer corresponding to the LMM selected by the user by referring to the storage unit 131.

[0116] In addition, in the generated AI chat screens 210, 220, and 230, the left-hand display area 211, 221, and 231 shows the history of generated AI chats. However, the history of generated AI chats may also be displayed in several groups, such as "Recent" and "2024 / 12 / 11" (date).

[0117] Additionally, display areas 211, 221, and 231 display a search field, and by entering a search term into this field, the user can view the history of generated AI chats that match that search term.

[0118] According to one embodiment of the server 12 and PC 14, in the generation AI chat processing, the user can select a combination of a vision encoder (a model that vectorizes non-text elements) and an LLM, thereby providing the user with a more accurate or user-desired response.

[0119] In the above embodiment, the "input text" entered by the user includes both text and an image, but is not limited to this. For example, the "input text" entered by the user may include both text and information other than images (e.g., audio).

[0120] Although preferred embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications or changes are possible within the scope of the gist of the present invention as described in the claims.

[0121] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, server 12 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein. [Explanation of symbols]

[0122] 10 Information Processing Systems 12 servers 14 PC 16 Communication Networks 101 Acquisition Department 102 Image vectorization unit 102A Vision Encoder 103 First Conversion Unit 103A adapter 104 Processing Request Unit 105 Second Conversion Unit 105A Conversion Layer 106 Output section 107 Conversion Layer Generation Unit 111 The First Large-Scale Language Model (LLM) 112. The Second Large-Scale Language Model (LLM) 121 Input Reception Section 122 Communications Department 123 Display section 131 Storage section 132 Communications Department 133 Specific part 134 Response generation unit 135 Display Control Unit 210,220,230 Generated AI chat screen 211,212,221,222,231,232 Display area [Prior art documents] [Patent Documents]

[0123] [Patent Document 1] Japanese Patent Publication No. 2024-160599

Claims

1. A first transformation unit converts the output of an information vectorization model, which converts non-text information into vectors and outputs them, into a form that can be processed by a first large-scale language model. A second conversion unit converts the output of the first conversion unit into a form that can be processed by a second large-scale language model, When the second conversion unit converts the output of the first conversion unit, a processing request unit outputs the output of the second conversion unit to the second large-scale language model. The system includes an output unit that outputs a response based on the processing results of the second large-scale language model. Information processing device.

2. The second conversion unit is, Using a transformation layer, the output of the first transformation unit is transformed into a form that can be processed by the second large-scale language model by performing a linear transformation. The information processing apparatus according to claim 1.

3. The system includes a conversion layer generation unit that generates the conversion layer based on the relationship between a base of vectors obtained by a first text vectorization model, which is included in the first large-scale language model, and a base of vectors obtained by a second text vectorization model, which is included in the second large-scale language model, which vectorizes text. The information processing apparatus according to claim 2.

4. The aforementioned conversion layer generation unit, Based on a first word vector generated by averaging multiple vectors obtained by inputting a string into the first large-scale language model, and a second word vector generated by averaging multiple vectors obtained by inputting the string into the second large-scale language model, the transformation layer is generated to linearly transform the first word vector into the second word vector. The information processing apparatus according to claim 3.

5. A storage unit stores the conversion layer and adapter layer generated by the conversion layer generation unit in association with the combination of the vision encoder and the large-scale language model. An identification unit identifies the conversion layer and the adapter layer corresponding to the combination of the vision encoder and the large-scale language model selected by the user by referring to the storage unit, A response generation unit generates a response sentence to an input sentence entered by the user based on the output of a large-scale language model having the conversion layer and the adapter layer identified by the identification unit. The information processing apparatus according to claim 3 or 4, comprising:

6. The information processing apparatus according to claim 5, Terminal device and Equipped with, The aforementioned terminal device is An input receiving unit that accepts the user's selection of a combination of the vision encoder and the large-scale language model, and the input of the input sentence, A display unit that displays the response statement generated by the response generation unit. An information processing system having

7. A first transformation unit converts the output of an information vectorization model, which converts non-text information into vectors and outputs them, into a form that can be processed by a first large-scale language model. A second conversion unit converts the output of the first conversion unit into a form that can be processed by a second large-scale language model, When the second conversion unit converts the output of the first conversion unit, a processing request unit outputs the output of the second conversion unit to the second large-scale language model. The system includes an output unit that outputs a response based on the processing results of the second large-scale language model. Information processing system.

8. An information processing method performed by an information processing device, A first transformation step involves converting the output of an information vectorization model, which converts non-text information into vectors and outputs them, into a form that can be processed by a first large-scale language model. A second transformation step that transforms the output of the first transformation step into a form that can be processed by a second large-scale language model, If the output of the first conversion step is converted in the second conversion step, a processing request step is made to output the output of the second conversion step to the second large-scale language model, The process includes an output step of outputting a response based on the processing results of the second large-scale language model. Information processing methods.

9. Computers, A first transformation unit that converts the output of an information vectorization model, which converts non-text information into vectors and outputs them, into a form that can be processed by a first large-scale language model. A second conversion unit converts the output of the first conversion unit into a form that can be processed by a second large-scale language model. When the second conversion unit converts the output of the first conversion unit, a processing request unit outputs the output of the second conversion unit to the second large-scale language model, and This will function as an output unit that outputs a response based on the processing results of the second large-scale language model. program.

Images