Method for generating a vision foundation model based on multimodality prompts and a model generation apparatus using the same.

Abstract

This invention provides a method and apparatus for generating a vision foundation model based on multimodality prompts. [Solution] The method involves pre-training an initial vision foundation model that includes an initial prompt encoder for encoding initial modality data, adding a first prompt encoder for encoding first modality data and an nth prompt encoder for encoding nth modality data, and then fine-tuning each of the first to nth prompt encoders by utilizing the distillation loss between the embedding vector of the initial prompt encoder and the embedding vector of the first prompt encoder, as well as the alignment loss calculated between each of the first to nth prompt encoders.

Description

Technical Field

[0001] The present invention relates to a method for generating a vision foundation model based on a multimodality prompt and a model generation device using the same.

Background Art

[0002] A vision foundation model (VFM) is a machine learning model or deep learning model pre-trained through large-scale data so as to be applicable to various domains related to human visual information. For example, in the case of a CLIP (Contrastive Language-Image Pretraining) model trained by contrastive learning in which about 400 million sets of image-text pairs collected by OpenAI are input as training materials, it is known that the model is trained. Such a vision foundation model has the effect of enhancing the usability of prompts of various vision modalities such as boxes, sketches, and photos. However, as shown in the example of CLIP, in order to train a VFM, a large amount of training data needs to be collected, the collected data needs to be classified into pairs corresponding to each other, and pre-training such as evaluating each of the classified pairs is performed. Therefore, there is a problem that the cost required for learning is very high.

[0003] Thus, various methods have been studied for developing a model that can handle the image modality and the text modality simultaneously and does not have a high learning cost. As an example, Q-former is a Querying Transformer trained to project an image embedding vector onto a text embedding vector. When using Q-former, a pre-trained image encoder and LLM can be utilized as they are, and thus it has the effect of reducing the learning cost compared to the conventional VFM.

[0004] However, in such cases, the model basically does not perform operations on data composed of image-text modality pairs, that is, it does not perform cross-modality operations like CLIP, but only operations between text modalities. Therefore, projection-based models like Q-Former have the limitation of being less expressive than models that have undergone contrast learning.

[0005] Therefore, the applicant proposes the present invention to reduce the training cost of a VFM model that performs cross-modality operations between image and text modalities. [Overview of the project] [Problems that the invention aims to solve]

[0006] The purpose of this invention is to solve all of the problems of the prior art described above.

[0007] Another objective of this invention is to support the initial vision foundation model in receiving multimodality prompt inputs even when the model architecture does not change significantly, by adding a first contrastive prompt encoder to an nth contrastive prompt encoder to an initial vision foundation model that includes a pre-trained initial prompt encoder, and training the first to nth contrastive prompt encoders by referring to the output of the initial prompt encoder (i.e., the embedding vector).

[0008] The present invention also aims to gradually reduce the influence of the initial prompt encoder during the fine-tuning process of the first contrastive prompt encoder to the nth contrastive prompt encoder, by increasing the amount of training data input to each of the first contrastive prompt encoder to the nth contrastive prompt encoder with each iteration of the fine-tuning process. [Means for solving the problem]

[0009] According to one embodiment of the present invention, in a method for generating a vision foundation model based on multimodality prompts, (a) a model generation device adds a first contrastive prompt encoder to an nth contrastive prompt encoder (where n is an integer of 2 or more, and the first contrastive prompt encoder to the nth contrastive prompt encoder is pre-trained by contrastive learning of a first prompt encoder that encodes first modality data to an nth prompt encoder that encodes nth modality data, and the first modality data corresponds to the initial modality data);Furthermore, (b) when at least a portion of the initial modality learning data and the first modality learning data to the nth modality learning data corresponding to a specific domain is acquired, the model generation device executes at least a portion of the following: (i) a first subprocess which inputs the initial modality learning data to the initial prompt encoder and uses the initial prompt encoder to encode the initial modality learning data and generate an initial modality embedding vector; and (ii) a second subprocess which inputs each of the first modality learning data to the nth modality learning data to the first contrastive modality prompt encoder to the nth contrastive modality prompt encoder and uses the first contrastive modality prompt encoder to the nth contrastive modality prompt encoder to encode each of the first modality learning data to the nth modality learning data and generate a first modality embedding vector to the nth modality embedding vector; and (iii) the initial modality A method is provided that includes the steps of: (iv) generating a vision foundation model based on multimodality prompts by performing at least part of a third subprocess for fine-tuning the first contrastive modality prompt encoder using distillation losses generated with reference to the i-embedding vector and the first modality embedding vector; and (iv) generating a vision foundation model based on multimodality prompts by performing at least part of a fourth subprocess for fine-tuning the first contrastive modality prompt encoder or the n-contrastive modality prompt encoder using alignment losses generated with reference to the first modality embedding vector or the n-modality embedding vector.

[0010] In one example, the model generation device fine-tunes each of the first contrastive prompt encoders to the nth contrastive prompt encoders by either a full fine-tuning method that updates each parameter of the first contrastive prompt encoder to the nth contrastive prompt encoder, or by a Parameter Efficient Fine Tuning (PEFT) method that updates each parameter of a first specific intermediate layer to an nth specific intermediate layer corresponding to one of the first intermediate layers to the nth intermediate layer included in each of the first to nth contrastive prompt encoders.

[0011] In one example, in step (b), with the first low-rank adapter to the nth low-rank adapter each provided in parallel with the first specific intermediate layer to the nth specific intermediate layer, the model generation device updates the parameters of the first low-rank adapter to the nth low-rank adapter, and then updates the parameters of the first specific intermediate layer to the nth specific intermediate layer using the updated parameters of the first low-rank adapter to the nth low-rank adapter.

[0012] In one example, in step (b), when each of the first modality learning data or the nth modality learning data is input to each of the first contrastive modality encoder or the nth contrastive prompt encoder, each of the first contrastive modality encoder or the nth contrastive prompt encoder (i) generates each of the first specific modality hidden state or the nth specific modality hidden state corresponding to each of the first modality learning data or the nth modality learning data through each of the first specific intermediate layer or the nth specific intermediate layer of each of the first contrastive prompt encoder or the nth contrastive prompt encoder The process executes (ii) a subprocess that generates a first low-rank modality hidden state to an nth low-rank modality hidden state corresponding to each of the first modality learning data to the nth modality learning data through each of the first low-rank adapters to the nth low-rank adapters of the first contrastive prompt encoder to the nth contrastive prompt encoder, and (iii) a subprocess that generates a first modality embedding vector by referring to the first specific modality hidden state and the first low-rank modality hidden state, and a subprocess that generates an nth modality embedding vector by referring to each of the nth specific modality hidden states and each of the nth low-rank modality hidden states.

[0013] In one example, in step (b), (b1) each of the first specific intermediate layer to the n specific intermediate layer includes a d×k dimension (where d and k are each integers greater than 0) first intermediate layer parameter matrix to the nth intermediate layer parameter matrix, and each of the first low-rank adapter to the nth low-rank adapter includes a d×r dimension (where r is an integer less than k and greater than 0) 1_1 low-rank parameter matrix and an r×k dimension 1_2 low-rank parameter matrix to a d×r dimension n_1 low-rank parameter matrix and an r×k dimension n_2 low-rank parameter matrix, the model generator updates each of the 1_1 low-rank parameter matrix and the 1_2 low-rank parameter matrix to the n_1 low-rank parameter matrix and the n_2 low-rank parameter matrix corresponding to each of the first contrastive prompt encoder to the nth contrastive prompt encoder, with reference to at least a portion of the distillation loss and the alignment loss;(b) The further step includes (b) detecting that learning for each of the first contrastive prompt encoders to the nth contrastive prompt encoders has been completed, the model generator generates each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders by referring to each of the first_1 low-rank parameter matrix and the first_2 low-rank parameter matrix to the n_1 low-rank parameter matrix and the n_2 low-rank parameter matrix, and updating each of the first intermediate layer parameter matrix to the nth intermediate layer parameter matrix corresponding to each of the first specific intermediate layer to the nth specific intermediate layer by referring to each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders;

[0014] In one example, in step (b), a learning schedule is determined which includes the initial encoder input probability and the first contrastive encoder input probability to the nth contrastive encoder input probability corresponding to the initial prompt encoder and the first contrastive prompt encoder to the nth contrastive prompt encoder, respectively, and which includes the initial modality learning data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group, respectively. With this set, the model generator refers to the learning schedule and performs the initial modality learning By extracting at least a portion of the data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group, the initial modality learning data and the first modality learning data to the nth modality learning data are obtained, respectively, such that the ratio of the amount of data contained in the initial modality learning data and the first modality learning data to the nth modality learning data follows the initial encoder input probability and the first contrastive encoder input probability to the nth contrastive encoder input probability, respectively.

[0015] In one example, step (b) further includes the step of (b3) having the model generation device train each of the first contrastive prompt encoders and the nth contrastive prompt encoders, with the rate of change of the initial encoder input probability after updating being 0 or less, and the rate of change of each of the first modality target input probability to the nth modality target input probability after updating being 0 or more, and if the initial encoder input probability is greater than 0, the device linearly updates the initial encoder input probability and each of the first modality target input probability to the nth modality target input probability included in the training schedule, and if it detects that the updated initial encoder input probability is 0, it determines that the rate of change of the initial encoder input probability and each of the first modality target input probability to the nth modality target input probability after updating is 0.

[0016] In one example, in step (b), with the updated rate of change of the first modality target input probability to the nth modality target input probability determined to be 0, the model generation device executes the second subprocess and the fourth subprocess for a predetermined number of additional learning iterations, and then terminates learning for the first contrastive prompt encoder to the nth contrastive prompt encoder.

[0017] In one example, in step (a), the distillation loss is the InfoNCE (information Noise-Contrastive Estimation) loss between the initial modality embedding vector and the first modality embedding vector, and the alignment loss is the sigmoid loss between each of the first modality embedding vectors and the nth modality embedding vectors.

[0018] In one example, in step (b), after the model generation device has completed fine-tuning for each of the first contrastive prompt encoders to the n contrastive prompt encoders, it removes the initial prompt encoder from the vision foundation model.

[0019] Furthermore, according to another embodiment of the present invention, a model generation device for generating a vision foundation model based on multimodality prompts includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor performs the process of (I) adding a first contrastive prompt encoder to an nth contrastive prompt encoder (where n is an integer of 2 or more, and the first contrastive prompt encoder to the nth contrastive prompt encoder is pre-trained by contrastive learning of a first prompt encoder that encodes first modality data to an nth prompt encoder that encodes nth modality data, and the first modality data corresponds to the initial modality data);Furthermore, (II) when at least a portion of the initial modality learning data and the first modality learning data to the nth modality learning data corresponding to a specific domain is acquired, (i) a first subprocess inputs the initial modality learning data into the initial prompt encoder, and uses the initial prompt encoder to encode the initial modality learning data and generate an initial modality embedding vector, and (ii) inputs each of the first modality learning data to the nth modality learning data into the first contrastive modality prompt encoder to the nth contrastive modality prompt encoder, and uses the first contrastive modality prompt encoder to the nth contrastive modality prompt encoder to encode each of the first modality learning data to the nth modality learning data and generate a first modality embedding vector to the nth modality embedding vector A model generation device is provided that performs a process of generating a vision foundation model based on multimodality prompts by: (iii) executing at least a portion of a second subprocess to generate each of the prompts; (iii) executing at least a portion of a third subprocess to fine-tune the first contrastive modality prompt encoder using distillation losses generated with reference to the initial modality embedding vector and the first modality embedding vector; and (iv) executing at least a portion of a fourth subprocess to fine-tune the first contrastive modality prompt encoder or the nth contrastive modality prompt encoder using alignment losses generated with reference to the first modality embedding vector or the nth modality embedding vector.

[0020] In one example, in process (II), the processor fine-tunes each of the first contrastive prompt encoders to the nth contrastive prompt encoders by updating each parameter of the first contrastive prompt encoder to the nth contrastive prompt encoder, or by updating each parameter of a first specific intermediate layer to the nth specific intermediate layer corresponding to any one of the first intermediate layers to the nth intermediate layer included in each of the first contrastive prompt encoder to the nth contrastive prompt encoder.

[0021] In one example, in the process (II) described above, with a first low-rank adapter and an nth low-rank adapter each provided in parallel with the first specific intermediate layer and the nth specific intermediate layers, the processor updates the parameters of the first low-rank adapter and the nth low-rank adapter, and then updates the parameters of the first specific intermediate layer and the nth specific intermediate layers using the updated parameters of the first low-rank adapter and the nth low-rank adapter.

[0022] In one example, in process (II), when each of the first modality learning data or the nth modality learning data is input to each of the first contrastive modality encoder or the nth contrastive prompt encoder, each of the first contrastive modality encoder or the nth contrastive prompt encoder (i) generates each of the first specific modality hidden state or the nth specific modality hidden state corresponding to each of the first modality learning data or the nth modality learning data through the first specific intermediate layer or the nth specific intermediate layer of each of the first contrastive prompt encoder or the nth contrastive prompt encoder The process executes: (ii) a subprocess that generates a first low-rank modality hidden state to an n low-rank modality hidden state corresponding to each of the first modality learning data to the n modality learning data through each of the first low-rank adapters to the n low-rank adapters of the first contrastive prompt encoder to the n contrastive prompt encoder; and (iii) a subprocess that generates a first modality embedding vector by referring to the first specific modality hidden state and the first low-rank modality hidden state, or a subprocess that generates each of the n specific modality hidden states and each of the n low-rank modality hidden states.

[0023] In one example, in the (II) process, the process updates the respective 1_1 low-rank parameter matrix and the 1_2 low-rank parameter matrix and the n_1 low-rank parameter matrix corresponding to each of the first contrastive prompt encoder and the n-th contrastive prompt encoder, by reference to the alignment loss, with each of the first specific intermediate layer to the n specific intermediate layers including each of the d × k dimension (where d and k are integers greater than 0), and each of the first low-rank adapter to the n low-rank adapter including each of the 1_1 low-rank parameter matrix and the 1_2 low-rank parameter matrix and the n_1 low-rank parameter matrix and the n_2 low-rank parameter matrix, respectively;Furthermore, (II2) upon detecting that learning for each of the first contrastive prompt encoders to the nth contrastive prompt encoders has been completed, the system further executes a process of generating each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders by referring to each of the first_1 low-rank parameter matrix and the first_2 low-rank parameter matrix to the n_1 low-rank parameter matrix and the n_2 low-rank parameter matrix, and updating each of the first intermediate layer parameter matrix to the nth intermediate layer parameter matrix corresponding to each of the first specific intermediate layer to the nth specific intermediate layer by referring to each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders;

[0024] In one example, in process (II), a learning schedule is determined which includes the initial encoder input probability and the first contrastive encoder input probability to the nth contrastive encoder input probability corresponding to the initial prompt encoder and the first contrastive prompt encoder to the nth contrastive prompt encoder, respectively, and which includes the initial modality learning data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group, and the processor refers to the learning schedule to perform the initial modality learning By extracting at least a portion of the data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group, the initial modality learning data and the first modality learning data to the nth modality learning data are obtained, respectively, such that the ratio of the amount of data contained in the initial modality learning data and the first modality learning data to the nth modality learning data follows the initial encoder input probability and the first contrastive encoder input probability to the nth contrastive encoder input probability, respectively.

[0025] In one example, in the (II) process, (II3) after the update of the initial encoder input probability, the change rate is 0 or less, and in a state where the change rate after the update of each of the first modality target input probability to the n-th modality target input probability is 0 or more, after the processor learns each of the first contrastive prompt encoder to the n-th contrastive prompt encoder, if the initial encoder input probability is greater than 0, the processor linearly updates each of the initial encoder input probability and the first modality target input probability to the n-th modality target input probability included in the learning schedule, and when detecting that the updated initial encoder input probability is 0, determines the change rate after the update of each of the initial encoder input probability and the first modality target input probability to the n-th modality target input probability to be 0; and further executes the process.

[0026] In one example, in the (II) process, in a state where the change rate after the update of each of the first modality target input probability to the n-th modality target input probability is determined to be 0, after the processor executes the second sub-process and the fourth sub-process a predetermined number of additional learning times, the learning for each of the first contrastive prompt encoder to the n-th contrastive prompt encoder is terminated.

[0027] In one example, in the (I) process, the distillation loss is the InfoNCE loss between the initial modality embedding vector and the first modality embedding vector, and the alignment loss is the sigmoid loss between each of the first modality embedding vector to the n-th modality embedding vector.

[0028] In one example, in the (II) process, after the processor completes fine-tuning for each of the first contrastive prompt encoder to the n-th contrastive prompt encoder, the initial prompt encoder is removed from the vision foundation model.

Advantages of the Invention

[0029] The present invention adds a first contrastive prompt encoder to an n-th contrastive prompt encoder to an initial vision foundation model including a pre-trained initial prompt encoder, and learns the first contrastive prompt encoder to the n-th contrastive prompt encoder by referring to the output (i.e., embedding vector) of the initial prompt encoder, so that the initial vision foundation model can receive multi-modal prompt inputs even when the model architecture does not change significantly.

[0030] In the process of fine-tuning the first contrastive prompt encoder to the n-th contrastive prompt encoder added to the initial vision foundation model, the present invention increases the amount of data of the training data input to each of the first contrastive prompt encoder to the n-th contrastive prompt encoder for each iteration of the fine-tuning process, so as to gradually reduce the influence of the initial prompt encoder in the process of fine-tuning the first contrastive prompt encoder to the n-th contrastive prompt encoder.

Brief Description of the Drawings

[0031] The following drawings, attached for use in describing embodiments of the present invention, represent only a portion of the embodiments, and a person with ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary art") can obtain other drawings from these drawings without performing any inventive work.

[0032] [Figure 1a] This figure schematically shows a model generation device for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention. [Figure 1b] This figure schematically illustrates the operation mode of a vision foundation model based on multimodality prompts generated by learning a model generation device according to one embodiment of the present invention. [Figure 2] This is a flowchart showing a learning method for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention. [Figure 3] This figure schematically shows the data pipeline in the learning process for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention. [Figure 4] This figure schematically illustrates how alignment loss and distillation loss are calculated during the learning process for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention. [Figure 5] This figure schematically illustrates that, in a learning process for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention, a first low-rank adapter, consisting of a 1_1 low-rank adapter and a 1_2 low-rank adapter, is added in parallel to a first specific intermediate layer included in the first contrastive prompt encoder. [Modes for carrying out the invention]

[0033] The detailed description of the present invention, as described below, will refer to the accompanying drawings illustrating specific embodiments in which the present invention may be carried out, in order to clarify the object, technical solution, and advantages of the present invention. These embodiments will be described in sufficient detail so that a person of the ordinary skill can carry out the present invention.

[0034] Furthermore, throughout the detailed description and claims of the present invention, the term “including” and its variations are not intended to exclude other technical features, appendices, components, or parts. To an ordinary person, some of the other purposes, advantages, and characteristics of the present invention will become apparent from this specification and some from the practice of the present invention. The following examples and drawings are provided as illustrative examples and are not intended to limit the present invention.

[0035] Furthermore, the present invention encompasses all possible combinations of the embodiments described herein. It should be understood that while the various embodiments of the present invention differ from one another, they do not necessarily have to be mutually exclusive. For example, certain shapes, structures, and characteristics described herein can be realized in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. It should also be understood that the position or arrangement of individual components within each disclosed embodiment can be modified without departing from the spirit and scope of the invention. Therefore, the detailed descriptions set forth below should not be construed as restrictive, and the scope of the present invention is limited only by the appended claims, along with all equivalent scopes claimed by those claims, provided they are adequately described. In the drawings, similar reference numerals indicate identical or similar components across various aspects.

[0036] In the following, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that persons with ordinary skill in the art to which the present invention pertains can easily implement the present invention.

[0037] Figure 1a is a schematic diagram showing a model generation device that generates a vision foundation model based on multimodality prompts according to one embodiment of the present invention.

[0038] Referring to Figure 1a, the model generation device 1000 that generates a vision foundation model based on multimodality prompts may include memory 1100 and a processor 1200. In this case, the model generation device 1000 may be a server computer, but is not limited to that, and may include various computing devices.

[0039] First, the memory 1100 of the model generation device 1000, which generates a vision foundation model based on multimodality prompts, can store instructions to be executed by the processor 1200. Specifically, the instructions are code generated for the purpose of enabling the model generation device 1000, which generates a vision foundation model based on multimodality prompts, to function in a particular way, and the memory 1100 for storing these instructions may be computer-accessible or computer-readable memory that can be used by a computer or other programmable data processing equipment. The instructions may correspond to processes for performing the functions described in the specification of the present invention.

[0040] Furthermore, the processor 1200 of the model generation device 1000 may include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and a data bus. The model generation device 1000 may also further include an operating system and software components for applications that perform specific purposes.

[0041] However, this does not exclude the case in which the processor 1200 of the model generation device 1000 includes an integrated processor in which a medium, processor, and memory are integrated for carrying out the present invention.

[0042] Furthermore, referring to the example disclosed in Figure 1a, the model generation device 1000 may further include, but is not limited to, a vision foundation model 2000, and may also be linked to a vision foundation model 2000 located outside the model generation device 1000. In this case, the model generation device 1000 may further include a hardware and software configuration for receiving a multimodality prompt 3100 input to the vision foundation model 2000, and through the received multimodality prompt 3100, an image data 3200 on which the vision foundation model 2000 is to perform a task (e.g., object detection), and a configuration for transmitting the result 3300 calculated from the vision foundation model 2000.

[0043] In this example, to facilitate understanding of the drawing, the multimodality prompt 3100 and image data 3200 are shown as being directly input to the vision foundation model 2000, and the result 3300 is shown as being directly output from the vision foundation model 2000. However, this is not the only possible configuration. The multimodality prompt 3100 and image data 3200 may be input to the memory 1100 of the model generation device 1000, and then input to the vision foundation model 2000 via the processor 1200. The result 3300 calculated from the vision foundation model 2000 may then be stored in the memory 1100 and transmitted to a connected user terminal (not shown). Various modifications are possible.

[0044] Furthermore, as described above, another computing device (not shown) may include a vision foundation model (not shown), and the model generation device 1000 may transmit instructions for training and computation to the computing device (not shown), thereby enabling the computing device (not shown) to learn the vision foundation model (not shown). For this purpose, the model generation device 1000 may further include hardware or software configurations for communicating with the computing device (not shown). In addition, depending on the context in which the present invention is being implemented, the model generation device 1000 may further include configurations for communicating with another database (not shown). In this case, the database may include, but is not limited to, at least one of the following storage media: flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. It may also include any media capable of storing data.

[0045] On the other hand, a detailed explanation of the structure of the Vision Foundation Model 2000 will be provided later with reference to Figure 1b.

[0046] Figure 1b schematically shows the operation mode of a vision foundation model based on multimodality prompts generated by learning a model generation device according to one embodiment of the present invention.

[0047] Referring to Figure 1b, the vision foundation model 2000, trained by the method disclosed in this invention, can receive multimodality prompts 3100 and image data 3200 as inputs, perform calculations on them, and output a result 3300. On the other hand, the multimodality prompts 3100 and image data 3200 may be obtained from a user terminal (not shown) that can communicate with the vision foundation model 2000.

[0048] Specifically, in the example shown in Figure 1b, a user corresponding to a user terminal (not shown) inputs an image containing multiple balls and multiple girls as image data 3200, and the Vision Foundation Model 2000 is configured to detect the balls and girls shown on the image data 3200. At this time, the user may further input a multimodality prompt 3100 for the purpose of providing the Vision Foundation Model 2000 with semantic context about the balls and girls, that is, for the Vision Foundation Model 2000 to refer to when detecting the balls and girls from the image data 3200. The multimodality prompt 3100 includes a text modality prompt 3110 and an image modality prompt 3120, each of which may include a text prompt 3111 and an image prompt 3121 that represent the meaning of a ball, and a text prompt 3112 and an image prompt 3122 that represent the meaning of a girl.

[0049] On the other hand, the multimodality prompt 3100 can be embedded through the prompt encoder 2100 of the vision foundation model 2000. Here, in the case of the prompt image 3121 for ball 3121 and the prompt image 3122 for girl 3122 included in the image modality prompt 3120, although they are shown as photographs in the example in Figure 1b, they are not limited to this, and for example, a sketch drawn by a person or a screenshot cropped by the user from a portion of the image data 3200 may be determined as the image modality prompt 3120, and various modifications are possible.

[0050] Furthermore, the prompt encoder 2100 of the vision foundation model 2000 shown in Figure 1b can simultaneously perform calculations on the text modality prompt 3110 and the image modality prompt 3120. For example, to specifically explain the example shown in Figure 1b, the text prompt 3110 input to the vision foundation model 2000 contains the texts "ball 3111" and "girl 3112," and the image prompt 3120 also contains prompt image 3121 for "ball 3121" and prompt image 3122 for "girl 3122." Then, the vision foundation model 2000 can detect the ball class and the girl class from the image data 3200 by performing cross-modality calculations using the text modality prompt 3110 and the image modality prompt 3120.

[0051] In this case, as mentioned above, the multimodality prompt 3100 entered by the user is understood to provide semantic context for the vision foundation model 2000 to detect objects from the image data 3200. That is, it goes without saying that, depending on the user's needs, abstract text such as sphere instead of ball 3111, or concrete text such as volley ball, can also be entered as text modality prompts 3110 that include semantic context. Furthermore, in the example shown in Figure 1b, ball 3121 and girl 3122 in the image modality prompt 3120 do not mean to search for balls that are exactly the same in properties such as color or trademark, or for a person identical to girl 3122, but rather to search for the ball class and the girl class.

[0052] Furthermore, it goes without saying that the vision foundation model 2000 of the present invention can perform object detection from image data 3200 by referring to only one of either the text prompt 3110 or the image prompt 3120.

[0053] Such cross-modality calculations can help users flexibly utilize the Vision Foundation Model 2000. For example, if it is difficult for a user to express prompts using linguistic means such as text, they can input photographs they have taken or sketches they have created into the Vision Foundation Model 2000 and obtain results from the model.

[0054] On the other hand, in the example shown in Figure 1b, the image data 3200 is an image, and the vision foundation model 2000 includes an image encoder 2200 for embedding it. Then, the vision foundation model 2000 can obtain a prompt embedding vector with the multimodality prompt 3100 embedded through the prompt encoder 2100, and an image embedding vector with the image data 3200 embedded through the image encoder 2200, and input these to the transformer 2300 to obtain the result 3300. In this case, in the example in Figure 1b, as an example of object detection, it can be confirmed that the result 3300 includes a bounding box 3301 for the ball and a bounding box 3302 for the girl. In this case, for convenience, reference numbers are assigned only to one of the bounding boxes 3301 for the ball in the image data 3200, and to one of the bounding boxes 3302 for the girls 3112 and 3122. It is understood that all bounding boxes shown with dashed lines correspond to the ball, and all bounding boxes shown with solid lines correspond to the girls.

[0055] In addition, the multimodality prompt 3100 of the present invention is not limited to including text and image modalities 3110 and 3120. For example, if the purpose is to utilize the vision foundation model 2000 in the autonomous driving domain, the vision foundation model 2000 can be trained to perform object detection by further referencing distance modality data such as LiDAR and RADAR. Furthermore, the purpose of the vision foundation model 2000 of the present invention is not limited to object detection, and various modifications are possible, such as the vision foundation model 2000 receiving text data (not shown) from a text modality and input from a multimodality prompt to generate images or videos corresponding to the multimodality prompt and text data (not shown).

[0056] A detailed explanation of how to generate the Vision Foundation Model 2000, as shown in Figure 1b, will be provided further with reference to Figures 2 through 5.

[0057] Figure 2 is a flowchart showing a learning method for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention.

[0058] Referring to Figure 2, in step S21, the model generation device 1000 can add a first contrastive prompt encoder to the nth contrastive prompt encoder to the initial vision foundation model, which includes an initial prompt encoder for encoding initial modality data, after the initial vision foundation model has been pre-trained. In this case, n is an integer of 2 or more, and the first contrastive prompt encoder to the nth contrastive prompt encoder may be pre-trained by contrastive learning of the first prompt encoder for encoding the first modality data to the nth prompt encoder for encoding the nth modality data. Also, the initial modality data may be in a state corresponding to the first modality data.

[0059] To explain in more detail, it is as follows: First, the initial vision foundation model may include an initial prompt encoder that has been pre-trained to have a certain level of expressive power or higher for the initial modality data. Similarly, each of the first contrastive prompt encoder and the nth contrastive prompt encoder may be pre-trained by contrastive learning. In this case, as a result of being pre-trained by contrastive learning, each of the first contrastive prompt encoder and the nth contrastive prompt encoder can encode first modality data and nth modality data, at least partially belonging to different modalities, into embedding vectors in the same latent space. This allows data with similar meanings to be close together, and data with different meanings to be far apart. In this case, by training the embedding vectors of the first modality data and the initial modality data to be represented similarly to each other, it becomes possible to induce the embedding vectors of the second modality data to the nth modality data, which are in a contrasting relationship with the first modality data, to also be in a contrasting relationship with the embedding vector of the initial modality data.

[0060] In this description of the present invention, the correspondence between the first modality data and the initial modality data may mean that the first modality data and the initial modality data are the same modality data (for example, text modality data). For example, the initial modality data may be Korean text data and the first modality data may be English text data, that is, they belong to the text modality but differ in their method of expression, etc.

[0061] Furthermore, at least a portion of each of the first modality data to the nth modality data may correspond to each other's modalities (as in the example of Korean text data and English text data mentioned above), and each of the first modality data to the nth modality data may correspond to different modalities. For example, in the autonomous driving domain mentioned above, n may be 3, and the first modality data may be text data, the second modality data may be image data, and the third modality data may be LiDAR data.

[0062] In step S21, after each of the first contrastive prompt encoders to the nth contrastive prompt encoder is added to the initial vision foundation model, in step S22, the model generation device 1000 may execute at least part of a first subprocess that uses the initial prompt encoder to generate an initial modality embedding vector by inputting the initial modality learning data into the initial prompt encoder, and a second subprocess that uses each of the first and nth contrastive prompt encoders to generate a first modality embedding vector to the nth modality embedding vector by inputting the first modality learning data and the nth modality learning data into each of the first and nth contrastive prompt encoders. At this time, the initial modality learning data and the first modality learning data to the nth modality learning data can correspond to a specific domain.

[0063] Here, a specific domain may correspond to an industrial sector where the generated Vision Foundation Model 2000 is primarily used, or it may correspond to a specific task within the Vision Foundation Model 2000, such as object detection or image generation.

[0064] Subsequently, the model generation device 1000 calculates the distillation loss between the initial prompt encoder and the first contrastive prompt encoder through a third subprocess, and then calculates the alignment loss for the first contrastive prompt encoder or the nth contrastive prompt encoder through a fourth subprocess. This will be described later with reference to Figures 3 and 4.

[0065] Figure 3 is a schematic diagram showing the data pipeline in the learning process for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention, and Figure 4 is a schematic diagram showing how alignment loss and distillation loss are calculated, respectively, in the learning process for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention.

[0066] For reference, Figures 3 and 4 are shown assuming n is 2 for the sake of explanation and understanding, but it goes without saying that n can be 3 or greater, as mentioned above (for example, as in the autonomous driving domain described earlier).

[0067] First, referring to Figure 3, the Vision Foundation Model 2000 is the initial Vision Foundation Model, which includes the initial prompt encoder 2110, with the addition of the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220.

[0068] Furthermore, referring to Figure 4, during the third subprocess, the initial prompt encoder 2110, upon receiving the initial modality learning data 4111, can generate the initial modality embedding vector 4112, and the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220, upon receiving the first modality learning data 4211 and the second modality learning data 4221, respectively, can generate the first modality embedding vector 4212 and the second modality embedding vector 4222, respectively.

[0069] Subsequently, the distillation loss 4300 can be calculated by first evaluating the similarity between the initial modality embedding vector 4112 and the first modality embedding vector 4212. Here, the distillation loss 4300 between the initial modality embedding vector 4112 and the first modality embedding vector 4212 may be, but is not limited to, InfoNCE (information noise-contrastive entropy). After that, the model generation device 1000 can fine-tune the first contrastive prompt encoder 2210 by referring to the distillation loss 4300.

[0070] In this case, the initial modality embedding vector 4112 and the first modality embedding vector 4212 may be embedding vectors of the same dimension. For example, if the initial prompt encoder 2110 is based on BERT and the first contrastive prompt encoder 2210 is based on the Vit-L / 14-based CLIP text encoder, both the initial modality embedding vector 4112 and the first modality embedding vector 4212 may be latent vectors on 768 dimensions. Of course, this does not exclude the case where the initial modality embedding vector 4112 is on a higher dimension than the first modality embedding vector 4212, or vice versa, such as when the first contrastive prompt encoder 2210, based on the CLIP text encoder using Vit-L / 32, generates a latent vector on 512 dimensions. In this way, when calculating the distillation loss 4300 between the initial modality embedding vector 4112 and the first modality embedding vector 4212, and thereby fine-tuning the first contrastive prompt encoder 2210, the first contrastive prompt encoder 2210 can be trained through the distillation loss 4300 so that the first modality embedding vector 4212 generated from the first contrastive prompt encoder 2210 calculates a latent vector similar to the initial modality embedding vector 4112.

[0071] However, in the fine-tuning process using the distillation loss 4300 in this manner, the alignment between the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220 may be compromised. Therefore, the model generation device 1000 can fine-tune each of the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220 by referring to the alignment loss 4400 between the first modality embedding vector 4212 and the second modality embedding vector 4222. In this case, the alignment loss 4400 may be, but is not limited to, a sigmoid loss.

[0072] It should be noted that the initial prompt encoder 2110 is not fine-tuned. That is, the parameters of the initial prompt encoder 2110 are frozen, and only fine-tuning is performed on the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220. As a result, the completed vision foundation model 2000 can use the embedding vectors generated from the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220 without changing the transformer 2300 that was included in the initial vision foundation model 2000. Furthermore, by freezing the parameters of the initial prompt encoder 2110, the first contrastive prompt encoder 2210 can completely mimic the expressiveness of the initial modality encoder 2110.

[0073] Such a fine-tuning process can be repeated over multiple iterations. That is, after the first contrastive prompt encoder 2210 is fine-tuned to mimic the representation of the initial prompt encoder 2110 through the distillation loss 4300, fine-tuning is performed on the first contrastive prompt encoder 2210 and the second contrastive prompt 2220 through the alignment loss 4400. Therefore, it is necessary to recalculate the distillation loss 4300 between the first contrastive prompt encoder 2210 and the initial prompt encoder 2110, and then perform fine-tuning on the first contrastive prompt encoder 2210 again using the newly calculated distillation loss 4300. Also, when the fine-tuning process is performed over multiple iterations, the number of times each of the first to fourth subprocesses is performed may differ in each iteration.

[0074] Specifically, for each of the initial prompt encoder 2110 and the first contrastive prompt encoder 2210 to the second contrastive prompt encoder 2220, the initial encoder input probability and the first contrastive encoder input probability to the second contrastive encoder input probability, as well as the initial modality training data candidate group and the first modality training data candidate group to the second modality training data candidate group, may already be determined.

[0075] Here, the initial modality training data candidate group and the first modality training data candidate group or the second modality training data candidate group may each be a set of data selected for a general domain, or a set of training data specialized for a specific domain. Also, as mentioned above, if the initial modality data and the first modality data are the same, then the initial modality training data candidate group and the first modality training data candidate group may also be the same.

[0076] Furthermore, the initial encoder input probability and the first contrastive encoder input probability or the second contrastive encoder input probability can correspond to the probability that the initial modality training data 4111, the first modality training data 4211, and the second modality training data 4221 are input to the initial prompt encoder 2110 and the first contrastive prompt encoder 2210 or the second contrastive prompt encoder 2220, respectively. Such probabilities can be achieved by adjusting the batch size of the training data input to the initial prompt encoder 2110 and the first contrastive prompt encoder 2210 or the second contrastive prompt encoder 2220, respectively.

[0077] In other words, the amount of data (i.e., batch size) contained in the initial modality training data and the first or second modality training data can follow the initial encoder input probability and the first or second contrastive encoder input probability included in the training schedule. For example, assuming that the initial encoder input probability is 1 and the first or second contrastive encoder input probability is 0, only the initial modality training data may be acquired. If the initial encoder input probability and the first or second contrastive encoder input probability are each 1 / 3, then the amount of data contained in the acquired initial modality training data and the first or second modality training data may all be the same.

[0078] With the learning schedule and the candidate sets of initial modality learning data and the candidate sets of first or second modality learning data determined in this way, the initial modality learning data and the first or second modality learning data can be obtained by referencing the initial encoder input probability and the first or second contrastive encoder input probability, respectively, and extracting at least a portion of the candidate sets of initial modality learning data and the first or second modality learning data.

[0079] At this time, by adjusting the initial encoder input probability and the first contrastive encoder input probability or the second contrastive encoder input probability as described above, it is possible to adjust the degree to which the influence of the initial prompt encoder and the first contrastive prompt encoder or the second contrastive prompt encoder is reflected during one iteration in the learning process of the vision foundation model of the present invention.

[0080] As an example, the model generation device 1000 can acquire only information about the initial prompt encoder by starting the learning of the vision foundation model 2000 with an initial encoder input probability of 1 and the input probabilities of the first contrastive encoder and the second contrastive encoder, each set to 0. Subsequently, by reducing the initial encoder input probability to 0.9 and increasing the input probabilities of the first contrastive encoder and the second contrastive encoder to 0.05, each set can be used to confirm the effects of the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220 while retaining a significant influence from the initial prompt encoder 2110. Such changes in the initial encoder input probability and the input probabilities of the first and second contrastive encoders can be achieved linearly, but are not limited to this.

[0081] In other words, the rate of change of the initial encoder input probability after updating may be 0 or less (i.e., decreasing or remaining the same), and the rate of change of the first modality target input probability or the second modality target input probability after updating may be 0 or greater (i.e., increasing or remaining the same). Then, after fine-tuning the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220 respectively, if the initial encoder input probability is greater than 0, the model generator 1000 can linearly update the initial encoder input probability and the first modality target input probability or the second modality target input probability included in the learning schedule. In the example described above, it can be assumed that the rate of change of the initial encoder input probability after updating is -0.1, and the rate of change of the first modality target input probability or the second modality target input probability after updating is 0.05. As a result, the initial encoder input probability decreases to 1, 0.9, 0.8, ..., 0 with each update, while the input probabilities for the first modality and the input probabilities for the second modality increase to 0, 0.05, 0.1, ..., 0.5, respectively. Furthermore, since it is mathematically impossible for the probability to be less than 0, if the model generator 1000 detects that the initial encoder input probability after the update is 0, it can also determine that the rate of change of the initial encoder input probability and the input probabilities for the first modality or the second modality after the update is 0.

[0082] On the other hand, once the initial encoder input probability after the update becomes 0, the fine-tuning of the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220 may be terminated, or the additional tuning process may be continued.

[0083] As in the example above, assuming that the rate of change of the initial encoder input probability after the update is -0.1, and the rate of change of the first modality target input probability or the second modality target input probability after the update is 0.05, then when the initial encoder input probability becomes 0, the first modality target input probability and the second modality target input probability will each be at 0.5. As mentioned above, the initial encoder input probability and the first modality target input probability or second modality target input probability correspond to the amount of data contained in the initial modality training data 4111 and the first modality training data 4211 to the second modality training data 4221, respectively. In this situation, the initial modality training data 4111 is not acquired, and the first modality training data 4211 to the second modality training data 4221 can be acquired from the first modality training data candidate group or the second modality training data candidate group, respectively, so that the amount of data contained in each of the first modality training data 4211 to the second modality training data 4221 is the same.

[0084] In this state, the model generation device 1000 executes only the following additional learning iterations: a second subprocess that inputs the first modality learning data 4211 and the second modality learning data 4221 to the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220, respectively, to obtain the first modality embedding vector 4212 and the second modality embedding vector 4222, respectively; and a fourth subprocess that calculates the alignment loss 4400 between the first modality embedding vector 4212 and the second modality embedding vector 4222, and then fine-tunes the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220. After this, the fine-tuning is terminated, thereby ensuring that the first contrastive prompt encoder 2210 and the second contrastive prompt encoder 2220 are properly aligned with each other after fine-tuning. Of course, this is not the only option, and the number of additional training iterations is not fixed. Fine-tuning can also be performed using the alignment loss 4400 until the alignment loss 4400 between the first modality embedding vector 4212 and the second modality embedding vector 4222 falls below a predetermined critical loss value.

[0085] Furthermore, after fine-tuning is complete, the initial prompt encoder 2110 is removed from the vision foundation model 2000, thereby reducing the computational load on the vision foundation model 2000 during inference.

[0086] On the other hand, steps S21 and S22 described above may be performed through a full fine-tuning method that updates each parameter of the first contrastive prompt encoder to the nth contrastive prompt encoder, or through a Parameter Efficient Fine Tuning (PEFT) method that updates each parameter of the first specific intermediate layer to the nth specific intermediate layer corresponding to any one of the first intermediate layers to any one of the nth intermediate layers included in each of the first to nth contrastive prompt encoders. An example in which the present invention is realized through the PEFT method will be explained with reference to Figure 5.

[0087] Figure 5 schematically shows that, in the learning process for generating a vision foundation model based on multimodality prompts according to one embodiment of the present invention, a first low-rank adapter consisting of a 1_1 low-rank adapter and a 1_2 low-rank adapter is added in parallel to a first specific intermediate layer included in the first contrastive prompt encoder.

[0088] First, for the sake of simplicity, only a portion of the first contrastive prompt encoder 2210 is shown in Figure 5. It goes without saying that the same principles described later, referring to Figure 5, also apply to the second contrastive prompt encoder and the nth contrastive prompt encoder.

[0089] First, referring to Figure 5, the t-th layer among the first intermediate layers included in the first contrastive prompt encoder 2210 has been determined to be the first specific intermediate layer 2211. Furthermore, the first specific intermediate layer 2211 has the first low-rank adapter 2212, which includes the first low-rank adapter 2212_1 and the first low-rank adapter 2212_2, arranged in parallel.

[0090] In this state, the first modality learning data 4211 input to the model generation device 1000 can calculate the hidden state in each layer through calculations in each of the first intermediate layers included in the first contrastive prompt encoder 2210. At this time, the calculations through the first specific intermediate layer 2211 and the first low-rank adapter 2212 can be applied to the first modality hidden state 4211_(t-1), which is the product of the intermediate layer immediately preceding the first specific intermediate layer 2211, i.e., the (t-1)th intermediate layer.

[0091] As an example, referring to Figure 5, upon receiving the input of the first modality hidden state 4211_(t-1), the first specific intermediate layer 2211 can calculate the first specific modality hidden state 4211_t1, and the first low-rank adapter 2212 can calculate the first low-rank modality hidden state 4211_t1. Through operations (such as concatenation) on the first specific modality hidden state 4211_t1 and the first low-rank modality hidden state 4211_t2, the first_t modality hidden state 4211_t corresponding to the first specific intermediate layer 2211, which is the t-th intermediate layer, can be calculated. Subsequently, through operations on the first modality hidden state 4211_t of other intermediate layers (not shown) and subsequent modality hidden states, the first modality embedding vector 4212 corresponding to the first modality training data 4211 is finally generated.

[0092] In this state, if fine tuning for the first contrastive prompt encoder 2210 is performed via PEFT, the model generation device 1000 can perform fine tuning for the first low-rank adapter 2212 with the parameters of the first specific intermediate layer 2211 frozen.

[0093] Specifically, the first particular intermediate layer 2211 may contain a d×k dimension first intermediate layer parameter matrix, and each of the 1_1 low-rank adapter 2212_1 and 1_2 low-rank adapter 2212_2, which are part of the first low-rank adapter 2212, may contain a d×r dimension 1_1 low-rank parameter matrix and an r×k dimension 1_2 low-rank parameter matrix, respectively. Here, d and k are integers greater than 0, and r may be an integer less than k and greater than 0.

[0094] Then, the model generator 1000 can update the first-first low-rank parameter matrix and the first-second low-rank parameter matrix, respectively, by referring to the distillation loss 4300 and the alignment loss 4400. Subsequently, when it detects that training for the first contrastive prompt encoder 2210 is complete, the model generator 1000 can complete fine-tuning for the first contrastive prompt encoder 2210 by generating the first low-rank adapter parameter matrix by referring to the first-first low-rank parameter matrix and the first-second low-rank parameter matrix, and by updating the first intermediate layer parameter matrix by referring to the first low-rank adapter parameter matrix. At this time, the process of generating the first low-rank adapter parameter matrix and the process of updating the first intermediate layer parameter matrix by referring to the first low-rank adapter parameter matrix may be performed through conventional LoRA (Low-Rank Adaptation) methods, but is not limited thereto.

[0095] As an example, as mentioned above, when generating the first low-rank adapter parameter matrix through the LoRA (Low-Rank Adaptation) method and updating the first intermediate layer parameter matrix by referencing the first low-rank adapter parameter matrix, the model generator 1000 can generate a d×k-dimensional first low-rank adapter parameter matrix by performing a matrix multiplication on the first low-rank adapter matrix and the first low-rank adapter matrix, with each of the first-first low-rank adapter 2212_1 and the first-second low-rank adapter 2212_2 containing the d×r-dimensional first-first low-rank parameter matrix and the r×k-dimensional first-second low-rank parameter matrix, respectively. Then, the model generator 1000 can update the first intermediate layer parameter matrix by adding the first low-rank adapter parameter matrix obtained by multiplying the existing first intermediate layer parameter matrix by a predetermined scaling factor.

[0096] The embodiments of the present invention described above are implemented in the form of program instructions that can be executed through various computer components and may be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or they may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0097] Although the present invention has been described above with specific details such as concrete components, and with limited embodiments and drawings, these are provided only to aid in a more overall understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention pertains can make various modifications and variations from this description.

[0098] Therefore, the concept of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below shall also fall within the scope of the concept of the present invention.

Claims

1. In a method for generating a vision foundation model based on multimodality prompts, (a) With an initial vision foundation model including an initial prompt encoder for encoding initial modality data pre-trained, the model generator adds a first contrastive prompt encoder to the nth contrastive prompt encoder (where n is an integer of 2 or more, and the first contrastive prompt encoder to the nth contrastive prompt encoder are pre-trained by contrastive learning of a first prompt encoder for encoding first modality data to the nth prompt encoder for encoding nth modality data, and the first modality data corresponds to the initial modality data), (b) When at least a portion of the initial modality learning data and the first modality learning data to the nth modality learning data corresponding to a specific domain is acquired, the model generation device executes at least a portion of the following: (i) a first subprocess which inputs the initial modality learning data to the initial prompt encoder and uses the initial prompt encoder to encode the initial modality learning data and generate an initial modality embedding vector; and (ii) a second subprocess which inputs each of the first modality learning data to the nth modality learning data to the first contrastive prompt encoder to the nth contrastive prompt encoder and uses the first contrastive prompt encoder to the nth contrastive prompt encoder to encode each of the first modality learning data to the nth modality learning data and generate a first modality embedding vector to the nth modality embedding vector; and (iii) the initial (iv) A step of generating a vision foundation model based on multimodality prompts by performing at least part of a third subprocess for fine-tuning the first contrastive prompt encoder using distillation loss generated by referencing the dependency embedding vector and the first modality embedding vector, and a fourth subprocess for fine-tuning the first contrastive prompt encoder or the nth contrastive prompt encoder using alignment loss generated by referencing the first modality embedding vector or the nth modality embedding vector, A method that includes this.

2. In step (b) above, The method according to claim 1, wherein the model generation device fine-tunes each of the first contrastive prompt encoders to the nth contrastive prompt encoders by a full fine-tuning method that updates each parameter of each of the first contrastive prompt encoders to the nth contrastive prompt encoders, or by a PEFT (Parameter Efficient Fine Tuning) method that updates each parameter of each of the first specific intermediate layer to the nth specific intermediate layer corresponding to each of the first to nth intermediate layers included in each of the first to nth contrastive prompt encoders.

3. In step (b) above, The method according to claim 2, wherein a first low-rank adapter to an nth low-rank adapter is provided in parallel with each of the first specific intermediate layer to the nth specific intermediate layer, the model generation device updates each of the parameters of the first low-rank adapter to the nth low-rank adapter, and then updates each of the parameters of the first specific intermediate layer to the nth specific intermediate layer using the updated parameters of the first low-rank adapter to the nth low-rank adapter.

4. In step (b) above, When each of the first modality learning data or the nth modality learning data is input to each of the first contrastive prompt encoder or the nth contrastive prompt encoder, each of the first contrastive prompt encoder or the nth contrastive prompt encoder (i) generates a subprocess through each of the first specific intermediate layer or the nth specific intermediate layer of each of the first contrastive prompt encoder or the nth contrastive prompt encoder that generates a first specific modality hidden state or the nth specific modality hidden state corresponding to each of the first modality learning data or the nth modality learning data, and (ii) the first control The method according to claim 3, comprising: (iii) executing a subprocess for generating each of the first low-rank modality hidden states to the nth low-rank modality hidden states corresponding to each of the first modality learning data to the nth modality learning data through each of the first low-rank adapters to the nth low-rank adapters of the rastic prompt encoder or the nth contrastive prompt encoder; and (iii) executing a subprocess for generating the first modality embedding vector by reference to the first specific modality hidden state and the first low-rank modality hidden state, or a subprocess for generating each of the nth modality embedding vector by reference to each of the nth specific modality hidden states and each of the nth low-rank modality hidden states.

5. In step (b) above, (b1) Each of the first specific intermediate layer to the n specific intermediate layer includes each of the first intermediate layer parameter matrix to the nth intermediate layer parameter matrix of d × k dimensions (where d and k are each integers greater than 0), and each of the first low-rank adapter to the nth low-rank adapter includes each of the first_1 low-rank parameter matrix of d × r dimensions (where r is an integer less than k and greater than 0), and each of the first_2 low-rank parameter matrix of r × k dimensions, to the n_1 low-rank parameter matrix of d × r dimensions, and the n_2 low-rank parameter matrix of r × k dimensions, the model generator updates each of the first_1 low-rank parameter matrix and the first_2 low-rank parameter matrix to the n_1 low-rank parameter matrix and the n_2 low-rank parameter matrix corresponding to each of the first contrastive prompt encoder to the nth contrastive prompt encoder, by referring to at least a portion of the distillation loss and the alignment loss. (b2) When the model generator detects that learning has been completed for each of the first contrastive prompt encoders to the nth contrastive prompt encoders, it generates each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders by referring to each of the first_1 low-rank parameter matrix and the first_2 low-rank parameter matrix to the n_1 low-rank parameter matrix and the n_2 low-rank parameter matrix, and updates each of the first intermediate layer parameter matrix to the nth intermediate layer parameter matrix corresponding to each of the first specific intermediate layer to the nth specific intermediate layer by referring to each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders, The method according to claim 3, further comprising:

6. In step (b) above, The method according to claim 1, wherein a learning schedule is determined which includes the initial encoder input probability and the first modality target input probability to the nth modality target input probability corresponding to the initial prompt encoder and the first contrastive prompt encoder to the nth contrastive prompt encoder, respectively, and which includes the initial modality learning data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group, respectively, and the model generation device, by referring to the learning schedule, extracts at least a portion of the initial modality learning data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group to acquire the initial modality learning data and the first modality learning data to the nth modality learning data, respectively, and the ratio of the amount of data contained in the initial modality learning data and the first modality learning data to the nth modality learning data is determined which includes the initial encoder input probability and the first modality target input probability to the nth modality target input probability, respectively.

7. In step (b) above, (b3) The rate of change of the initial encoder input probability after updating is 0 or less, and the rate of change of each of the first modality target input probability to the nth modality target input probability after updating is determined to be 0 or more. After the model generation device has trained each of the first contrastive prompt encoder to the nth contrastive prompt encoder, if the initial encoder input probability is greater than 0, the initial encoder input probability and each of the first modality target input probability to the nth modality target input probability included in the learning schedule are linearly updated, and when it is detected that the updated initial encoder input probability is 0, the rate of change of each of the first modality target input probability to the nth modality target input probability after updating is determined to be 0. The method according to claim 6, further comprising:

8. In step (b) above, The method according to claim 7, wherein, with the update rate of change of each of the first modality target input probability and the nth modality target input probability determined to be 0, the model generation device executes the second subprocess and the fourth subprocess for a predetermined number of additional learning times, and then terminates learning for each of the first contrastive prompt encoder and the nth contrastive prompt encoder.

9. In step (a) above, The method according to claim 1, characterized in that the distillation loss is the InfoNCE (Information Noise-Contrastive Estimation) loss between the initial modality embedding vector and the first modality embedding vector, and the alignment loss is the sigmoid loss between each of the first modality embedding vectors to the nth modality embedding vector.

10. In step (b) above, The method according to claim 1, wherein the model generation device removes the initial prompt encoder from the vision foundation model after completing fine-tuning for each of the first contrastive prompt encoders to the nth contrastive prompt encoders.

11. In a model generation device that generates a vision foundation model based on multimodality prompts, At least one memory to store instructions, Includes at least one processor configured to execute the instructions, The processor performs the process of adding a first contrastive prompt encoder to an nth contrastive prompt encoder (where n is an integer of 2 or more, and the first to nth contrastive prompt encoders are pre-trained by contrastive learning of a first prompt encoder that encodes first modality data to an nth prompt encoder that encodes nth modality data, and the first modality data corresponds to the initial modality data) to the initial vision foundation model, which is pre-trained and includes an initial prompt encoder that encodes initial modality data;Furthermore, (II) when at least a portion of the initial modality learning data and the first modality learning data to the nth modality learning data corresponding to a specific domain is acquired, (i) a first subprocess inputs the initial modality learning data into the initial prompt encoder, and uses the initial prompt encoder to encode the initial modality learning data and generate an initial modality embedding vector, and (ii) inputs each of the first modality learning data to the nth modality learning data into the first contrastive prompt encoder to the nth contrastive prompt encoder, and uses the first contrastive prompt encoder to the nth contrastive prompt encoder to encode each of the first modality learning data to the nth modality learning data and generate a first modality embedding vector to the nth modality embedding vector A model generation device that performs a process of generating a vision foundation model based on multimodality prompts by: (iii) executing at least a portion of a second subprocess to generate each of the prompts; (iii) executing a third subprocess to fine-tune the first contrastive prompt encoder using distillation losses generated with reference to the initial modality embedding vector and the first modality embedding vector; and (iv) executing at least a portion of a fourth subprocess to fine-tune the first contrastive prompt encoder or the nth contrastive prompt encoder using alignment losses generated with reference to the first modality embedding vector or the nth modality embedding vector.

12. In the above process (II), The model generation apparatus according to claim 11, wherein the processor fine-tunes each of the first contrastive prompt encoders to the nth contrastive prompt encoders by a full fine-tuning method that updates each parameter of each of the first contrastive prompt encoders to the nth contrastive prompt encoders, or by a PEFT method that updates each parameter of each of the first specific intermediate layer to the nth specific intermediate layer corresponding to any one of the first intermediate layers to the nth intermediate layer included in each of the first contrastive prompt encoders to the nth contrastive prompt encoders.

13. In the above process (II), The model generation apparatus according to claim 12, wherein a first low-rank adapter to an nth low-rank adapter is provided in parallel with each of the first specific intermediate layer to the nth specific intermediate layer, the processor updates each parameter of the first low-rank adapter to the nth low-rank adapter, and then updates each parameter of the first specific intermediate layer to the nth specific intermediate layer using the updated parameters of the first low-rank adapter to the nth low-rank adapter.

14. In the above process (II), When each of the first modality learning data or the nth modality learning data is input to each of the first contrastive prompt encoder or the nth contrastive prompt encoder, each of the first contrastive prompt encoder or the nth contrastive prompt encoder (i) generates a first specific modality hidden state or the nth specific modality hidden state corresponding to each of the first modality learning data or the nth modality learning data through the first specific intermediate layer or the nth specific intermediate layer of each of the first contrastive prompt encoder or the nth contrastive prompt encoder (ii) the first contrastive The model generation apparatus according to claim 13, comprising: (iii) executing a subprocess for generating each of the first low-rank modality hidden states to the nth low-rank modality hidden states corresponding to each of the first modality learning data to the nth modality learning data through each of the first low-rank adapters to the nth low-rank adapters of the prompt encoder or the nth contrastive prompt encoder; and (iii) executing a subprocess for generating the first modality embedding vector by reference to the first specific modality hidden state and the first low-rank modality hidden state, or a subprocess for generating each of the nth modality embedding vector by reference to each of the nth specific modality hidden states and each of the nth low-rank modality hidden states.

15. In the above process (II), The process is a process of updating the respective first and second low-rank parameter matrices, the first and second low-rank parameter matrices, the first and second low-rank parameter matrices, the first and second low-rank parameter matrices, the first and second low-rank parameter matrices, the first and second low-rank parameter matrices, the first and second low-rank parameter matrices, the nth and second low-rank parameter matrices, the first and second low-rank adapters, the first and second low-rank adapters, the first and second low-rank adapters, the nth parameter matrices, the nth and second low-rank parameter matrices, the first and second low-rank adapters, the nth and second low-rank adapters, the first and second low-rank adapters, the first and second low-rank adapters, the nth and second low-rank parameter matrices, the first and(II) When it is detected that learning has been completed for each of the first contrastive prompt encoders to the nth contrastive prompt encoders, the model generation apparatus according to claim 13 further performs the process of generating each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders by referring to each of the first_1 low-rank parameter matrix and the first_2 low-rank parameter matrix to the n_1 low-rank parameter matrix and the n_2 low-rank parameter matrix, and updating each of the first intermediate layer parameter matrix to the nth intermediate layer parameter matrix corresponding to each of the first specific intermediate layer to the nth specific intermediate layer by referring to each of the first low-rank adapter parameter matrix to the nth low-rank adapter parameter matrix corresponding to each of the first contrastive prompt encoders to the nth contrastive prompt encoders;

16. In the above process (II), A model generation device according to claim 11, wherein a learning schedule is determined which includes the initial encoder input probability and the first modality target input probability to the nth modality target input probability corresponding to the initial prompt encoder and the first contrastive prompt encoder to the nth contrastive prompt encoder, respectively, and which includes the initial modality learning data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group, respectively, and the processor, by referring to the learning schedule, extracts at least a portion of the initial modality learning data candidate group and the first modality learning data candidate group to the nth modality learning data candidate group to acquire the initial modality learning data and the first modality learning data to the nth modality learning data, respectively, such that the ratio of the amount of data contained in the initial modality learning data and the first modality learning data to the nth modality learning data follows the initial encoder input probability and the first modality target input probability to the nth modality target input probability, respectively.

17. In the above process (II), (II3) The model generation apparatus according to claim 16, wherein the rate of change of the initial encoder input probability after updating is 0 or less, and the rate of change of each of the first modality target input probability to the nth modality target input probability after updating is determined to be 0 or more, and after the processor has trained each of the first contrastive prompt encoder to the nth contrastive prompt encoder, if the initial encoder input probability is greater than 0, the processor linearly updates each of the initial encoder input probability and the first modality target input probability to the nth modality target input probability included in the learning schedule, and when it detects that the updated initial encoder input probability is 0, it further executes a process to determine that the rate of change of each of the initial encoder input probability and the first modality target input probability to the nth modality target input probability after updating is 0.

18. In the above process (II), The model generation apparatus according to claim 17, wherein, with the updated rate of change of each of the first modality target input probability and the nth modality target input probability determined to be 0, the processor executes the second subprocess and the fourth subprocess for a predetermined number of additional learning times, and then terminates learning for each of the first contrastive prompt encoder and the nth contrastive prompt encoder.

19. In the above process (I), The model generation apparatus according to claim 11, characterized in that the distillation loss is the InfoNCE loss between the initial modality embedding vector and the first modality embedding vector, and the alignment loss is the sigmoid loss between each of the first modality embedding vectors and the nth modality embedding vector.

20. In the above process (II), The model generation apparatus according to claim 11, wherein the processor removes the initial prompt encoder from the vision foundation model after completing fine-tuning for each of the first contrastive prompt encoders to the nth contrastive prompt encoder.

Images