A method for associative continuous learning of a large-scale multimodal model and a learning device utilizing the same.

Abstract

This invention provides a method for associative continuous learning of a large-scale multimodal model and a learning device utilizing this method. [Solution] The method involves generating the j-th hidden state vector by calculating the (j-1)-th hidden state vector by referring to the m_j-th merged prompt vector, and repeatedly performing a training iteration to update at least a portion of the m_k-th transformer block or the m_1-th transformer block and the m_k-th local prompt generator or the m_1-th local prompt generator using the loss generated by referring to the k-th hidden state vector output from the m_k-th transformer block and the ground truth corresponding to any one of the training data.

Description

Technical Field

[0001] The present invention relates to a method for federated continual learning of a large multimodal model and a learning device using the same {FEDERATED CONTINUAL LEARNING METHOD FOR LARGE MULTIMODAL MODEL AND LEARNING DEVICE USING THE SAME}.

Background Art

[0002] A large language model (LLM) is a language model trained using large-scale data, which can process natural language like humans, answer complex questions, and execute various language tasks.

[0003] However, real-world data includes not only text but also data of various modalities such as images, audio, and videos.

[0004] Therefore, there is a need for a model that can not only understand text but also simultaneously understand and process data of various modalities, and thus, large multimodal models (LMM: Large Multimodal Model or MM-LLM: Multimodal-Large Language Model) that can process data of multiple modalities simultaneously have been published.

[0005] These large-scale multimodal models are designed to generate output for queries given visual input, and utilize architectures such as vision encoders that convert images to text, LLMs, and vision-language cross-modal connectors that connect the vision encoder and LLM. Examples include OpenAI's GPT-4, Google's Gemini, and LLaVA (Large Language and Vision Assistant), which has shown excellent ability to grasp context in photographs.

[0006] On the other hand, when it comes to training large-scale multimodal models, federated learning is suitable because it is necessary for data privacy and security, and continual learning is required for adaptation to new data.

[0007] Previous techniques related to associative continuous learning include the papers "Zhang et al., TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation, ETH Zurich & Sony AI, ICCV 2023", "Qi et al., Better Generative Replay for Continual Federated Learning, Adobe Research, ICLR 2023", "Bakman et al., Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning, USC, ICLR 2024", and "Wuerkaixi et al., Accurate Forgetting for Heterogeneous Federated Continual Learning, Tsinghua Univ, ICLR 2024".

[0008] According to the associative continuous learning presented in the above paper, the data used by clients in associative continuous learning is different from each other, but each client performs the same task.

[0009] However, in the real world, client downstream tasks are diverse, including VCR (Visual Commonsense Reasoning) tasks that predict the correct answer and provide a convincing reasoning path, VQA (Visual Question and Answering) tasks that provide questions and answers, and VD (Visual Dialog) tasks that provide conversations. Therefore, the associative continuous learning presented in the above paper has the problem of not being able to fully reflect this.

[0010] Furthermore, the papers "Wang et al., Learning to Prompt for Continual Learning, CVPR 2022" and "Wang et al., DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning, ECCV 2022" describe a continuous learning method based on a prompt pool, while the paper "Jung et al., Generating Instance-level Prompts for Rehearsal-free Continual Learning, ICCV 2023 Oral" describes a continuous learning method based on a prompt generator.

[0011] Prompt pool-based continuous learning can prevent catastrophic forgetting in continuous learning by using the task ID for the task the model is trying to perform to learn different prompts for each task. However, it has the drawbacks of requiring the task ID during training, not performing sufficient domain generalization, and not working properly with data that differs from the image distribution used during training.

[0012] Furthermore, while continuous learning based on prompt generators has the advantages of not requiring task IDs and achieving sufficient domain generalization, it has the drawback of catastrophic forgetting due to updated prompt generator parameters.

[0013] Therefore, the applicant seeks to propose an associative continuous learning method that does not require a task ID and can prevent catastrophic forgetting. [Overview of the project] [Problems that the invention aims to solve]

[0014] The purpose of this invention is to solve all of the problems of the prior art described above.

[0015] Another objective of the present invention is to provide a method for federated continuous learning of a large-scale multimodal model without using task IDs.

[0016] Furthermore, another objective of the present invention is to provide a method for associative continuous learning of large-scale multimodal models that can prevent catastrophic forgetting. [Means for solving the problem]

[0017] According to one embodiment of the present invention, in a method for federated continuous learning of a large-scale multimodal model, (a) a learning device for federated continuous learning of an m-th local large-scale multimodal model which is one of a first local large-scale multimodal model generated to perform a first task or an n-th local large-scale multimodal model generated to perform an n-th task (where n is an integer of 2 or more) by referring to a trained global large-scale multimodal model stored on a server, inputs at least one training data (the training data includes training natural language instruction data and training vision data) obtained from the m-th training dataset generated for federated continuous learning of the m-th local large-scale multimodal model into the m-th local large-scale multimodal model, and uses the m-th local large-scale multimodal model to (i) generate word tokens by embedding the training natural language instruction data through a text encoder and through a vision encoder (ii)(ii_1(ii)(iii (iii) The steps of (iii) merging the m_j global prompt vector with the m_j merged prompt vector to generate the m_j merged prompt vector, (iii) referencing the m_j merged prompt vector through the m_j transformer block of the large language model to compute the (j-1) hidden state vector to generate the j hidden state vector, (iv) repeating a learning iteration to update at least a portion of the m_k transformer block or the m_1 transformer block and the m_k local prompt generator or the m_1 local prompt generator using the loss generated by referencing the k hidden state vector output from the m_k transformer block and the ground truth corresponding to any one of the learning data; and (b) when the number of iterations of the learning iteration reaches a predetermined number and one learning round is completed,A method is provided which includes the step of the learning device transmitting at least a portion of the updated information of the m_1 local prompt generator to the m_k local prompt generator to the server so that the first global prompt generator to the k global prompt generator of the global large-scale multimodal model are updated by referring to at least a portion of the updated information of the first_1 local prompt generator to the first_k local prompt generator corresponding to the first local large-scale multimodal model or the updated information of the n_1 local prompt generator to the n_k local prompt generator corresponding to the n local large-scale multimodal model.

[0018] In one example, in step (a), when the learning device updates the m_k local prompt generator or the m_1 local prompt generator, it updates the parameters of the m_k local prompt generator or the m_1 local prompt generator, respectively, using the m_k EMA (Exponential Moving Average) corresponding to the gradient change amount of the m_k local prompt generator or the m_1 EMA corresponding to the gradient change amount of the m_1 local prompt generator.

[0019] In one example, in step (a), the learning device uses the m-local large-scale multimodal model to (ii-1) (ii-1-a) self-attention the embedding vector through the m_1 local attention layer of the m_1 local prompt generator to generate the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector; combine the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector to generate the m_1 local self-attention vector; generate the m_1 local value prompt vector from the m_1 local self-attention vector through the m_1 local value prompt generator of the m_1 local prompt generator; and combine the m_1 local self-attention vector through the m_1 local key prompt generator of the m_1 local prompt generator (ii-1-b) The first m_1 local key prompt vector is generated, the first m_1 local downsampling prompt vector is generated from the first m_1 local self-attention vector through the first m_1 local downsampling prompt generator of the first m_1 local prompt generator, (ii-1-b) the embedding vector is self-attentioned through the first global attention layer of the first global prompt generator to generate a first global value vector, a first global key vector, and a first global downsampling vector, the first global value vector, the first global key vector, and the first global downsampling vector are combined to generate a first global self-attention vector, and the first global value prompt vector is generated from the first global self-attention vector through the first global value prompt generator of the first global prompt generator,(ii-1-c) The first global key prompt vector is generated from the first global self-attention vector through the first global key prompt generator of the first global prompt generator, the first global downsampling prompt vector is generated from the first global self-attention vector through the first global downsampling prompt generator of the first global prompt generator, (ii-1-c) the m_1 local value prompt vector and the first global value prompt vector are merged to generate the m_1 merged value prompt vector, the m_1 local key prompt vector and the first global key prompt vector are merged to generate the m_1 merged key prompt vector, and the m_1 local downsampling prompt vector and the first global downsampling prompt vector are merged to generate the m_1 merged downsampling prompt vector The following is done: (iii_1) (iii-1-a) The (j-1) hidden state vector is self-attentioned through the m_j local attention layer of the m_j local prompt generator to generate the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector; the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector are combined to generate the m_j local self-attention vector; the m_j local value prompt vector is generated from the m_j local self-attention vector through the m_j local value prompt generator of the m_j local prompt generator; and the m_j local key prompt vector is generated from the m_j local self-attention vector through the m_j local key prompt generator of the m_j local prompt generator.(iii-1-b) The (j-1) hidden state vector is self-attentioned through the j global attention layer of the j global prompt generator to generate the j global value vector, the j global key vector, and the j global downsampling vector, and the j global value vector, the j global key vector, and the j global downsampling vector are combined to generate the j global self attention vector, and the j global value prompt vector is generated from the j global self attention vector through the j global value prompt generator of the j global prompt generator, and the j global prompt The generator is configured to generate the j-th global key prompt vector from the j-th global self-attention vector through the j-th global key prompt generator, to generate the j-th global downsampling prompt vector from the j-th global self-attention vector through the j-th global downsampling prompt generator of the j-th global prompt generator, (iii-1-c) to merge the m_j local value prompt vector and the j-th global value prompt vector to generate the m_j merged value prompt vector, to merge the m_j local key prompt vector and the j-th global key prompt vector to generate the m_j merged key prompt vector, and to merge the m_j local downsampling prompt vector and the j-th global downsampling prompt vector to generate the m_j merged downsampling prompt vector.

[0020] In one example, in step (a), the learning device uses the m-local large-scale multimodal model to calculate the embedding vector and generate the first hidden state vector in (ii_2) by referring to the m_1 merged value prompt vector, the m_1 merged key prompt vector, and the m_1 merged downsampling prompt vector through the m_1 transformer block, and in (iii-2), calculate the (j-1) hidden state vector and generate the j hidden state vector by referring to the m_j merged value prompt vector, the m_j merged key prompt vector, and the m_j merged downsampling prompt vector through the m_j transformer block.

[0021] In one example, in step (b), the learning device transmits the updated information of the m_1 local prompt generator to the m_k local prompt generator to the server, and updates the first global prompt generator to the k global prompt generator, using at least a portion of each of the first average parameter, which is the average or weighted average of the parameters of the first_1 local prompt generator or the n_1 local prompt generator, or the k average parameter, which is the average or weighted average of the parameters of the first_k local prompt generator or the n_k local prompt generator.

[0022] Furthermore, according to another embodiment of the present invention, a learning device for federated continuous learning of a large-scale multimodal model includes: a memory storing instructions for federated continuous learning of a large-scale multimodal model; and a processor that performs operations for federated continuous learning of the large-scale multimodal model in accordance with the instructions stored in the memory, wherein the processor (i) references a trained global large-scale multimodal model stored in a server to cause a m-th local large-scale multimodal model, which is one of a first local large-scale multimodal model generated to perform a first task or an n-th local large-scale multimodal model generated to perform an n-th task (where n is an integer of 2 or more), to federated continuous learning of the m-th local large-scale multimodal model (where the learning data includes learning natural language instruction data and learning The m-local large-scale multimodal model is input to the m-local large-scale multimodal model, and the m-local large-scale multimodal model is used to (i) embed the learning natural language indicator data through a text encoder to generate word tokens, embed the learning vision data through a vision encoder to generate visual features, transform the visual features through a projection layer to generate visual tokens, embed the word tokens and the visual tokens through an embedding layer to generate embedding vectors, and (ii)(ii_1) generate the m_1 local prompt vector from the embedding vectors through the m_1 local prompt generator, and a first global prompt generator (the first global prompt generator corresponds to the global large-scale multimodal model and, at the current time,(ii) The first m_1 global prompt vector is generated from the embedding vector through the first_1 local prompt generator corresponding to the first local large multimodal model (which is updated by referencing at least a part of the n_1 local prompt generator corresponding to the n local large multimodal model), the m_1 local prompt vector and the m_1 global prompt vector are merged to generate the m_1 merged prompt vector, (ii) the first hidden state vector is generated by referencing the m_1 merged prompt vector through the m_1 transformer block of the large language model and performing calculations on the embedding vector, (iii) the m_j local prompt vector is generated from the (j-1) hidden state vector through the (iii) (iii) local prompt generator (where j is an integer increasing from 2 to k, and k is an integer greater than or equal to 2), and the j global prompt generator (where j global prompt (iii_2) The prompt generator corresponds to the global large-scale multimodal model and is updated at the current time by referencing at least some of the first_j local prompt generators corresponding to the first local large-scale multimodal model to the n_j local prompt generators corresponding to the n local large-scale multimodal model) to generate the m_j global prompt vector from the (j-1) hidden state vector, (iii_2) the m_j local prompt vector and the m_j global prompt vector are merged to generate the m_j merged prompt vector, (iv) the (j-1) hidden state vector is computed to generate the j hidden state vector by referencing the m_j merged prompt vector through the m_j transformer block of the large-scale language model, and the loss generated by referencing the k hidden state vector output from the m_k transformer block and the ground truth corresponding to any one of the training data,A learning system is provided that performs a process of repeatedly performing a learning iteration to update at least a portion of the m_k transformer block to the m_1 transformer block and the m_k local prompt generator to the m_1 local prompt generator, and (II) when the number of repetitions of the learning iteration reaches a preset number and one learning round is completed, transmit at least a portion of the updated information of the m_1 local prompt generator to the m_k local prompt generator to the server, and perform a process of updating the first global prompt generator to the kth global prompt generator of the global large-scale multimodal model by referring to at least a portion of the updated information of the first_1 local prompt generator to the first_k local prompt generator corresponding to the first local large-scale multimodal model or the updated information of the n_1 local prompt generator to the n_k local prompt generator corresponding to the n local large-scale multimodal model.

[0023] In one example, when the processor updates the m_k local prompt generator or the m_1 local prompt generator in the (I) process, it updates the parameters of the m_k local prompt generator or the m_1 local prompt generator, respectively, using the m_k EMA (Exponential Moving Average) corresponding to the gradient change amount of the m_k local prompt generator or the m_1 EMA corresponding to the gradient change amount of the m_1 local prompt generator.

[0024] In one example, the processor, in process (I), uses the m-local large-scale multimodal model to generate the m-local value vector, the m-local key vector, and the m-local downsampling vector in (ii-1) (ii-1-a) by self-attentioning the embedding vector through the m-1 local attention layer of the m-1 local prompt generator; combines the m-1 local value vector, the m-1 local key vector, and the m-1 local downsampling vector to generate the m-1 local self-attention vector; generates the m-1 local value prompt vector from the m-1 local self-attention vector through the m-1 local value prompt generator of the m-1 local prompt generator; and generates the m-1 local self-attention vector through the m-1 local key prompt generator of the m-1 local prompt generator (ii-1-b) Generate the first m_1 local key prompt vector from the first m_1 local self-attention vector through the first m_1 local downsampling prompt generator of the m_1 local prompt generator, (ii-1-b) Self-attention the embedding vector through the first global attention layer of the first global prompt generator to generate the first global value vector, the first global key vector, and the first global downsampling vector, by combining the first global value vector, the first global key vector, and the first global downsampling vector to generate the first global self-attention vector, and generate the first global value prompt vector from the first global self-attention vector through the first global value prompt generator of the first global prompt generator,(ii-1-c) The first global key prompt vector is generated from the first global self-attention vector through the first global key prompt generator of the first global prompt generator, the first global downsampling prompt vector is generated from the first global self-attention vector through the first global downsampling prompt generator of the first global prompt generator, (ii-1-c) the m_1 local value prompt vector and the first global value prompt vector are merged to generate the m_1 merged value prompt vector, the m_1 local key prompt vector and the first global key prompt vector are merged to generate the m_1 merged key prompt vector, and the m_1 local downsampling prompt vector and the first global downsampling prompt vector are merged to generate the m_1 merged downsampling prompt vector The following are configured to generate, and in (iii_1), (iii-1-a) the (j-1) hidden state vector is self-attentioned through the m_j local attention layer of the m_j local prompt generator to generate the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector, by combining the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector to generate the m_j local self-attention vector, the m_j local value prompt vector is generated from the m_j local self-attention vector through the m_j local value prompt generator of the m_j local prompt generator, and the m_j local key prompt vector is generated from the m_j local self-attention vector through the m_j local key prompt generator of the m_j local prompt generator,(iii-1-b) Generate the (j-1) hidden state vector through the j global attention layer of the j global prompt generator to generate the j global value vector, the j global key vector, and the j global downsampling vector by combining them to generate the j global self-attention vector, and generate the j global value prompt vector from the j global self-attention vector through the j global value prompt generator of the j global prompt generator, and the j global prompt generator The neator generates the j-th global key prompt vector from the j-th global self-attention vector through the j-th global key prompt generator, generates the j-th global downsampling prompt vector from the j-th global self-attention vector through the j-th global downsampling prompt generator of the j-th global prompt generator, (iii-1-c) merges the m_j local value prompt vector and the j-th global value prompt vector to generate the m_j merged value prompt vector, merges the m_j local key prompt vector and the j-th global key prompt vector to generate the m_j merged key prompt vector, and merges the m_j local downsampling prompt vector and the j-th global downsampling prompt vector to generate the m_j merged downsampling prompt vector.

[0025] In one example, in the (I) process, the processor uses the m-th local large multi-modal model, and in the (ii_2), through the m_1-th transformer block, referring to the m_1-th merged value prompt vector, the m_1-th merged key prompt vector, and the m_1-th merged downsampling prompt vector, calculates the embedding vector to generate the first hidden state vector, and in the (iii-2), through the m_j-th transformer block, referring to the m_j-th merged value prompt vector, the m_j-th merged key prompt vector, and the m_j-th merged downsampling prompt vector, calculates the (j - 1)-th hidden state vector to generate the j-th hidden state vector.

[0026] In one example, in the (II) process, the processor transmits the updated information of the m_1-th local prompt generator to the m_k-th local prompt generator to the server, and uses at least a part of each of the first average parameter obtained by averaging or weighted averaging the parameters of the first_1-th local prompt generator to the n_1-th local prompt generator or the k-th average parameter obtained by averaging or weighted averaging the parameters of the first_k-th local prompt generator to the n_k-th local prompt generator to update each of the first global prompt generator to the k-th global prompt generator.

Advantages of the Invention

[0027] According to the present invention, it becomes possible to perform collaborative continuous learning of a large multi-modal model without using a task ID.

[0028] Furthermore, the present invention makes it possible to prevent catastrophic forgetting of large-scale multimodal models that have undergone associative continuous learning. [Brief explanation of the drawing]

[0029] The following drawings, attached for use in describing embodiments of the present invention, represent only a portion of the embodiments, and a person with ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary art") can obtain other drawings based on these drawings without performing any inventive work.

[0030] [Figure 1] Figure 1 is a schematic diagram showing a learning device for associative continuous learning of a large-scale multimodal model according to one embodiment of the present invention. [Figure 2] Figure 2 is a schematic diagram showing a system for federative continuous learning of a large-scale multimodal model according to one embodiment of the present invention. [Figure 3] Figure 3 is a schematic diagram illustrating a method for performing associative continuous learning on a large-scale multimodal model according to one embodiment of the present invention. [Figure 4] Figure 4 is a schematic diagram showing one prompt generator that constitutes a large-scale multimodal model in a method for federated continuous learning of a large-scale multimodal model according to one embodiment of the present invention. [Figure 5] Figure 5 is a schematic diagram showing one large-scale language model layer constituting a large-scale multimodal model in a method for federative continuous learning of a large-scale multimodal model according to one embodiment of the present invention. [Modes for carrying out the invention]

[0031] The detailed description of the present invention described below will be accompanied by illustrations of specific embodiments in which the present invention may be carried out. These embodiments are described in sufficient detail to enable a person of the ordinary skill to carry out the present invention. It should be understood that the various embodiments of the present invention are different from one another but do not need to be mutually exclusive. For example, certain shapes, structures and characteristics described herein can be modified and implemented from one embodiment to another without departing from the spirit and scope of the present invention. It should also be understood that the position or arrangement of individual components in each embodiment can be modified without departing from the spirit and scope of the present invention. Therefore, the detailed description described below should not be taken as restrictive, and the scope of the present invention should be understood to include the scope described in the claims and all equivalent scopes. In the drawings, similar reference numerals refer to identical or similar functions in various aspects.

[0032] In the following, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that a person with ordinary skill in the art to which the present invention pertains can easily implement the present invention.

[0033] Figure 1 schematically shows a learning device for federated continuous learning of a large-scale multimodal model according to one embodiment of the present invention. Referring to Figure 1, the learning device 1000_m may include a memory 1100 that stores instructions for federated continuous learning of a large-scale multimodal model, and a processor 1200 that performs operations for federated continuous learning of a large-scale multimodal model according to the instructions stored in the memory 1100.

[0034] Specifically, the learning device 1000_m may, but is not limited to, achieve desired system performance by utilizing a combination of a typical computing device (e.g., a device that may include a computer processor, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that enable the computing device to function in a particular manner).

[0035] Furthermore, the processor of a computing device may include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and a data bus. The computing device may also further include an operating system and software components for applications that perform specific purposes.

[0036] However, this does not exclude cases where the computing device includes an integrated processor, which is a medium, processor, and memory integrated for carrying out the present invention.

[0037] Meanwhile, the processor 1200 of the learning device 1000_m, in accordance with the instructions stored in the memory 1100, (i) refers to a trained global large-scale multimodal model stored in the server and inputs at least one training data (the training data includes training natural language instruction data and training vision data) obtained from the m-th training dataset for the federated continuous training of the m-th local large-scale multimodal model to the m-th local large-scale multimodal model, which is one of the first local large-scale multimodal model generated to perform the first task or the nth local large-scale multimodal model generated to perform the nth task (where n is an integer of 2 or more) to the m-th local large-scale multimodal model, and uses the m-th local large-scale multimodal model to (i) embed the training natural language instruction data through a text encoder to generate word tokens, and embed the training vision data through a vision encoder to generate visual features and projection (ii)(ii_1) The visual feature is transformed through a layer to generate a visual token, the word token and the visual token are embedded through an embedding layer to generate an embedding vector, (ii)(ii_1) the m_1 local prompt vector is generated from the embedding vector through the m_1 local prompt generator, the m_1 global prompt vector is generated from the embedding vector through a first global prompt generator (the first global prompt generator corresponds to the global large multimodal model and is updated at the current time by referring to at least a portion of the 1_1 local prompt generator corresponding to the first local large multimodal model or the n_1 local prompt generator corresponding to the n local large multimodal model), and the m_1 local prompt vector and the m_1 global prompt vector are merged to generate the m_1 merged prompt vector,(ii_2) The embedding vector is computed to generate a first hidden state vector by referencing the m_1 merged prompt vector through the m_1 transformer block of the large language model, (iii)(iii_1) The m_j local prompt vector is generated from the (j-1) hidden state vector through the m_j local prompt generator (where j is an integer increasing from 2 to k, and k is an integer greater than or equal to 2), and the m_j global prompt vector is generated from the (j-1) hidden state vector through the j global prompt generator (where the j global prompt generator corresponds to the global large multimodal model and is in an updated state at the current time by referencing at least a portion of the first_j local prompt generator corresponding to the first local large multimodal model or the n_j local prompt generator corresponding to the n local large multimodal model), and the m_j local prompt vector and the m_j global prompt vector (iii_2) The process of generating a merged prompt vector by merging the prompt vector and the (m_j) merged prompt vector by referencing the (j-1) hidden state vector through the (m_j) transformer block of the large-scale language model to generate the (j) hidden state vector by calculating the (j-1) hidden state vector, (iv) the process of repeating a learning iteration in which the m_k transformer block to the m_1 transformer block and at least a part of the m_k local prompt generator to the m_1 local prompt generator are updated using the loss generated by referencing the k hidden state vector output from the m_k transformer block and the ground truth corresponding to any one of the learning data, and (II) when the number of iterations of the learning iteration reaches a predetermined number and one learning round is completed, at least a part of the updated information of the m_1 local prompt generator to the m_k local prompt generator is transmitted to the server.A process can be executed to update the first global prompt generator to the kth global prompt generator of the global large-scale multimodal model by referring to at least a portion of the updated information of the first_1 local prompt generator to the first_k local prompt generator corresponding to the first local large-scale multimodal model, or the updated information of the n_1 local prompt generator to the n_k local prompt generator corresponding to the n local large-scale multimodal model.

[0038] A method for performing associative continuous learning of a large-scale multimodal model using a learning device according to one embodiment of the present invention configured in this manner will be explained below with reference to Figures 2 to 5.

[0039] First, referring to Figure 2, each of the first client 200_1 through the nth client 200_n can refer to the trained global large-scale multimodal model stored in the server 100 to generate a first local large-scale multimodal model for executing the first task or an nth local large-scale multimodal model for executing the nth task. The nth may be an integer of 2 or more. In this case, the local large-scale multimodal model may include a global prompt generator and a local prompt generator.

[0040] Then, at least some of the first client 200_1 through the nth client 200_n can use their own training datasets to train their own local large-scale multimodal models to perform their own tasks, and can transmit updated information of the local prompt generators included in their trained large-scale multimodal models to the server. The server 100 then updates the global prompt generators included in the global large-scale multimodal model by referring to the updated information of the local prompt generators received from the clients, i.e., the information of the abbreviated parameters, and transmits the updated information of the global prompt generators to the first client 200_1 through the nth client 200_n, so that the first client 200_1 through the nth client 200_n can update their respective global prompt generators included in their respective local large-scale multimodal models.

[0041] In this case, each of the first client 200_1 through the nth client 200_n can perform federated continuous learning of the first local large-scale multimodal model or the nth local large-scale multimodal model, respectively, using the first learning device or the nth learning device, respectively, which corresponds to the first client 200_1, and utilizing the respective first learning dataset or the nth learning dataset.

[0042] In this state, the method by which the mth client, which owns the mth local large-scale multimodal model (either the first local large-scale multimodal model or the nth local large-scale multimodal model), uses the mth learning dataset it owns to perform associative continuous learning of the mth local large-scale multimodal model through the learning device, i.e., the mth learning device 1000_m, is explained below with reference to Figure 3. Although the explanation will focus on the mth learning device 1000_m, the operation of other learning devices of other clients is similar, so a detailed explanation will be omitted.

[0043] The mth learning device 1000_m, which corresponds to the mth client that owns the mth local large-scale multimodal model, can input at least one training data obtained from the mth training dataset generated for federated continuous learning of the mth local large-scale multimodal model into the mth local large-scale multimodal model. In this case, the training data may include training natural language referential data and training vision data.

[0044] Then, the mth learning device 1000_m of the mth local large-scale multimodal model can generate word tokens by embedding learning natural language instruction data through the text encoder 311, generate visual features by embedding learning vision data through the vision encoder 312, and generate visual tokens by transforming the visual features through the projection layer 313.

[0045] In this case, the vision encoder 312 can use various encoders such as NFNet-F6, ViT (Vision Transformer), CLIP ViT, and Eva-CLIP ViT, and the projection layer 313 can be implemented as a linear transformation (Linear Projector) or MLP (Multi-Layer Perception), such as the projection matrix in LLaVA, and can also be implemented as a complex scheme such as Cross-attention, Q-Former, or P-Former.

[0046] Subsequently, the mth learning device 1000_m of the mth local large-scale multimodal model can generate embedding vectors by embedding word tokens and visual tokens through the embedding layer 314.

[0047] Next, the mth learning device 1000_m of the mth local large-scale multimodal model can input the embedding vector to the mth local large-scale language model so that the mth local large-scale language model can perform the mth task on the embedding vector. In this case, the mth local large-scale language model can include the m_1 large-scale language model layer 320_1 to the m_k large-scale language model layer 320_k. Each of the m_1 large-scale language model layer 320_1 to the m_k large-scale language model layer 320_k is implemented to have the same configuration including a local prompt generator, a global prompt generator, and a transformer block, but their parameters can be set differently from each other.

[0048] In other words, the mth learning device 1000_m of the mth local large-scale multimodal model can generate the mth local prompt vector from the embedding vector through the mth local prompt generator 321_1 of the mth_1 large-scale language model layer 320_1, and generate the mth global prompt vector from the embedding vector through the first global prompt generator 322_1. Then, it can merge the mth local prompt vector and the mth global prompt vector to generate the mth merged prompt vector, and the merging of the mth local prompt vector and the mth global prompt vector can be performed by a weighted sum operation or an average operation.

[0049] In this case, the m_1 local prompt generator 321_1 and the first global prompt generator 322_1 are implemented with the same structure, but have different parameters, and the first global prompt generator 322_1 may be updated at the current time by referencing at least a portion of the 1_1 local prompt generator corresponding to the first local large multimodal model or the n_1 local prompt generator corresponding to the n local large multimodal model.

[0050] As an example, referring to Figure 4, the first m_1 local prompt generator 321_1 and the first global prompt generator 322_1 may include an attention layer 410 that generates a self-attention vector from an embedding vector, a value prompt generator 420 that generates a value prompt vector from the self-attention vector, a key prompt generator 430 that generates a key prompt vector from the self-attention vector, and a downsampling prompt generator 440 that generates a downsampling prompt vector from the self-attention vector. In this case, the attention layer 410 can be implemented as a Scaled Dot-Product Attention layer or a Multi-Head Attention layer as described in the "Attention is All You Need" paper presented by Google Brain at NIPS in 2017, and the value prompt generator 420, key prompt generator 430, and downsampling prompt generator 440 can each be implemented as an encoder-decoder pair.

[0051] As a result, the mth learning device 1000_m of the mth local large-scale multimodal model can self-attention the embedding vector through the m_1 local attention layer of the m_1 local prompt generator to generate the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector, and then combine the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector to generate the m_1 local self-attention vector. Furthermore, the m-th learning device 1000_m of the m-th local large-scale multimodal model can generate the m-th local value prompt vector from the m-th local self-attention vector through the m-th local value prompt generator of the m-th local prompt generator, generate the m-th local key prompt vector from the m-th local self-attention vector through the m-th local key prompt generator of the m-th local prompt generator, and generate the m-th local downsampling prompt vector from the m-th local self-attention vector through the m-th local downsampling prompt generator of the m-th local prompt generator.

[0052] Furthermore, the mth learning device 1000_m of the mth local large-scale multimodal model can generate a first global value vector, a first global key vector, and a first global downsampling vector by self-attentioning the embedding vector through the first global attention layer of the first global prompt generator, and then combine the first global value vector, the first global key vector, and the first global downsampling vector to generate a first global self-attention vector. Then, the mth learning device 1000_m of the mth local large-scale multimodal model can generate a first global value prompt vector from the first global self-attention vector through the first global value prompt generator of the first global prompt generator, generate a first global key prompt vector from the first global self-attention vector through the first global key prompt generator of the first global prompt generator, and generate a first global downsampling prompt vector from the first global self-attention vector through the first global downsampling prompt generator of the first global prompt generator.

[0053] Subsequently, the mth learning device 1000_m of the mth local large-scale multimodal model can merge the m_1 local value prompt vector and the first global value prompt vector to generate the m_1 merged value prompt vector, merge the m_1 local key prompt vector and the first global key prompt vector to generate the m_1 merged key prompt vector, and merge the m_1 local downsampling prompt vector and the first global downsampling prompt vector to generate the m_1 merged downsampling prompt vector.

[0054] Referring again to Figure 3, the mth learning device 1000_m of the mth local large multimodal model can reference the mth-1 merged prompt vector through the mth-1 transformer block 324_1 of the mth-1 large language model layer 320_1 to compute the embedding vector and generate the first hidden state vector.

[0055] In other words, the mth learning unit 1000_m of the mth local large multimodal model can generate a first hidden state vector by referencing the mth-1 merged value prompt vector, the mth-1 merged key prompt vector, and the mth-1 merged downsampling prompt vector through the mth-1 transformer block 324_1 of the mth-1 large language model layer 320_1 to compute an embedding vector.

[0056] As an example, referring to Figure 5, the first m_1 transformer block 324_1 self-attentions the input vector, i.e., the input embedding vector, to generate the query vector Q, key vector K, and value vector V, and merges the key prompt vector l k After performing the dot product of the key vector K and the query vector Q, a softmax operation is applied, and the vector obtained by performing the dot product of the merged value prompt vector lv and the value vector V is summed to generate the summed vector z. This is then passed through an FFN (Feed-Forward Network) consisting of two linear layers (Dense) and one nonlinearity activation function (Nonlinearity), where the dimension of the z vector is increased in the first linear layer, and after passing through the activation function, the downsampled prompt vector l ff The hidden state vector can be generated by taking the inner product of the vector obtained through the activation function and compressing it with a second linear layer to restore it to its original dimension.

[0057] Figure 5 schematically shows the configuration of the m_1 transformer block 324_1 to the m_k transformer block 324_k included in the m_-local large-scale multimodal model. Each transformer block in the m_-local large-scale multimodal model can be implemented as a transformer described in the "Attention is All You Need" paper presented by Google Brain at NIPS in 2017, and can be implemented as a transformer encoder, a transformer decoder, or both. However, the present invention is not limited to this, and it can also be implemented as various models generated based on the transformer.

[0058] Referring again to Figure 3, the mth learning device 1000_m of the mth local large-scale multimodal model can repeat the process of inputting the first hidden state vector into the next transformer block to generate the next hidden state vector.

[0059] In other words, the mth learning unit 1000_m of the mth local large-scale multimodal model can generate the mth_j local prompt vector from the (j-1)th hidden state vector through the mth_j local prompt generator 321_j of the mth_j large-scale language model layer 320_j for each of the integers j increasing from 2 to k, and generate the mth_j global prompt vector from the (j-1)th hidden state vector through the jth global prompt generator 322_j. Then, it can merge the mth_j local prompt vector and the mth_j global prompt vector to generate the mth_j merged prompt vector.

[0060] In this case, the m_j local prompt generator 321_j and the j global prompt generator 322_j are implemented with the same structure, but have different parameters, and the j global prompt generator 322_j may be updated at the current time by referencing at least a portion of the first_j local prompt generator corresponding to the first local large multimodal model or the n_j local prompt generator corresponding to the n local large multimodal model.

[0061] More specifically, the m-th learning device 1000_m of the m-th local large-scale multimodal model can self-attention the (j-1)th hidden state vector through the m_j local attention layer of the m_j local prompt generator to generate the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector, and then combine the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector to generate the m_j local self-attention vector. Furthermore, the m-th learning device 1000_m of the m-th local large-scale multimodal model can generate the m-th local value prompt vector from the m-th local self-attention vector through the m-th local value prompt generator of the m-th local prompt generator, generate the m-th local key prompt vector from the m-th local self-attention vector through the m-th local key prompt generator of the m-th local prompt generator, and generate the m-th local downsampling prompt vector from the m-th local self-attention vector through the m-th local downsampling prompt generator of the m-th local prompt generator.

[0062] Furthermore, the mth learning device 1000_m of the mth local large-scale multimodal model can generate the jth global value vector, the jth global key vector, and the jth global downsampling vector by self-attentioning the (j-1)th hidden state vector through the jth global attention layer of the jth global prompt generator, and then combine the jth global value vector, the jth global key vector, and the jth global downsampling vector to generate the jth global self-attention vector. Then, the mth learning device 1000_m of the mth local large-scale multimodal model can generate the jth global value prompt vector from the jth global self-attention vector through the jth global value prompt generator of the jth global prompt generator, generate the jth global key prompt vector from the jth global self-attention vector through the jth global key prompt generator of the jth global prompt generator, and generate the jth global downsampling prompt vector from the jth global self-attention vector through the jth global downsampling prompt generator of the jth global prompt generator.

[0063] Subsequently, the m-th learning device 1000_m of the m-th local large-scale multimodal model can merge the m_j local value prompt vector and the j-th global value prompt vector to generate the m_j merged value prompt vector, merge the m_j local key prompt vector and the j-th global key prompt vector to generate the m_j merged key prompt vector, and merge the m_j local downsampling prompt vector and the j-th global downsampling prompt vector to generate the m_j merged downsampling prompt vector.

[0064] Then, the m-th learning unit 1000_m of the m-th local large-scale multimodal model can reference the m-j merged prompt vector through the m-j transformer block 324_j of the m-j large-scale language model layer 320_j to compute the (j-1) hidden state vector and generate the j-th hidden state vector.

[0065] In other words, the mth learning unit 1000_m of the mth local large multimodal model can generate the jth hidden state vector by referencing the m_j merged value prompt vector, the m_1 merged key prompt vector, and the m_1 merged downsampling prompt vector through the m_j transformer block 324_1 of the m_j large language model layer 320_j to compute the (j-1)th hidden state vector.

[0066] Subsequently, when the kth hidden state vector is output from the m_k transformer block 324_k, the mth learning unit 1000_m of the mth local large-scale multimodal model can perform a learning iteration that updates at least some of the m_k transformer block 324_k to the m_1 transformer block 324_1 and the m_k local prompt generator 321_k to the m_1 local prompt generator 321_1, using the loss generated by referencing the kth hidden state vector and the ground truth corresponding to any one of the training data.

[0067] In this case, when the m-th learning device 1000_m of the m-th local large-scale multimodal model updates the m-th local prompt generator 321_k or the m-1st local prompt generator 321_1, it can update the parameters of the m-th local prompt generator 321_k or the m-1st local prompt generator 321_1, respectively, by utilizing the m-k EMA (Exponential Moving Average) corresponding to the gradient change of the m-th local prompt generator 321_k or the m-1st EMA corresponding to the gradient change of the m-1st local prompt generator 321_1. In other words, the m-th learning device 1000_m of the m-th local large-scale multimodal model can update the m-th local prompt generator 321_k or the m-1st local prompt generator 321_1 using algorithms that utilize EMA, such as the Adadelta algorithm or RMSprop.

[0068] In this manner, the m-th learning device 1000_m repeatedly performs learning iterations of the m-th local large-scale multimodal model. When the number of learning iterations reaches a preset number and one learning round is completed, at least a portion of the updated information from the m_1 local prompt generator 321_1 to the m_k local prompt generator 321_k is transmitted to the server. This allows the first global prompt generator to the k-th global prompt generator of the global large-scale multimodal model to be updated by referring to at least a portion of the updated information from the 1_1 local prompt generator to the 1_k local prompt generator corresponding to the first local large-scale multimodal model, or the updated information from the n_1 local prompt generator to the n_k local prompt generator corresponding to the n-th local large-scale multimodal model. At this time, the first global prompt generator and the kth global prompt generator can be updated using at least a portion of each of the first average parameter, which is the average or weighted average of the parameters of the first_1 local prompt generator or the parameters of the n_1 local prompt generator, or the kth average parameter, which is the average or weighted average of the parameters of the first_k local prompt generator or the parameters of the n_k local prompt generator.

[0069] In other words, referring to Figure 2, the mth client 200_m, which corresponds to the mth learning device 1000_m of the mth local large-scale multimodal model, can transmit updated information about the mth_* local prompt generator, i.e., information about the updated parameters of the mth_* local prompt generator, to the server 100 at the time the learning round is completed.

[0070] Then, server 100 can update the *Global Prompt Generator corresponding to the global large-scale multimodal model by utilizing at least a portion of the update information for the 1_*Local Prompt Generator corresponding to the 1st Local Large-Scale Multimodal Model of the 1st Client 200_1, which includes the updated information for the m_*Local Prompt Generator corresponding to the mth Local Large-Scale Multimodal Model of the mth Client 200_m, or the update information for the n_*Local Prompt Generator corresponding to the nth Local Large-Scale Multimodal Model of the nth Client 200_n. Server 100 can then transmit the updated update information for the *Global Prompt Generator, that is, information regarding the updated parameters of the *Global Prompt Generator, to the 1st Client 200_1 through the nth Client 200_n, so that each of the 1st Client 200_1 through the nth Client 200_n updates its respective local prompt generator.

[0071] In this case, when server 100 receives update information for the local prompt generator from any one of the first client 200_1 to the nth client 200_n, it can update the global prompt generator, or it can update the global prompt generator using the local prompt generator update information received at a predetermined time interval. Furthermore, server 100 can also update the global prompt generator when it receives update information for the local prompt generator from a predetermined number or more of the first client 200_1 to the nth client 200_n. However, the present invention is not limited thereto, and the global prompt generator can be updated by various methods.

[0072] The embodiments of the present invention described above are implemented in the form of program instructions that can be executed through various computer components and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or they may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0073] Although the present invention has been described above based on specific details such as concrete components, limited embodiments, and drawings, these are provided only to aid in a more overall understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention pertains can make various modifications and variations from this description.

[0074] Therefore, the concept of the present invention shall not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below shall also fall within the scope of the concept of the present invention. [Explanation of symbols]

[0075] 1000: m-th learning device 1100: Memory 1200: Processor

Claims

1. In a method for associative continuous learning of large-scale multimodal models, (a) A learning device for federated continuous learning of an m-th local large-scale multimodal model, which is one of a first local large-scale multimodal model generated to perform a first task or an nth local large-scale multimodal model generated to perform an nth task (where n is an integer of 2 or more), by referring to a trained global large-scale multimodal model stored on a server, inputs at least one training data (the training data includes training natural language instruction data and training vision data) obtained from the m-th training dataset generated for federated continuous learning of the m-th local large-scale multimodal model into the m-th local large-scale multimodal model, and uses the m-th local large-scale multimodal model to (i) generate word tokens by embedding the training natural language instruction data through a text encoder, and generate visual features by embedding the training vision data through a vision encoder and project the visual features through a projection layer - is converted to generate a visual token, the word token and the visual token are embedded through an embedding layer to generate an embedding vector, (ii) (iii_1) the m_1 local prompt vector is generated from the embedding vector through the m_1 local prompt generator, the m_1 global prompt vector is generated from the embedding vector through a first global prompt generator (the first global prompt generator corresponds to the global large multimodal model and is updated at the current time by referring to at least a portion of the first_1 local prompt generator corresponding to the first local large multimodal model or the n_1 local prompt generator corresponding to the n local large multimodal model), the m_1 local prompt vector and the m_1 global prompt vector are merged to generate the m_1 merged prompt vector,(ii_2) The embedding vector is computed to generate a first hidden state vector by referencing the m_1 merged prompt vector through the m_1 transformer block of the large language model, (iii)(iii_1) The m_j local prompt vector is generated from the (j-1) hidden state vector through the m_j local prompt generator (where j is an integer increasing from 2 to k, and k is an integer greater than or equal to 2), and the m_j global prompt vector is generated from the (j-1) hidden state vector through the j global prompt generator (where the j global prompt generator corresponds to the global large multimodal model and is in an updated state at the current time by referencing at least a portion of the first_j local prompt generator corresponding to the first local large multimodal model to the n_j local prompt generator corresponding to the n local large multimodal model), and the m_j local prompt vector and (iii_2) The steps of the learning iteration are repeated, in which the m_k transformer block to the m_1 transformer block and the m_k local prompt generator to the m_k local prompt generator are updated by referencing the m_j merged prompt vector through the m_j transformer block of the large-scale language model to calculate the (j-1) hidden state vector to generate the j-th hidden state vector, and (iv) the loss generated by referencing the k-th hidden state vector output from the m_k transformer block and the ground truth corresponding to any one of the training data, and the steps of the learning iteration being repeated, (b) When the number of iterations of the learning iteration reaches a predetermined number and one learning round is completed, the learning device transmits at least a portion of the updated information of the m_1 local prompt generator to the m_k local prompt generator to the server so that the first global prompt generator to the k global prompt generator of the global large-scale multimodal model are updated by referring to at least a portion of the updated information of the first_1 local prompt generator to the first_k local prompt generator corresponding to the first local large-scale multimodal model or the updated information of the n_1 local prompt generator to the n_k local prompt generator corresponding to the n local large-scale multimodal model; A method that includes this.

2. In step (a) above, The method according to claim 1, wherein when the learning device updates the m_k local prompt generator or the m_1 local prompt generator, it updates the parameters of the m_k local prompt generator or the m_1 local prompt generator, respectively, using the m_k EMA (Exponential Moving Average) corresponding to the gradient change amount of the m_k local prompt generator or the m_1 EMA corresponding to the gradient change amount of the m_1 local prompt generator.

3. In step (a) above, The learning device uses the m-local large-scale multimodal model, In (ii-1) above, (ii-1-a) the embedding vector is self-attentioned through the m_1 local attention layer of the m_1 local prompt generator to generate the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector; the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector are combined to generate the m_1 local self-attention vector; the m_1 local value prompt vector is generated from the m_1 local self-attention vector through the m_1 local value prompt generator of the m_1 local prompt generator; and the m_1 local key prompt vector is generated from the m_1 local self-attention vector through the m_1 local key prompt generator of the m_1 local prompt generator. (ii-1-b) The embedding vector is self-attentioned through the first global attention layer of the first global prompt generator to generate a first global value vector, a first global key vector, and a first global downsampling vector, the first global value vector, the first global key vector, and the first global downsampling vector are combined to generate a first global self-attention vector, and the first global value prompt vector is generated from the first global self-attention vector through the first global value prompt generator of the first global prompt generator,(ii-1-c) The first global key prompt vector is generated from the first global self-attention vector through the first global key prompt generator of the first global prompt generator, the first global downsampling prompt vector is generated from the first global self-attention vector through the first global downsampling prompt generator of the first global prompt generator, (ii-1-c) the m_1 local value prompt vector and the first global value prompt vector are merged to generate the m_1 merged value prompt vector, the m_1 local key prompt vector and the first global key prompt vector are merged to generate the m_1 merged key prompt vector, the m_1 local downsampling prompt vector and the first global downsampling prompt vector are merged to generate the m_1 merged downsampling prompt vector, In (iii_1) above, (iii-1-a) the (j-1) hidden state vector is self-attentioned through the m_j local attention layer of the m_j local prompt generator to generate the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector; the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector are combined to generate the m_j local self-attention vector; the m_j local value prompt vector is generated from the m_j local self-attention vector through the m_j local value prompt generator of the m_j local prompt generator; and the m_j local key prompt vector is generated from the m_j local self-attention vector through the m_j local key prompt generator of the m_j local prompt generator. The following is done: the first m_j local downsampling prompt vector is generated from the m_j local self-attention vector through the first m_j local downsampling prompt generator of the m_j local prompt generator; (iii-1-b) the (j-1) hidden state vector is self-attentioned through the j global attention layer of the j global prompt generator to generate the j global value vector, the j global key vector, and the j global downsampling vector; the j global value vector, the j global key vector, and the j global downsampling vector are combined to generate the j global self-attention vector; and the j global value prompt vector is generated from the j global self-attention vector through the j global value prompt generator of the j global prompt generator.The method according to claim 1, wherein the j-th global key prompt vector is generated from the j-th global self-attention vector through the j-th global key prompt generator of the j-th global prompt generator, the j-th global downsampling prompt vector is generated from the j-th global self-attention vector through the j-th global downsampling prompt generator of the j-th global prompt generator, (iii-1-c) the m-j local value prompt vector and the j-th global value prompt vector are merged to generate the m-j merged value prompt vector, the m-j local key prompt vector and the j-th global key prompt vector are merged to generate the m-j merged key prompt vector, and the m-j local downsampling prompt vector and the j-th global downsampling prompt vector are merged to generate the m-j merged downsampling prompt vector.

4. In step (a) above, The learning device uses the m-local large-scale multimodal model, In (ii_2) above, the first hidden state vector is generated by calculating the embedding vector by referencing the m_1 merged value prompt vector, the m_1 merged key prompt vector, and the m_1 merged downsampling prompt vector through the m_1 transformer block, The method according to claim 3, wherein in (iii-2), the (j-1) hidden state vector is calculated to generate the j hidden state vector by referring to the m_j merged value prompt vector, the m_j merged key prompt vector, and the m_j merged downsampling prompt vector through the m_j transformer block.

5. In step (b) above, The method according to claim 1, wherein the learning device transmits updated information of the m_1 local prompt generator to the m_k local prompt generator to the server, and updates the first global prompt generator to the k global prompt generator, each using at least a portion of each of the first average parameter obtained by averaging or weighting the parameters of the first_1 local prompt generator or the n_1 local prompt generator, and the k average parameter obtained by averaging or weighting the parameters of the first_k local prompt generator or the n_k local prompt generator.

6. In a learning device that performs associative continuous learning of a large-scale multimodal model, Memory containing instructions for federated continuous learning of a large-scale multimodal model; and A processor that performs operations for federated continuous learning of the large-scale multimodal model in accordance with the instructions stored in the memory; Includes, The processor (i) references a trained global large-scale multimodal model stored in the server to cause a federated continuous learning of the m-th local large-scale multimodal model, which is one of the first local large-scale multimodal model generated to perform the first task or the nth local large-scale multimodal model generated to perform the nth task (where n is an integer of 2 or more), by inputting at least one training data (the training data includes training natural language indicator data and training vision data) obtained from the m-th training dataset for federated continuous learning of the m-th local large-scale multimodal model, and uses the m-th local large-scale multimodal model to (i) generate word tokens by embedding the training natural language indicator data through a text encoder, generate visual features by embedding the training vision data through a vision encoder, and project the visual features through a projection layer. Convert to generate a visual token, embed the word token and the visual token through an embedding layer to generate an embedding vector, (ii)(iii_1) generate the m_1 local prompt vector from the embedding vector through the m_1 local prompt generator, generate the m_1 global prompt vector from the embedding vector through a first global prompt generator (the first global prompt generator corresponds to the global large multimodal model and is updated at the current time by referring to at least a portion of the first_1 local prompt generator corresponding to the first local large multimodal model to the n_1 local prompt generator corresponding to the n local large multimodal model), and merge the m_1 local prompt vector and the m_1 global prompt vector to generate the m_1 merged prompt vector,(ii_2) The embedding vector is computed to generate a first hidden state vector by referencing the m_1 merged prompt vector through the m_1 transformer block of the large language model, (iii)(iii_1) The m_j local prompt vector is generated from the (j-1) hidden state vector through the m_j local prompt generator (where j is an integer increasing from 2 to k, and k is an integer greater than or equal to 2), and the m_j global prompt vector is generated from the (j-1) hidden state vector through the j global prompt generator (where the j global prompt generator corresponds to the global large multimodal model and is in an updated state at the current time by referencing at least a portion of the first_j local prompt generator corresponding to the first local large multimodal model or the n_j local prompt generator corresponding to the n local large multimodal model), and the m_j local prompt vector and the m_j global prompt (ii) The process of repeating a learning iteration to update at least a portion of the m_k transformer block or the m_1 transformer block and the m_k local prompt generator or the m_1 local prompt generator by referencing the m_j merged prompt vector through the m_j transformer block of the large-scale language model and calculating the (j-1) hidden state vector to generate the j-th hidden state vector, (iv) the process of updating at least a portion of the m_k transformer block or the m_1 transformer block and the m_k local prompt generator or the m_1 local prompt generator by referencing the k-th hidden state vector output from the m_k transformer block and the ground truth corresponding to any one of the learning data, and (ii) when the number of iterations of the learning iteration reaches a predetermined number and one learning round is completed, at least a portion of the updated information of the m_1 local prompt generator or the m_k local prompt generator is transmitted to the server.A learning device that performs a process to update the first global prompt generator to the k global prompt generator of the global large-scale multimodal model by referring to at least a portion of the updated information of the first_1 local prompt generator to the first_k local prompt generator corresponding to the first local large-scale multimodal model, or the updated information of the n_1 local prompt generator to the n_k local prompt generator corresponding to the n local large-scale multimodal model.

7. The aforementioned processor, The learning device according to claim 6, wherein in the process (I) above, when updating the m_k local prompt generator or the m_1 local prompt generator, the parameters of the m_k local prompt generator or the m_1 local prompt generator are updated using the m_k EMA (Exponential Moving Average) corresponding to the gradient change amount of the m_k local prompt generator or the m_1 EMA corresponding to the gradient change amount of the m_1 local prompt generator, respectively.

8. The aforementioned processor, In the process described in (I) above, using the m-local large-scale multimodal model, In (ii-1) above, (ii-1-a) the embedding vector is self-attentioned through the m_1 local attention layer of the m_1 local prompt generator to generate the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector; the m_1 local value vector, the m_1 local key vector, and the m_1 local downsampling vector are combined to generate the m_1 local self-attention vector; the m_1 local value prompt vector is generated from the m_1 local self-attention vector through the m_1 local value prompt generator of the m_1 local prompt generator; and the m_1 local key prompt vector is generated from the m_1 local self-attention vector through the m_1 local key prompt generator of the m_1 local prompt generator. (ii-1-b) The first global value vector, the first global key vector, and the first global downsampling vector are combined to generate a first global self-attention vector, and the first global value vector is generated from the first global self-attention vector through the first global value prompt generator of the first global prompt generator,(ii-1-c) The first global key prompt vector is generated from the first global self-attention vector through the first global key prompt generator of the first global prompt generator, the first global downsampling prompt vector is generated from the first global self-attention vector through the first global downsampling prompt generator of the first global prompt generator, (ii-1-c) the m_1 local value prompt vector and the first global value prompt vector are merged to generate the m_1 merged value prompt vector, the m_1 local key prompt vector and the first global key prompt vector are merged to generate the m_1 merged key prompt vector, the m_1 local downsampling prompt vector and the first global downsampling prompt vector are merged to generate the m_1 merged downsampling prompt vector, In (iii_1) above, (iii-1-a) the (j-1) hidden state vector is self-attentioned through the m_j local attention layer of the m_j local prompt generator to generate the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector, the m_j local value vector, the m_j local key vector, and the m_j local downsampling vector are combined to generate the m_j local self-attention vector, the m_j local value prompt vector is generated from the m_j local self-attention vector through the m_j local value prompt generator of the m_j local prompt generator, and the m_j local key prompt vector is generated from the m_j local self-attention vector through the m_j local key prompt generator of the m_j local prompt generator. The following are configured to generate the first m_j local downsampling prompt vector from the m_j local self-attention vector through the first m_j local downsampling prompt generator of the m_j local prompt generator, (iii-1-b) self-attention the (j-1) hidden state vector through the j global attention layer of the j global prompt generator to generate the j global value vector, the j global key vector, and the j global downsampling vector, by combining the j global value vector, the j global key vector, and the j global downsampling vector to generate the j global self-attention vector, and the j global value prompt vector from the j global self-attention vector through the j global value prompt generator of the j global prompt generator,The learning device according to claim 6, wherein the j-th global key prompt vector is generated from the j-th global self-attention vector through the j-th global key prompt generator of the j-th global prompt generator, the j-th global downsampling prompt vector is generated from the j-th global self-attention vector through the j-th global downsampling prompt generator of the j-th global prompt generator, (iii-1-c) the m-j local value prompt vector and the j-th global value prompt vector are merged to generate the m-j merged value prompt vector, the m-j local key prompt vector and the j-th global key prompt vector are merged to generate the m-j merged key prompt vector, and the m-j local downsampling prompt vector and the j-th global downsampling prompt vector are merged to generate the m-j merged downsampling prompt vector.

9. The aforementioned processor, In the process described in (I) above, using the m-local large-scale multimodal model, In (ii_2) above, the first hidden state vector is generated by calculating the embedding vector by referencing the m_1 merged value prompt vector, the m_1 merged key prompt vector, and the m_1 merged downsampling prompt vector through the m_1 transformer block, The learning device according to claim 8, wherein in (iii-2), the (j-1) hidden state vector is calculated and the j hidden state vector is generated by referring to the m_j merged value prompt vector, the m_j merged key prompt vector, and the m_j merged downsampling prompt vector through the m_j transformer block.

10. The aforementioned processor, The learning device according to claim 6, wherein in the process of (II), updated information of the m_1 local prompt generator to the m_k local prompt generator is transmitted to the server, and at least a portion of each of the first average parameter obtained by averaging or weighting the parameters of the first_1 local prompt generator or the n_1 local prompt generator, or the k average parameter obtained by averaging or weighting the parameters of the first_k local prompt generator or the n_k local prompt generator, is used to update each of the first global prompt generator to the k global prompt generator.

Images