A method for learning a large-scale multimodal video model through iterative self-retrospective judgment, and a learning device using the same.

Abstract

We train a large-scale multimodal video model through iterative self-retrospective judgments. [Solution] The method involves inputting a training preference dataset (number k) including the kth visual context, the kth_1 response, the kth_2 response, video data, and query data into a (k-1)th trained video large-scale multimodal model. The model then refers to the training preference dataset to determine one of the training kth_1 response and the training kth_2 response as the training kth preferred response, and the other response as the training kth disliked response. Training preference feedback data (number k) including the kth preferred response, the kth disliked response, video data, and query data is generated. The kth DPO loss is generated using the DPO loss, and the parameters of the (k-1)th trained video large-scale multimodal model are updated using the kth DPO loss to generate a kth trained video large-scale multimodal model.

Description

Technical Field

[0001] The present invention relates to a method for training a video large multimodal model through iterative self-retrospective judgment and a learning device using the same {METHOD FOR TRAINING VIDEO LARGE MULTIMODAL MODEL THROUGH ITERATIVE SELF-RETROSPECTIVE JUDGMENT AND LEARNING DEVICE USING THE SAME}.

Background Art

[0002] Large Language Models (LLMs) are used in solutions such as conversational chatbots like ChatGPT. However, in order to learn to appropriately answer by mimicking human preferences in a state where a large language model pre-trained and fine-tuned by supervised learning using a high-quality dataset suitable for a specific task is generated, a reward model is learned using learning data labeled with human preferences, and the fine-tuned large language model is reinforced using the labeled reward output from the learned reward model. The RLHF (Reinforcement Learning from Human Feedback) method is applied.

[0003] While such RLHF methods can maximize language generation and dialogue capabilities, they have a drawback: they require not only a large-scale language model but also another reward model for training, as well as sampling the output of the large-scale language model. This increases computational cost and complexity, leading to an unstable learning process. Therefore, these problems were solved by using the Direct Preference Optimization (DPO) method, which omits the reward modeling process and optimizes the large-scale language model using preference data, by using the large-scale language model itself in the same role as the reward model without using a separate reward model.

[0004] Recently, attempts have been made to apply this DPO method not only to large-scale language models but also to video large-scale multimodal models (VLMMs). Referring to Figure 1, in Figure 1(a), it can be seen that by repeatedly applying the DPO method to a video large-scale multimodal model, the length of the response output by the video large-scale multimodal model tends to gradually increase. At this time, as shown in Figure 1(b), when video data consisting of multiple frames and query data in text format are acquired by the video large-scale multimodal model, it can be seen that the length of the response output by the video large-scale multimodal model differs depending on the number of iterations when the DPO method is applied 1 time (i.e., 1 training iteration), 5 times (i.e., 5 training iterations), and 9 times (i.e., 9 training iterations). However, when the number of iterations of the DPO method is small (e.g., 1 and 5 times), the responses output by the video large-scale multimodal model include content shown in bold, i.e., content that is based on the video data. However, when the number of iterations of the DPO method is large (e.g., 9 times), a hallucination phenomenon occurs in which the responses output by the video large-scale multimodal model include content shown in underline, i.e., inappropriate content that is unrelated to the video data.

[0005] Therefore, there is a need for improvement measures to address the above-mentioned problems. [Overview of the project] [Problems that the invention aims to solve]

[0006] The purpose of this invention is to solve all of the problems mentioned above.

[0007] Furthermore, the present invention provides that (i) by inputting training video data and training query data into an initial video large-scale multimodal model, the initial video large-scale multimodal model outputs a training first visual context corresponding to the training video data, and two distinct training responses, the training 1_1 response and the training 1_2 response, corresponding to the training query data, and (ii) by inputting the training video data, training query data, training first visual context, training 1_1 response and training 1_2 response as a training first preference dataset into the initial video large-scale multimodal model, the initial video Another objective is to generate a first trained video large-scale multimodal model by using a large-scale multimodal model to refer to a training first preference dataset, determining one of the training first preference response and training first second response as the training first preference response, and (iii) generating a first DPO loss for the training first preference feedback data using the training video data, training query data, training first preference response and training first dislike response as training first preference feedback data, and updating the parameters of the initial video large-scale multimodal model using the first DPO loss.

[0008] Furthermore, the present invention provides that (i) training video data, training query data, and training (k-1)th visual context are input into a (k-1)th trained large-scale multimodal video model, so that the (k-1)th trained large-scale multimodal video model outputs a training (k)th visual context corresponding to the training video data, and two distinct training responses corresponding to the training query data, namely the training (k_1)th response and the training (k_2)th response; and (ii) the training video data, training query data, training (k)th visual context, training (k_1)th response and the training (k_2)th response are used as a training (k)th preference dataset, and the training (k)th preference dataset is input into a (k-1)th trained large-scale multimodal video model. (iii) Another objective is to generate a k-th trained video large-scale multimodal model by using the (k-1)th trained video large-scale multimodal model to refer to the training k-th preference dataset, determining one of the training k-1 response and training k-2 response as the training k-th preferred response and the other response as the training k-th disliked response, (iii) using the training video data, training query data, training k-th preferred response and training k-th disliked response as training k-th preference feedback data, generating the k-th DPO loss for the training k-th preference feedback data, and updating the parameters of the (k-1)th trained video large-scale multimodal model using the k-th DPO loss. [Means for solving the problem]

[0009] According to one embodiment of the present invention, in a method for learning a video large-scale multimodal model through iterative self-retrospective judgment, (a) a learning device inputs (i) training video data and training query data in text format into an initial video large-scale multimodal model, and uses the initial video large-scale multimodal model to generate a first training visual context for the training video data, and generates a first training response and a first training response corresponding to the training query data by referring to the training video data (the first training response is a response that is different from the first training response), thereby generating the first training visual context, the first training response, the first training response, (ii) Generate a first preference dataset for training, including the training video data and the training query data; (ii) Input the first preference dataset for training into the initial video large-scale multimodal model, and use the initial video large-scale multimodal model to determine, by referencing the first preference dataset for training, one of the training 1_1 response and the training 1_2 response as the first preference response for training, and the other response as the first dislike response for training, thereby generating first preference feedback data for training, including the first preference response for training, the first dislike response for training, the training video data and the training query data; (iii) Generate a first DPO loss for the first preference feedback data for training using DPO (Direct Preference Optimization); and generate a first trained video large-scale multimodal model by updating the parameters of the initial video large-scale multimodal model using the first DPO loss;and (b) the learning device inputs (i) the (k-1)th trained large-scale multimodal video model, the (k-1)th trained large-scale multimodal video model to reference the (k-1)th trained large-scale multimodal video model to generate the kth trained visual context for the learning video data by referencing the (k-1)th trained visual context and the learning video data, and to reference the learning video data to generate the kth trained response and the kth trained response corresponding to the learning query data (the kth trained response is different from the kth trained response) to generate the kth trained response and the kth trained response corresponding to the learning query data, the kth trained response and the kth trained response, the kth trained response and the kth trained response, the learning video data and the learning query data to generate the kth trained preference dataset A method is provided that includes the steps of: (ii) generating a training k-th preference dataset; inputting the training k-th preference dataset into the (k-1) trained video large-scale multimodal model; using the (k-1) trained video large-scale multimodal model, referencing the training k-th preference dataset to determine one of the training k-1 response and the training k-2 response as the training k-th preference response, and determining the other response as the training k-th dislike response; thereby generating training k-th preference feedback data including the training k-th preference response, the training k-th dislike response, the training video data, and the training query data; and (iii) generating a k-th DPO loss for the training k-th preference feedback data using the DPO; and generating a k-th trained video large-scale multimodal model by updating the parameters of the (k-1) trained video large-scale multimodal model using the k-th DPO loss.

[0010] In one example, in step (a), the learning device, in step (iii), uses the DPO to create a first DPO loss by referring to each of the first learning preference response and the first learning dislike response in the first learning preference feedback data and each of the learning reference preference response and learning reference dislike response output from the reference video large-scale multimodal model by inputting the learning video data and the learning query data into the reference video large-scale multimodal model corresponding to the initial video large-scale multimodal model. In step (b), the learning device, in step (iii), uses the DPO to create a kDPO loss by referring to each of the learning k preference response and the learning k dislike response in the k learning preference feedback data and each of the learning reference preference response and learning reference dislike response output from the reference video large-scale multimodal model by inputting the learning video data and the learning query data into the reference video large-scale multimodal model.

[0011] In one example, the reference video large-scale multimodal model is a supervised and trained model and is characterized by being a base model for generating the first DPO loss to the kDPO loss.

[0012] In one example, in step (a), the learning device sets the temperature hyperparameter of the initial video large-scale multimodal model to a specific temperature hyperparameter value greater than or equal to a preset threshold, inputs the training video data and training query data into the initial video large-scale multimodal model, and uses the initial video large-scale multimodal model to generate the training 1_1 response and the training 1_2 response, respectively, using the specific temperature hyperparameter value; and in step (b), the learning device sets the temperature hyperparameter of the (k-1) trained video large-scale multimodal model to the specific temperature hyperparameter value, inputs the training video data and training query data into the (k-1) trained video large-scale multimodal model, and uses the (k-1) trained video large-scale multimodal model to generate the training k_1 response and the training k_2 response, respectively, using the specific temperature hyperparameter value.

[0013] In one example, in step (a), the learning device generates a first embedding vector by embedding the training video data and the training query data in text format through the embedding layer using the initial video large multimodal model, and generates the first training visual context, the training 1_1 response and the training 1_2 response using the first embedding vector through the large language model; in step (b), the learning device generates a k-th embedding vector by embedding the training video data and the training query data through the embedding layer using the (k-1) trained video large multimodal model, and generates the training k-th visual context, the training k_1 response and the training k_2 response using the k-th embedding vector through the large language model.

[0014] Furthermore, according to another embodiment of the present invention, a learning device for learning a video large-scale multimodal model through iterative self-retrospective judgment includes: at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor (I)(i) inputs training video data and training query data in text format into an initial video large-scale multimodal model, causing the initial video large-scale multimodal model to generate a first training visual context for the training video data; and generates a first training response and a second training response (the first training response being different from the first training response) corresponding to the training query data by referring to the training video data, thereby enabling the learning (ii) Generate a first preference dataset for learning, which includes a first visual context, the first_1 learning response, the first_2 learning response, the video data for learning, and the query data for learning; (ii) Input the first preference dataset for learning into the initial video large multimodal model, which uses the initial video large multimodal model to determine which of the first_1 learning response and the first_2 learning response is the first preference response for learning, by referring to the first preference dataset for learning, and to determine which of the other response is the first disliked response for learning, thereby generating first preference feedback data for learning, which includes the first preference response for learning, the first disliked response for learning, the video data for learning, and the query data for learning; (iii) DPO (Direct A process to generate a first DPO loss for the first preference feedback data for training using Preference Optimization, and to generate a first trained video large-scale multimodal model by updating the parameters of the initial video large-scale multimodal model using the first DPO loss;(II)(i) Input the (k-1)th trained video large multimodal model into the (k-1)th trained video large multimodal model, and use the (k-1)th trained video large multimodal model to generate the kth trained visual context for the video data by referencing the (k-1)th trained visual context and the video data, and use the video data to generate the kth trained response and the kth trained response corresponding to the video data (the kth trained response is different from the kth trained response) to generate the kth trained response and the kth trained response corresponding to the video data by referencing the video data, thereby generating the kth trained preference dataset including the kth trained visual context, the kth trained response, the kth trained response, the video data, and the video data. A learning device is provided that performs the following process: (ii) inputting the k-th preference dataset for learning into the (k-1) trained large-scale multimodal video model, and using the (k-1) trained large-scale multimodal video model, referencing the k-th preference dataset for learning, determining one of the k-1 and k-2 responses as the k-th preferred response for learning, and determining the other response as the k-th disliked response for learning, thereby generating k-th preference feedback data for learning, which includes the k-th preference response for learning, the k-th disliked response for learning, the video data for learning, and the query data for learning; and (iii) generating the k-th DPO loss for the k-th preference feedback data using the DPO, and generating the k-th trained large-scale multimodal video model by updating the parameters of the (k-1) trained large-scale multimodal video model using the k-th DPO loss.

[0015] In one example, the processor generates the first DPO loss in (iii) of process (I) by using the DPO to refer to each of the first learning preference response and the first learning dislike response in the first learning preference feedback data and each of the learning reference preference response and learning reference dislike response output from the reference video large-scale multimodal model by inputting the learning video data and the learning query data into the reference video large-scale multimodal model corresponding to the initial video large-scale multimodal model, and in (iii) of process (II), the processor generates the k DPO loss in by using the DPO to refer to each of the k learning preference response and the k learning dislike response in the k learning preference feedback data and each of the learning reference preference response and learning reference dislike response output from the reference video large-scale multimodal model by inputting the learning video data and the learning query data into the reference video large-scale multimodal model.

[0016] In one example, the reference video large-scale multimodal model is a supervised and trained model and is characterized by being a base model for generating the first DPO loss to the kDPO loss.

[0017] In one example, the processor sets the temperature hyperparameter of the initial video large-scale multimodal model to a specific temperature hyperparameter value greater than or equal to a preset threshold in process (I), inputs the training video data and training query data into the initial video large-scale multimodal model, and uses the initial video large-scale multimodal model to generate the training 1_1 response and the training 1_2 response, respectively, using the specific temperature hyperparameter value. In process (II), the processor sets the temperature hyperparameter of the (k-1) trained video large-scale multimodal model to the specific temperature hyperparameter value, inputs the training video data and training query data into the (k-1) trained video large-scale multimodal model, and uses the (k-1) trained video large-scale multimodal model to generate the training k_1 response and the training k_2 response, respectively, using the specific temperature hyperparameter value.

[0018] In one example, the processor generates a first embedding vector in the (I) process by embedding the training video data and the training query data in text format through the embedding layer using the initial video large-scale multimodal model, and generates the first training visual context, the training 1_1 response and the training 1_2 response using the first embedding vector through the large-scale language model; and in the (II) process, the learning device generates a k-th embedding vector by embedding the training video data and the training query data through the embedding layer using the (k-1) trained video large-scale multimodal model, and generates the training k-th visual context, the training k_1 response and the training k_2 response using the k-th embedding vector through the large-scale language model. [Effects of the Invention]

[0019] The present invention provides the following: (i) By inputting training video data and training query data into an initial video large-scale multimodal model, the initial video large-scale multimodal model outputs a training first visual context corresponding to the training video data, and two distinct training responses, the training 1_1 response and the training 1_2 response, corresponding to the training query data; and (ii) By inputting the training video data, training query data, training first visual context, training 1_1 response and training 1_2 response as a training first preference dataset into the initial video large-scale multimodal model, the initial video (iii) Using a large-scale multimodal model, the system refers to the training first preference dataset to determine which of the training first_1 response and training first_2 response is the training first preference response, and the other response is determined to be the training first dislike response. (iii) The training video data, training query data, training first preference response, and training first dislike response are used as training first preference feedback data to generate a first DPO loss for the training first preference feedback data, and the parameters of the initial video large-scale multimodal model are updated using the first DPO loss to generate a first trained video large-scale multimodal model.

[0020] Furthermore, the present invention provides that (i) training video data, training query data, and training (k-1)th visual context are input to a (k-1)th trained large-scale multimodal video model so that the (k-1)th trained large-scale multimodal video model outputs a training (k)th visual context corresponding to the training video data, and two distinct training responses corresponding to the training query data, namely the training (k_1)th response and the training (k_2)th response; and (ii) the training video data, training query data, training (k)th visual context, training (k_1)th response and the training (k_2)th response are used as a training (k)th preference dataset, and the training (k)th preference dataset is used to a (k-1)th trained large-scale multimodal video model By inputting into Dell, the (k-1)th trained video large-scale multimodal model is used to refer to the training k-th preference dataset, determine one of the training k_1 response and training k_2 response as the training k-th preferred response, and determine the other response as the training k-th disliked response. (iii) The training video data, training query data, training k-th preferred response, and training k-th disliked response are used as training k-th preference feedback data to generate the k-th DPO loss for the training k-th preference feedback data, and update the parameters of the (k-1)th trained video large-scale multimodal model using the k-th DPO loss to generate the k-th trained video large-scale multimodal model. [Brief explanation of the drawing]

[0021] The following drawings, attached for use in describing embodiments of the present invention, represent only a portion of the embodiments, and a person with ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary art") can obtain other drawings based on these drawings without performing any inventive work.

[0022] [Figure 1] Figure 1 is a schematic diagram illustrating an example of a conventional technique that applies the DPO method to a large-scale multimodal video model. [Figure 2] Figure 2 is a diagram schematically showing a learning device for learning a video large-scale multimodal model through iterative self-reflective judgment according to an embodiment of the present invention. [Figure 3] Figure 3 is a diagram schematically showing a flowchart for learning a video large-scale multimodal model through iterative self-reflective judgment according to an embodiment of the present invention. [Figure 4a] Figure 4a is a diagram showing in detail a process for learning a video large-scale multimodal model through iterative self-reflective judgment according to an embodiment of the present invention. [Figure 4b] Figure 4b is a diagram showing in detail a process for learning a video large-scale multimodal model through iterative self-reflective judgment according to an embodiment of the present invention. [Figure 5] Figure 5 is a diagram schematically showing an example of a learning preference dataset for learning a video large-scale multimodal model according to an embodiment of the present invention. [Figure 6] Figure 6 is a diagram schematically showing an example of response results generated by inputting the same data into the learned video large-scale multimodal model and the conventional video large-scale multimodal model of the present invention, respectively, in a state where iterative learning for a predetermined number of times has been completed for the video large-scale multimodal model according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0023] The following detailed description of the present invention refers to the accompanying drawings which illustrate specific embodiments in which the present invention may be implemented, in order to clarify the object, technical solution and advantages of the present invention. These embodiments are described in sufficient detail so that an ordinary technician can implement the present invention.

[0024] Furthermore, in the detailed description and claims of the present invention, the word "including" and its variations are not intended to exclude other technical features, additions, components, or steps. To an ordinary person, some of the other purposes, advantages, and characteristics of the present invention will be apparent from this specification, and some from the practice of the present invention. The following examples and drawings are provided as illustrative examples and are not intended to limit the present invention.

[0025] Furthermore, the present invention encompasses all possible combinations of the embodiments shown herein. It should be understood that while the various embodiments of the present invention differ from one another, they do not necessarily have to be mutually exclusive. For example, certain shapes, structures, and characteristics described herein can be realized in other embodiments in relation to one embodiment without departing from the spirit and scope of the invention. It should also be understood that the position or arrangement of individual components within each disclosed embodiment can be modified without departing from the spirit and scope of the invention. Therefore, the detailed descriptions below should not be taken as restrictive, and the scope of the present invention is limited only by the appended claims, along with all equivalent scopes claimed by those claims, provided they are adequately described. Similar reference numerals in the drawings refer to identical or similar functions across various aspects.

[0026] In the following, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that persons with ordinary skill in the art to which the present invention pertains can easily implement the present invention.

[0027] Figure 2 schematically shows a learning device that learns a large-scale video multimodal model through iterative self-retrospective judgment according to one embodiment of the present invention.

[0028] Referring to Figure 2, the learning device 100 may include a memory 110 that stores instructions for learning a large-scale multimodal video model through iterative self-retrospective judgment, and a processor 120 that learns the large-scale multimodal video model through iterative self-retrospective judgment in response to the instructions stored in memory 110. In this case, the learning device 100 may include a PC (Personal Computer), a mobile computer, etc.

[0029] Specifically, the learning device 100 may achieve desired system performance using a combination of a typical computing device (for example, a device that may include a computer processor, memory, storage, input and output devices, and other components of conventional computing devices; electronic communication devices such as routers and switches; and electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that enable the computing device to function in a particular manner).

[0030] Furthermore, the processor of a computing device may include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and a data bus. The computing device may also further include an operating system and software components for applications that perform specific purposes.

[0031] However, this does not exclude cases where the computing device includes an integrated processor, which is a medium, processor, and memory integrated for carrying out the present invention.

[0032] Figure 3 is a schematic diagram illustrating a flow chart for learning a large-scale video multimodal model through iterative self-retrospective judgment according to one embodiment of the present invention.

[0033] First, the learning device 100 inputs the training video data and training query data in text format into an initial video large-scale multimodal model, and uses the initial video large-scale multimodal model to generate a first visual context for training the training video data. It then generates a first training response and a first-second training response corresponding to the training query data by referring to the training video data, thereby generating a first preference dataset for training that includes the first visual context for training, the first training response, the first-second training response, the training video data, and the training query data (S210_1).

[0034] With respect to the S210_1 process, referring to Figure 4a(a), the learning device 100 may be configured to generate two distinct responses corresponding to a learning query, namely the first learning response and the second learning response, by setting the temperature hyperparameter related to the output sensitivity of the initial video large-scale multimodal model 300_1 to a specific temperature hyperparameter value greater than or equal to a preset threshold.

[0035] For example, if the value of a specific temperature hyperparameter is close to 0 (e.g., 0.1), the output sensitivity to the initial video large-scale multimodal model 300_1 will be low. Conversely, if the value of a specific temperature hyperparameter is close to 1 (e.g., 0.7 or 0.8), the output sensitivity to the initial video large-scale multimodal model 300_1 will be high.

[0036] Therefore, the learning device 100 may input learning video data and learning query data into the initial video large-scale multimodal model 300_1, and use the initial video large-scale multimodal model 300_1 to generate the first learning response and the first learning response, respectively, using a specific temperature hyperparameter value. In other words, if the value of the specific temperature hyperparameter is low, close to 0, the first learning response and the first learning response may be the same / similar response, while if the value of the specific temperature hyperparameter is high, close to 1, the first learning response and the first learning response may be different responses. For example, in the present invention, the learning device 100 may set the specific temperature hyperparameter value to 0.7 so that the initial video large-scale multimodal model 300_1 generates different first learning responses, respectively, but is not limited to this.

[0037] At this time, the learning device 100 may (i) input the learning video data and learning query data into the initial video large-scale multimodal model 300_1 only once, and use the initial video large-scale multimodal model 300_1 to generate two different learning first_1 response and learning first_2 response, respectively; or (ii) input the learning video data and learning query data into the initial video large-scale multimodal model 300_1, and use the initial video large-scale multimodal model 300_1 to generate the learning first_1 response, and then input the learning video data and learning query data into the initial video large-scale multimodal model 300_1 again, and use the initial video large-scale multimodal model 300_1 to generate a learning first_2 response that is different from the learning first_1 response, but is not limited to these.

[0038] Furthermore, the learning device 100 may also be configured to use an initial video large-scale multimodal model 300_1 to embed training video data and training query data in text format through an embedding layer included in the initial video large-scale multimodal model 300_1 to generate a first embedding vector, and to use the first embedding vector to generate a first visual context for training, a first-first training response, and a second training response through a large-scale language model included in the initial video large-scale multimodal model 300_1, but is not limited to this configuration.

[0039] On the other hand, the learning device 100 can generate a first visual context for learning corresponding to the training video data using the initial video large-scale multimodal model 300_1. The first visual context for learning is data that describes what the training video data is or what state it is in, and can be used in the first iteration (i.e., processes S210_1 to S230_1 in Figure 3) to determine the preference for the first-first and second responses for learning, respectively, output from the initial video large-scale multimodal model 300_1. It can also be used in the second iteration to generate a second visual context for learning that is more specific than the first visual context for learning. An example of such a visual context will be shown later in Figure 5.

[0040] In this way, with the learning device 100 configured to generate a first visual context for learning, a first response for learning, and a first response for learning using the initial video large-scale multimodal model 300_1, it is possible to generate a first preference dataset 310_1 for learning, which includes the first visual context for learning, the first response for learning, the first response for learning, the first response for learning, and the first query data for learning, as well as the video data and query data used as input to the initial video large-scale multimodal model 300_1. At this time, the first preference dataset 310_1 for learning can be used as input data for input to the initial video large-scale multimodal model 300_1 in a subsequent process described later.

[0041] Next, referring to Figure 3, the learning device 100 inputs the first preference dataset for learning into the initial video large-scale multimodal model, and uses the initial video large-scale multimodal model to refer to the first preference dataset for learning and determine one of the first 1_1 response and the first 1_2 response for learning as the first preference response for learning, and the other response as the first disliked response for learning, thereby generating first preference feedback data for learning (S220_1) which includes the first preference response for learning, the first disliked response for learning, the video data for learning, and the query data for learning.

[0042] Referring to Figure 4a(b) for the S220_1 process, in the S210_1 process, the initial video large-scale multimodal model 300_1 was used to generate a first visual context for training and to generate the first-first and second training responses. However, in the S220_1 process, the initial video large-scale multimodal model 300_1 may be used as a model to judge the degree of preference for each of the first-first and second training responses by referring to the first preference dataset 310_1 for training. In other words, unlike conventional RLHF methods that use a reward model to determine human preference, the present invention reduces the amount of computation required for training compared to conventional RLHF methods using a reward model by having the initial video large-scale multimodal model 300_1 not only generate a visual context corresponding to the video data and two different responses corresponding to the query data, but also directly judge the degree of preference for the two different responses. This can be similarly applied to the subsequent process shown in Figure 4b, which will be described later. Furthermore, when the learning device 100 determines the first preferred response and the first disliked response for learning using the initial video large-scale multimodal model 300_1, it is also possible to improve the accuracy of preference determination by also referring to the first visual context for learning.

[0043] Therefore, the learning device 100 can generate first preference feedback data 320_1 for learning, which includes first preference responses and first dislike responses for learning determined by the initial video large-scale multimodal model 300_1, as well as video data and query data for learning included in the first preference dataset 310_1 for learning. The first preference feedback data 320_1 for learning can then be used to train the initial video large-scale multimodal model 300_1.

[0044] Next, referring again to Figure 3, the learning device 100 generates a first DPO loss for the first preference feedback data for learning using DPO (Direct Preference Optimization), and generates a first trained video large-scale multimodal model by updating the parameters of the initial video large-scale multimodal model using the first DPO loss (S230_1).

[0045] Referring to Figure 4a(c) for the S230_1 process, the learning device 100 can input the learning video data and learning query data contained in the first learning preference feedback data 320_1 into the reference video large-scale multimodal model 400 corresponding to the initial video large-scale multimodal model 300_1, and use the reference video large-scale multimodal model 400 to output the learning reference preference response and the learning reference dispatch response, respectively. Furthermore, the learning device 100 can use DPO to generate a first DPO loss by referring to the learning first preference response and the learning first dispatch response, respectively, contained in the first learning preference feedback data 320_1, and the learning reference preference response and the learning reference dispatch response, respectively. The large-scale multimodal reference video model 400 is a pre-supervised and trained model using training data for a given task, and can be used as a base model for generating the first DPO loss. The training reference preference response and training reference non-preference response can each be used as the Ground Truth (GT) for the training first preference response and training first non-preference response, respectively.

[0046] Specifically, the learning device 100 may input a first preferred response for learning, a first non-preferred response for learning, a reference preferred response for learning, and a reference non-preferred response for learning to the loss layer 330, and generate a first DPO loss by referring to the ratio of the first preferred response for learning to the reference preferred response through the loss layer 310, substituting reward modeling for preferred responses, and referring to the ratio of the first non-preferred response for learning to the reference non-preferred response for reward modeling for non-preferred responses.

[0047] Therefore, when the first DPO loss is generated through the loss layer 330, the learning device 100 can generate a first trained video large-scale multimodal model by updating the parameters of the initial video large-scale multimodal 300_1 using the first DPO loss.

[0048] Thus, once the first iteration is completed through the above process and the first trained large-scale multimodal video model is generated, the learning device 100 can perform the second iteration and subsequent iterations by repeating the same / similar process as the first iteration. To briefly explain the execution of the second and subsequent iterations, we can introduce a variable k, where k may be an integer increasing from 2 to n. With this in mind, the process in the kth iteration can be described as follows.

[0049] Referring to Figure 3, the learning device 100 inputs the (k-1)th visual context for learning, the video data for learning, and the query data for learning into the (k-1)th pre-trained large-scale multimodal video model. Using the (k-1)th pre-trained large-scale multimodal video model, it generates the kth visual context for learning for learning by referencing the (k-1)th visual context for learning and the video data for learning, and generates the k_1 and k_2 responses for learning corresponding to the query data for learning by referencing the video data. Thus, the learning device 100 can generate a kth preference dataset for learning that includes the kth visual context for learning, the k_1 response for learning, the k_2 response for learning, the video data for learning, and the query data for learning. (S210_k)

[0050] Referring to Figure 4b(a) regarding the S210_k process, in the first iteration (specifically, process 4a(a)), only training video data and training query data were input to the initial video large-scale multimodal model 300_1. However, in the kth iteration, the learning device 100 inputs not only the same training video data and training query data as in the first iteration, but also the (k-1)th training visual context acquired in the previous iteration, into the (k-1)th trained video large-scale multimodal model 300_k, which is different from the process in the first iteration.

[0051] In other words, when the learning device 100 generates the kth visual context for learning through the (k-1)th learning video large-scale multimodal model 300_k in the kth iteration, it may refer to the (k-1)th visual context for learning from the previous iteration to help generate a more specific and richer explanation of the learning video data.

[0052] Furthermore, similar to the first iteration, the learning device 100 can ensure that the k-2nd training response generated through the (k-1)th video large-scale multimodal model 300_k is different from the k-1st training response.

[0053] Specifically, the learning device 100 sets the temperature hyperparameter of the (k-1)th trained video large-scale multimodal model to a specific temperature hyperparameter value, inputs training video data and training query data into the (k-1)th trained video large-scale multimodal model, and uses the (k-1)th trained video large-scale multimodal model to generate the k-1st and k-2nd training responses, respectively, using the specific temperature hyperparameter value.

[0054] At this time, the learning device 100 may (i) input the training video data and training query data into the (k-1)th trained large-scale video multimodal model only once, and use the (k-1)th trained large-scale video multimodal model to generate two different training k_1 and training k_2 responses, respectively; or (ii) input the training video data and training query data into the (k-1)th trained large-scale video multimodal model, and use the (k-1)th trained large-scale video multimodal model to generate the training k_1 response, and then input the training video data and training query data into the (k-1)th trained large-scale video multimodal model again, and use the (k-1)th trained large-scale video multimodal model to generate a training k_2 response that is different from the training k_1 response, but is not limited to these.

[0055] Alternatively, the learning device 100 may use a (k-1)-trained large-scale multimodal video model 300_k to embed training video data and training query data through an embedding layer to generate the kth embedding vector, and then use the kth embedding vector through a large-scale language model to generate the kth-th visual context for training, the kth-1st training response, and the kth-2nd training response.

[0056] In this way, the learning device 100 generates a learning-grade k-th visual context, a learning-grade k-1 response, and a learning-grade k-2 response using the (k-1)-trained large-scale multimodal video model 300_k. The learning device 100 can then generate a learning-grade k-th preference dataset 310_k, which includes the learning-grade k-th visual context, the learning-grade k-1 response, the learning-grade k-2 response, and the learning-grade video data and learning-grade query data used as input to the (k-1)-trained large-scale multimodal video model 300_k. At this point, the learning-grade k-th preference dataset 310_k can be used as input data for the (k-1)-trained large-scale multimodal video model 300_k in a subsequent process described later. An example of the learning-grade k-th preference dataset 310_k will be explained with reference to Figure 5.

[0057] Figure 5 schematically shows an example of a preference dataset for training a large-scale video multimodal model according to one embodiment of the present invention.

[0058] First, it can be confirmed that Figure 5(a) shows examples of training image data and training query data, (b) shows an example of a training k-th visual context, and (c) shows examples of training k-1 and training k-2 responses.

[0059] In this case, the training video data and training query data are the same data that is used repeatedly from the first iteration to the nth iteration, the training k-th visual context is generated when the (k-1)-th trained large-scale multimodal video model references the training (k-1)-th visual context and training video data, and the training k_1 response and training k_2 response, respectively, may be generated when they reference the training video data and training query data.

[0060] Furthermore, when examining the k-1st training response (e.g., y1), the underlined content may indicate that some of the content is incorrect information unrelated to the training video data, and when examining the k-2nd training response (e.g., y2), the bolded content may indicate that some of the content is correct information supported by the training video data, but this is not limited to these interpretations. For example, if the k-1st and k-2nd training responses are generated in this manner, in a subsequent process described later, the learning device 100 may use the (k-1)th trained large-scale multimodal video model 300_k to determine the k-1st training response as a non-preferred response and the k-2nd training response as a preferred response.

[0061] Next, referring again to Figure 3, the learning device 100 inputs the k-th preference dataset for learning into the (k-1)-th trained large-scale video multimodal model, and uses the (k-1)-th trained large-scale video multimodal model to refer to the k-th preference dataset for learning and determine one of the k-1 and k-2 responses for learning as the k-th preferred response, and the other response as the k-th disliked response for learning, thereby generating k-th preference feedback data for learning (S220_k) which includes the k-th preferred response for learning, the k-th disliked response for learning, the video data for learning, and the query data for learning.

[0062] Referring to (b) in Figure 4b, in the S210_k process, the (k-1)th trained video large-scale multimodal model 300_k was used to generate the kth-th visual context for training and to generate the kth-1st and kth-2nd responses for training. However, in the S220_k process, the (k-1)th trained video large-scale multimodal model 300_k may be used as a model to determine the preference for the kth-1st and kth-2nd responses for training by referring to the kth-th preference dataset 310_k. Furthermore, the accuracy of preference determination can be improved by having the learning device 100 also refer to the kth-th visual context for training when determining the kth-th preferred and kth-th disliked responses for training using the (k-1)th trained video large-scale multimodal model 300_k.

[0063] Therefore, the learning device 100 can generate learning preference feedback data 320_k which includes the kth preferred response and the kth non-preferred response for learning determined by the (k-1)th trained large-scale multimodal video model 300_k, and the learning video data and learning query data included in the learning preference dataset 310_k. The learning preference feedback data 320_k can then be used to train the (k-1)th trained large-scale multimodal video model 300_k.

[0064] Next, referring again to Figure 3, the learning device 100 can generate the kth DPO loss for the kth preference feedback data for learning using the DPO, and then generate the kth-th trained video large-scale multimodal model (S230_k) by updating the parameters of the (k-1)th trained video large-scale multimodal model using the kth DPO loss.

[0065] Referring to Figure 4b(c) for the S230_k process, the learning device 100 can input the learning video data and learning query data contained in the learning k-th preference feedback data 320_k into the reference video large-scale multimodal model 400 corresponding to the (k-1)-th trained video large-scale multimodal model 300_k, so that the reference video large-scale multimodal model 400 can output the learning reference preference response and the learning reference dispatch response, respectively. Furthermore, the learning device 100 can use DPO to generate the k-th DPO loss by referring to the learning k-th preference response and the learning k-th dispatch response, respectively, contained in the learning k-th preference feedback data 320_k, and the learning reference preference response and the learning reference dispatch response, respectively. The reference video large-scale multimodal model 400 is a pre-supervised and trained model using training data for a given task, and can be used as a base model for generating the kth DPO loss. The training reference preference response and training reference non-preference response can each be used as a GT for the training kth preference response and training kth non-preference response, respectively.

[0066] Specifically, the learning device 100 may input the kth preferred response for learning, the kth disliked response for learning, the reference preferred response for learning, and the reference disliked response for learning to the loss layer 330, and through the loss layer 330, replace the reward modeling for the preferred responses by referring to the ratio of the kth preferred response for learning to the reference preferred response, and replace the reward modeling for the disliked responses by referring to the ratio of the kth disliked response for learning to the reference disliked response, thereby generating the kth DPO loss.

[0067] At this time, the kDPO loss can be expressed by the following formula:

number

[0068] In the above formula,

number

number

[0069] Therefore, when the k-th DPO loss is generated through the loss layer 330, the learning device 100 can generate a k-th trained video large-scale multimodal model by updating the parameters of the (k-1)-th trained video large-scale multimodal 300_k using the k-th DPO loss.

[0070] On the other hand, the results of comparing the differences in the responses generated when the same video data and query data were input to a large-scale multimodal video model trained according to the process of the present invention and a conventional large-scale multimodal video model will be explained with reference to Figure 6.

[0071] Figure 6 schematically shows an example of response results generated by inputting the same data into both the pre-trained video large-scale multimodal model and a conventional video large-scale multimodal model, respectively, after a predetermined number of iterative training sessions have been completed for the video large-scale multimodal model according to one embodiment of the present invention.

[0072] Referring to Figure 6(a), the upper section contains video data confirming that an athlete is performing a long jump, and query data asking what the athlete is doing in the video. This video data and query data may be input into the pre-trained large-scale multimodal video model of the present invention and a conventional large-scale multimodal video model, respectively.

[0073] Therefore, referring to Figure 6(b), when the iteration is applied five times, the response results generated by the trained video large-scale multimodal model of the present invention and the conventional video large-scale multimodal model can be seen that the response generated through the conventional video large-scale multimodal model contains some content shown in bold, i.e., content based on video data, but a considerable portion of the content shown in underline, i.e., content unrelated to video data, whereas the response generated through the video large-scale multimodal model of the present invention mainly contains content shown in bold, i.e., content based on video data, and does not contain unnecessary content.

[0074] Furthermore, when nine iterations are applied, the response generated by the conventional large-scale multimodal video model contains a large amount of underlined content, i.e., content unrelated to the video data, whereas the response generated by the large-scale multimodal video model of the present invention contains more specifically described content, i.e., content based on the video data, as shown in bold. In other words, the learning method of the present invention makes it possible for the large-scale multimodal video model to generate a response corresponding to query data by referring to video data, so that the generated response is described more specifically as the iteration increases, while not including content unrelated to the video data.

[0075] On the other hand, the advantages of a method for learning a large-scale video multimodal model through iterative self-retrospective judgment according to one embodiment of the present invention will be described below. [Table 1]

[0076] The experimental results described above compare the performance differences between the present invention (Iterative Self-Retrospective Judgment, i-SRT) and related cutting-edge technologies using an In-domain zero-shot video question answering (VAQ) dataset.

[0077] Specifically, the learning method that showed the best performance in both accuracy and score for each of the Activity Net-QA, VIDAL-QA, and WebVid-QA datasets is shown in bold, confirming that the i-SRT method of the present invention has the best performance. [Table 2]

[0078] The experimental results described above compare the performance difference between the present invention (i-SRT) and related cutting-edge technologies using an out-domain zero-shot video question answering dataset.

[0079] Specifically, the learning method that showed the best performance in both accuracy and score for each of the MSVD-QA, MSRVTT-QA, TGIF-QA, and SSV2-QA datasets is shown in bold, confirming that the i-SRT method of the present invention has the best performance.

[0080] Furthermore, the embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or they may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device can be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0081] Although the present invention has been described above based on specific details such as concrete components, limited embodiments, and drawings, these are provided only to aid in a more overall understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention pertains can make various modifications and variations from this description.

[0082] Therefore, the concept of the present invention shall not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below shall also fall within the scope of the concept of the present invention.

Claims

1. In a method for learning a large-scale multimodal video model through iterative self-retrospective judgment, (a) The learning device inputs learning video data and learning query data in text format into an initial video large multimodal model, and uses the initial video large multimodal model to generate a first learning visual context for the learning video data, and uses the learning video data to generate a first learning response and a first learning response corresponding to the learning query data (the first learning response is a response that is different from the first learning response), thereby generating the first learning visual context, the first learning response, the first learning response, the first learning response, the learning video data and the learning query (ii) Generate a first preference dataset for learning that includes the lead data; (ii) Input the first preference dataset for learning into the initial video large multimodal model, and use the initial video large multimodal model to refer to the first preference dataset for learning and determine one of the first 1_1 response and the first 2 response for learning as the first preference response for learning, and the other response as the first dislike response for learning, thereby generating first preference feedback data for learning that includes the first preference response for learning, the first dislike response for learning, the video data for learning, and the query data for learning; (iii) DPO (Direct The steps include: generating a first DPO loss for the first preference feedback data for training using Preference Optimization, and generating a first trained video large-scale multimodal model by updating the parameters of the initial video large-scale multimodal model using the first DPO loss; and (b) The learning device inputs (i) the (k-1)th trained video large multimodal model to generate the kth trained visual context for the video data by referencing the (k-1)th trained video large multimodal model to generate the kth trained visual context for the video data by referencing the (k-1)th trained visual context for the video data to generate the kth trained visual context for the video data by referencing the video data to generate the kth trained response and the kth trained response corresponding to the video data to generate the kth trained response and the kth trained response (the kth trained response is a response that is different from the kth trained response) to generate the kth trained response and the kth trained response corresponding to the video data to generate the kth trained response and the kth trained response, the kth trained response, the video data and the video data to generate the kth trained preference data (ii) generate a dataset; (ii) input the k-th preference dataset for training into the (k-1) trained video large multimodal model; and use the (k-1) trained video large multimodal model to determine which of the k-1 and k-2 training responses is the k-th preferred response, and which response is the k-th disliked response, by referring to the k-th preference dataset for training, thereby generating k-th preference feedback data for training, which includes the k-th preferred response, the k-th disliked response, the training video data, and the training query data; (iii) generate a k-th DPO loss for the k-th preference feedback data using the DPO; and use the k-th DPO loss to update the parameters of the (k-1) trained video large multimodal model, thereby generating a k-th trained video large multimodal model; A method that includes this.

2. In step (a) above, The learning device, in (iii) above, uses the DPO to create a first DPO loss by referring to each of the first learning preference response and the first learning dislike response in the first learning preference feedback data and each of the learning reference preference response and learning reference dislike response output from the reference video large multimodal model by inputting the learning video data and the learning query data into the reference video large multimodal model corresponding to the initial video large multimodal model. In step (b) above, The method according to claim 1, wherein the learning device generates the kDPO loss in (iii) by using the DPO to refer to each of the learning k-th preference response and the learning k-th dislike response in the learning k-th preference feedback data and each of the learning reference preference response and the learning reference dislike response output from the reference video large multimodal model by inputting the learning video data and the learning query data into the reference video large multimodal model.

3. The method according to claim 2, wherein the reference video large-scale multimodal model is a supervised-trained model and is a base model for generating the first DPO loss to the kDPO loss.

4. In step (a) above, The learning device sets the temperature hyperparameter of the initial video large-scale multimodal model to a specific temperature hyperparameter value that is greater than or equal to a preset threshold, inputs the training video data and the training query data into the initial video large-scale multimodal model, and uses the initial video large-scale multimodal model to generate the training first-first response and the training first-second response, respectively, using the specific temperature hyperparameter value. In step (b) above, The method according to claim 1, wherein the learning device is configured such that the temperature hyperparameter of the (k-1) trained video large multimodal model becomes the specific temperature hyperparameter value, and the learning video data and the learning query data are input to the (k-1) trained video large multimodal model, and the (k-1) trained video large multimodal model is used to generate the learning k-1 response and the learning k-2 response, respectively, using the specific temperature hyperparameter value.

5. In step (a) above, The learning device uses the initial video large-scale multimodal model to embed training video data and training query data in text format through an embedding layer to generate a first embedding vector, and uses the first embedding vector through a large-scale language model to generate the training first visual context, the training first_1 response, and the training first_2 response. In step (b) above, The method according to claim 1, wherein the learning device generates a k-th embedding vector by embedding the training video data and the training query data through the embedding layer using the (k-1) trained video large multimodal model, and generates the training k-th visual context, the training k-1 response and the training k-2 response through the large language model using the k-th embedding vector.

6. In a learning device that learns a large-scale multimodal video model through iterative self-retrospective judgment, At least one memory to store instructions; and Includes at least one processor configured to perform the aforementioned instructions; The processor (i) inputs training video data and training query data in text format into an initial video large multimodal model to generate a first training visual context for the training video data using the initial video large multimodal model, and generates a first training response and a first training response (the first training response is different from the first training response) corresponding to the training query data by referring to the training video data, thereby generating the first training visual context, the first training response, the first training response, the first training response, the training video data and the training (ii) Generate a first preference dataset for learning that includes query data; (ii) Input the first preference dataset for learning into the initial video large multimodal model, and use the initial video large multimodal model to refer to the first preference dataset for learning and determine one of the first 1_1 response and the first 2 response for learning as the first preference response for learning, and the other response as the first dislike response for learning, thereby generating first preference feedback data for learning that includes the first preference response for learning, the first dislike response for learning, the video data for learning, and the query data for learning; (iii) DPO (Direct A process to generate a first trained video large-scale multimodal model by generating a first DPO loss for the first preference feedback data for training using Preference Optimization, and updating the parameters of the initial video large-scale multimodal model using the first DPO loss;(II) (i) Input the (k-1)th trained video large multimodal model of the (k-1)th trained video large multimodal model to generate the kth trained visual context for the video data by referencing the (k-1)th trained visual context and the video data, and generate the k-1 and k-2 trained responses corresponding to the video data by referencing the video data (the k-2 trained response is different from the k-1 trained response) to generate the kth trained preference dataset of the (k)th visual context, the k-1 trained response, the k-2 trained response, the video data, and the video data. A learning device that generates (ii) the k-th preference dataset for learning input to the (k-1) trained video large multimodal model, and uses the (k-1) trained video large multimodal model to determine one of the k-1 and k-2 responses for learning as the k-th preference response, and the other response as the k-th dislike response for learning, thereby generating k-th preference feedback data for learning, which includes the k-th preference response, the k-th dislike response, the video data for learning, and the query data for learning; and (iii) generates the k-th DPO loss for the k-th preference feedback data using the DPO, and uses the k-th DPO loss to update the parameters of the (k-1) trained video large multimodal model for learning, thereby generating the k-th trained video large multimodal model.

7. The aforementioned processor, In step (iii) of the process described in (I), the DPO is used to generate the first DPO loss by referring to the first learning preference response and the first learning dislike response in the first learning preference feedback data, respectively, and to the first learning reference preference response and the first learning reference dislike response output from the reference video large multimodal model, which is output by inputting the learning video data and the learning query data into the reference video large multimodal model corresponding to the initial video large multimodal model. The learning apparatus according to claim 6, wherein in step (iii) of the process (II) described above, the DPO is used to generate the kDPO loss by referring to each of the k-th preferred response and the k-th disliked response in the k-th preferred feedback data for learning, and each of the reference preferred response and the k-th disliked response for learning output from the large-scale multimodal reference video model by inputting the learning video data and the learning query data into the large-scale multimodal reference video model.

8. The learning apparatus according to claim 7, wherein the reference video large-scale multimodal model is a supervised-trained model and is a base model for generating the first DPO loss to the kDPO loss.

9. The aforementioned processor, In the above process (I), The temperature hyperparameter of the initial video large-scale multimodal model is set to a specific temperature hyperparameter value that is greater than or equal to a predetermined threshold, the training video data and the training query data are input to the initial video large-scale multimodal model, and the initial video large-scale multimodal model is used to generate the training 1_1 response and the training 1_2 response, respectively, using the specific temperature hyperparameter value. In the above process (II), The learning device according to claim 6, wherein the temperature hyperparameter of the (k-1) trained video large multimodal model is set to the specific temperature hyperparameter value, the training video data and the training query data are input to the (k-1) trained video large multimodal model, and the (k-1) trained video large multimodal model is used to generate the training k-1 response and the training k-2 response, respectively, using the specific temperature hyperparameter value.

10. The aforementioned processor, In the above process (I), The initial video large-scale multimodal model is used to embed training video data and training query data in text format through an embedding layer to generate a first embedding vector, and the large-scale language model is used to generate the first training visual context, the first training response, and the first training response through the first embedding vector. In the above process (II), The learning device according to claim 6, wherein the learning device generates a k-th embedding vector by embedding the training video data and the training query data through the embedding layer using the (k-1) trained large-scale multimodal video model, and generates the training k-th visual context, the training k-1 response and the training k-2 response through the large-scale language model using the k-th embedding vector.

Images