Method for learning a large-scale multimodal model of video through iterative self-retrospective judgment and learning device using the same

Abstract

Learning video large-scale multimodal models through iterative self-retrospective judgment. [Solution] The method inputs a training k-th preference dataset including a k-th visual context, a k_1st response, a k_2nd response, video data, and query data into a (k-1)th trained video large-scale multimodal model, references the training k-th preference dataset in the model, determines one of the training k-th response and the training k_2nd response as the training k-th preferred response, and determines the other response as the training k-th non-preferred response, generates training k-th preference feedback data including the k-th preferred response, the k-th non-preference response, video data, and query data, generates a k-th DPO loss using DPO, and generates a k-th trained video large-scale multimodal model by updating the parameters of the (k-1)th trained video large-scale multimodal model using the k-th DPO loss.

Description

[Technical Field]

[0001] The present invention relates to a method for training a video large multimodal model through iterative self-retrospective judgment and a learning device using the same. [Background technology]

[0002] Large language models (LLMs) are used in solutions such as conversational chatbots like ChatGPT. A pre-trained large language model is fine-tuned through supervised learning using a high-quality dataset suited to a specific task, and a reward model is trained using training data labeled with human preferences to learn how to mimic human preferences and provide appropriate responses. The fine-tuned large language model is then reinforced using the labeled rewards output from the trained reward model, using the Reinforcement Learning from Human Feedback (RLHF) method.

[0003] While this RLHF method can maximize language generation and dialogue capabilities, it requires the preparation of a separate reward model in addition to the large-scale language model and training on the reward model, as well as the need to sample the output of the large-scale language model, which increases computational costs and complicates the computational process, resulting in an unstable training process.To address these issues, we used the Direct Preference Optimization (DPO) method, which uses the large-scale language model itself in the same role as the reward model without using a reward model, thereby omitting the modeling process of the reward model and optimizing the large-scale language model using preference data.

[0004] Recently, attempts have been made to apply this DPO method not only to large-scale language models but also to video large multimodal models (VLMMs). Referring to Figure 1, (a) of Figure 1 shows that the length of the answer output by a video large multimodal model tends to gradually increase as the DPO method is repeatedly applied to the video large multimodal model. As shown in (b) of Figure 1, when video data consisting of multiple frames and text query data are received by the video large multimodal model, the length of the answer output by the video large multimodal model varies depending on the number of iterations, such as when the DPO method is applied once (i.e., one learning iteration), five times (i.e., five learning iterations), and nine times (i.e., nine learning iterations). However, when the DPO method is repeated a small number of times (e.g., 1 and 5 times), the answers output by the video large-scale multimodal model contain content displayed in bold, i.e., content that is based on the video data. However, when the DPO method is repeated a large number of times (e.g., 9 times), the answers output by the video large-scale multimodal model contain content displayed in underline, i.e., inappropriate content that is unrelated to the video data. This is a hallucination phenomenon.

[0005] Therefore, there is a need for improvements to solve the above problems. Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to solve all of the above-mentioned problems.

[0007] The present invention also provides a method for (i) inputting training video data and training query data into an initial video large-scale multimodal model, so that the initial video large-scale multimodal model outputs a first training visual context corresponding to the training video data, and two different responses corresponding to the training query data, a training response 1_1 and a training response 1_2; and (ii) inputting the training video data, the training query data, the first training visual context, the training response 1_1 and the training response 1_2 as a first training preference dataset into the initial video large-scale multimodal model, thereby outputting the first training preference dataset. Another object is to use a large-scale multimodal model to determine one of the training first_1 response and the training first_2 response as the training first preference response by referring to the training first preference dataset, and determine the other response as the training first non-preference response; (iii) use the training video data, training query data, training first preference response, and training first non-preference response as training first preference feedback data, generate a first DPO loss for the training first preference feedback data, and use the first DPO loss to update the parameters of the initial video large-scale multimodal model, thereby generating a first trained video large-scale multimodal model.

[0008] The present invention also provides a method for (i) inputting training video data, training query data, and a (k-1)th training visual context into a (k-1)th trained video large-scale multimodal model, so that the (k-1)th trained video large-scale multimodal model outputs a k-th training visual context corresponding to the training video data, and two different responses corresponding to the training query data, the k_1st training response and the k_2nd training response; and (ii) inputting the training video data, training query data, the k-th training visual context, the k_1st training response, and the k_2nd training response as a k-th training preference dataset into the (k-1)th trained video large-scale multimodal model. (iii) using the training video data, training query data, training k-th preference response and training k-th non-preference response as training k-th preference feedback data, generate a k-th DPO loss for the training k-th preference feedback data, and update the parameters of the (k-1)-th trained video large-scale multimodal model by using the k-th DPO loss to generate a k-th trained video large-scale multimodal model. [Means for solving the problem]

[0009] According to one embodiment of the present invention, in a method for training a video massive multimodal model through iterative self-retrospective judgment, (a) a training device (i) inputs training video data and training query data in text format into an initial video massive multimodal model, and generates a training first visual context for the training video data using the initial video massive multimodal model, and generates a training first_1 response and a training first_2 response (the training first_2 response is a response different from the training first_1 response) corresponding to the training query data by referring to the training video data, thereby generating the training first visual context, the training first_1 response, the training first_2 response, (ii) generating a first training preference dataset including the training video data and the training query data; (ii) inputting the first training preference dataset into the initial video massive multimodal model, and using the initial video massive multimodal model to determine, with reference to the first training preference dataset, one of the training 1_1 response and the training 1_2 response as a first training preference response and determine the other response as a first training non-preference response, thereby generating first training preference feedback data including the first training preference response, the first training non-preference response, the training video data, and the training query data; (iii) generating a first DPO loss for the first training preference feedback data using DPO (Direct Preference Optimization), and updating parameters of the initial video massive multimodal model using the first DPO loss to generate a first trained video massive multimodal model;and (b) a learning device (i) inputs a (k-1)th training visual context (where k is an integer increasing from 2 to n, and n is an integer equal to or greater than 2), the training video data, and the training query data into a (k-1)th trained video large-scale multimodal model, and generates a k-th training visual context for the training video data by referring to the (k-1)th training visual context and the training video data using the (k-1)th trained video large-scale multimodal model, and generates a k_1st training response and a k_2nd training response corresponding to the training query data by referring to the training video data (the training k_2nd response is a response different from the k_1st training response), thereby generating a k-th training preference dataset including the k-th training visual context, the k_1st training response, the k_2nd training response, the training video data, and the training query data. (ii) inputting the k-th training preference data set into the (k-1) trained video massive multimodal model, and using the (k-1) trained video massive multimodal model, determining one of the k_1st training response and the k_2nd training response as the k-th training preference response by referring to the k-th training preference data set and determining the other response as the k-th training non-preference response, thereby generating k-th training preference feedback data including the k-th training preference response, the k-th training non-preference response, the training video data, and the training query data; and (iii) generating a k-th DPO loss for the k-th training preference feedback data using the DPO, and generating a k-th trained video massive multimodal model by updating parameters of the (k-1) trained video massive multimodal model using the k-th DPO loss.

[0010] In one example, in step (a), the learning device uses the DPO to create a first DPO loss by referring to each of the first learning preference response and first learning non-preference response in the first learning preference feedback data, and each of the learning reference preference response and learning reference non-preference response output from the reference video large-scale multimodal model corresponding to the initial video large-scale multimodal model by inputting the learning video data and the learning query data into the reference video large-scale multimodal model; and in step (b), the learning device uses the DPO to create the kDPO loss by referring to each of the learning kth preference response and the learning kth non-preference response in the kth learning preference feedback data, and each of the learning reference preference response and the learning reference non-preference response output from the reference video large-scale multimodal model by inputting the learning video data and the learning query data into the reference video large-scale multimodal model.

[0011] In one example, the reference video large-scale multimodal model is a supervised trained model and is a base model for generating the first DPO loss through the kth DPO loss.

[0012] In one example, in step (a), the learning device sets a temperature hyperparameter of the initial video massively multimodal model to a specific temperature hyperparameter value that is equal to or greater than a predetermined threshold, inputs the training video data and the training query data into the initial video massively multimodal model, and uses the initial video massively multimodal model to generate the training 1_1 response and the training 1_2 response, respectively, using the specific temperature hyperparameter value; and in step (b), the learning device sets the temperature hyperparameter of the (k-1) trained video massively multimodal model to the specific temperature hyperparameter value, inputs the training video data and the training query data into the (k-1) trained video massively multimodal model, and uses the (k-1) trained video massively multimodal model to generate the training k_1 response and the training k_2 response, respectively, using the specific temperature hyperparameter value.

[0013] In one example, in step (a), the learning device uses the initial video large scale multimodal model to embed the training video data and the training query data in text format through an embedding layer to generate a first embedding vector, and uses the first embedding vector to generate the first training visual context, the training response 1_1, and the training response 1_2 through a large scale language model; and in step (b), the learning device uses the (k-1) trained video large scale multimodal model to embed the training video data and the training query data through the embedding layer to generate a kth embedding vector, and uses the kth embedding vector through the large scale language model to generate the kth training visual context, the training response k_1, and the training response k_2.

[0014] According to another embodiment of the present invention, a learning device for learning a video massively multimodal model through iterative self-retrospective judgment includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor (I) (i) inputs training video data and text-based training query data into an initial video massively multimodal model, and generates a training first visual context for the training video data using the initial video massively multimodal model, and generates a training first_1 response and a training first_2 response corresponding to the training query data by referring to the training video data (the training first_2 response is a response different from the training first_1 response), and generates the training first_2 response corresponding to the training query data by referring to the training video data. (ii) inputting the first training preference dataset into the initial video large-scale multimodal model, and determining, with reference to the first training preference dataset, one of the training response 1_1 and the training response 1_2 as a first training preference response and determining the other response as a first training non-preference response, thereby generating first training preference feedback data including the first training preference response, the first training non-preference response, the training video data, and the training query data; and (iii) generating a Direct Preference Output (DPO). generating a first DPO loss for the first training preference feedback data using Preference Optimization (DPO), and updating parameters of the initial video massive multimodal model using the first DPO loss to generate a first trained video massive multimodal model;and (II) (i) inputting the (k-1)th training visual context (where k is an integer increasing from 2 to n, and n is an integer equal to or greater than 2), the training video data, and the training query data into a (k-1)th trained video large-scale multimodal model, and using the (k-1)th trained video large-scale multimodal model to generate a k-th training visual context for the training video data by referring to the (k-1)th training visual context and the training video data, and to generate a k_1st training response and a k_2nd training response corresponding to the training query data by referring to the training video data (the training k_2nd training response is a response different from the k_1st training response), thereby generating a k-th training preference dataset including the k-th training visual context, the k_1st training response, the k_2nd training response, the training video data, and the training query data. (ii) inputting the k-th training preference data set into the (k-1) trained video large-scale multimodal model, and using the (k-1) trained video large-scale multimodal model, determining one of the k_1st training response and the k_2nd training response as the k-th training preference response by referring to the k-th training preference data set and determining the other response as the k-th training non-preference response, thereby generating k-th training preference feedback data including the k-th training preference response, the k-th training non-preference response, the training video data, and the training query data; and (iii) generating a k-th DPO loss for the k-th training preference feedback data using the DPO, and updating parameters of the (k-1) trained video large-scale multimodal model using the k-th DPO loss to generate the k-th trained video large-scale multimodal model.

[0015] In one example, in (iii) of the (I) process, the processor uses the DPO to generate the first DPO loss by referring to each of the first training preference response and first training non-preference response in the first training preference feedback data and each of the training reference preference response and training reference non-preference response output from the reference video massive multimodal model corresponding to the initial video massive multimodal model by inputting the training video data and the training query data into the reference video massive multimodal model, and in (iii) of the (II) process, uses the DPO to generate the kth DPO loss by referring to each of the training kth preference response and the training kth non-preference response in the kth training preference feedback data and each of the training reference preference response and training reference non-preference response output from the reference video massive multimodal model by inputting the training video data and the training query data into the reference video massive multimodal model.

[0016] In one example, the reference video large-scale multimodal model is a supervised trained model and is a base model for generating the first DPO loss through the kth DPO loss.

[0017] In one example, the processor, in process (I), sets a temperature hyperparameter of the initial video massively multimodal model to a specific temperature hyperparameter value that is equal to or greater than a preset threshold, inputs the training video data and the training query data into the initial video massively multimodal model, and uses the initial video massively multimodal model to generate the training 1_1 response and the training 1_2 response, respectively, using the specific temperature hyperparameter value; and in process (II), sets the temperature hyperparameter of the (k-1) trained video massively multimodal model to the specific temperature hyperparameter value, inputs the training video data and the training query data into the (k-1) trained video massively multimodal model, and uses the (k-1) trained video massively multimodal model to generate the training k_1 response and the training k_2 response, respectively, using the specific temperature hyperparameter value.

[0018] In one example, in the process (I), the processor uses the initial video massive multimodal model to embed the training video data and the training query data in text format through an embedding layer to generate a first embedding vector, and uses the first embedding vector to generate the first training visual context, the training response 1_1, and the training response 1_2 through a large-scale language model; and in the process (II), the learning device uses the (k-1) trained video massive multimodal model to embed the training video data and the training query data through the embedding layer to generate a k-th embedding vector, and uses the k-th embedding vector through the large-scale language model to generate the k-th training visual context, the training response k_1, and the training response k_2. [Effects of the Invention]

[0019] The present invention provides a method for (i) inputting training video data and training query data into an initial video large-scale multimodal model, so that the initial video large-scale multimodal model outputs a training first visual context corresponding to the training video data and two different responses corresponding to the training query data, a training first_1 response and a training first_2 response; and (ii) inputting the training video data, the training query data, the training first visual context, the training first_1 response and the training first_2 response as a training first preference dataset into the initial video large-scale multimodal model, thereby obtaining the initial video preference dataset. (o) Using a large-scale multimodal model, one of the training first-1_1 response and the training first-2 response is determined as the training first preference response by referring to the training first preference dataset, and the other response is determined as the training first non-preference response; (iii) using the training video data, training query data, training first preference response, and training first non-preference response as training first preference feedback data, a first DPO loss is generated for the training first preference feedback data, and the parameters of the initial video large-scale multimodal model are updated using the first DPO loss, thereby generating a first trained video large-scale multimodal model.

[0020] The present invention also provides a method for (i) inputting training video data, training query data, and a (k-1)th training visual context into a (k-1)th trained video large-scale multimodal model, so that the (k-1)th trained video large-scale multimodal model outputs a k-th training visual context corresponding to the training video data, and two different responses corresponding to the training query data, a k_1st training response and a k_2nd training response; and (ii) using the training video data, training query data, the k-th training visual context, the k_1st training response, and the k_2nd training response as a k-th training preference dataset, and outputting the k-th training preference dataset from the (k-1)th trained video large-scale multimodal model. By inputting the training data into the database, the (k-1)th trained video large-scale multimodal model is used to determine one of the training k_1st response and the training k_2nd response as the training k-th preferred response by referring to the training k-th preference dataset, and the other response is determined as the training k-th non-preference response; (iii) the training video data, training query data, training k-th preferred response, and training k-th non-preference response are used as training k-th preference feedback data to generate a k-th DPO loss for the training k-th preference feedback data, and the parameters of the (k-1)th trained video large-scale multimodal model are updated using the k-th DPO loss, thereby generating a k-th trained video large-scale multimodal model. [Brief explanation of the drawings]

[0021] The following drawings attached for use in explaining embodiments of the present invention are only a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter referred to as "ordinary engineer") can derive other drawings based on these drawings without performing any inventive work.

[0022] [Figure 1] FIG. 1 is a schematic diagram illustrating a prior art example of applying the DPO method to a video large-scale multimodal model. [Figure 2] FIG. 2 is a schematic diagram of a learning apparatus for learning a large-scale multi-modal model of video through iterative self-reflection, according to one embodiment of the present invention. [Figure 3] FIG. 3 is a schematic diagram illustrating a flow diagram for learning a video large-scale multi-modal model through iterative self-retrospective judgment, according to one embodiment of the present invention. [Figure 4a] FIG. 4a is a detailed diagram illustrating the process of learning a large-scale multi-modal model for video through iterative self-reflection, according to one embodiment of the present invention. [Figure 4b] FIG. 4b is a detailed diagram illustrating the process of learning a video large-scale multi-modal model through iterative self-retrospective judgment, according to one embodiment of the present invention. [Figure 5] FIG. 5 is a diagram that schematically illustrates an example of a training preference dataset for training a video large-scale multi-modal model according to one embodiment of the present invention. [Figure 6] Figure 6 is a diagram showing an example of response results generated by inputting the same data into a trained video massive multimodal model of the present invention and a conventional video massive multimodal model, after a predetermined number of iterative training steps have been completed for the video massive multimodal model, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] In order to clarify the objectives, technical solutions and advantages of the present invention, the following detailed description of the present invention refers to the accompanying drawings which show, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those of ordinary skill in the art to practice the invention.

[0024] Moreover, in the detailed description of the present invention and the claims, the word "comprises" and variations thereof are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and characteristics of the present invention will become apparent in part from the specification and in part from the practice of the invention to those of ordinary skill in the art. The following examples and drawings are provided as illustrations and are not intended to limit the present invention.

[0025] Furthermore, the present invention encompasses all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention, while different from one another, are not necessarily mutually exclusive. For example, a particular shape, structure, and characteristic described herein, in connection with one embodiment, may be implemented in other embodiments without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the present invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly recited. Like reference numerals in the drawings refer to the same or similar functionality throughout the various aspects.

[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention.

[0027] FIG. 2 is a schematic diagram of a learning apparatus for learning a large-scale multi-modal model of video through iterative self-reflection, according to one embodiment of the present invention.

[0028] 2, the learning device 100 may include a memory 110 that stores instructions for learning a video large scale multimodal model through iterative self-retrospective judgment, and a processor 120 that learns the video large scale multimodal model through iterative self-retrospective judgment in accordance with the instructions stored in the memory 110. In this case, the learning device 100 may include a personal computer (PC), a mobile computer, etc.

[0029] Specifically, learning device 100 may typically use a combination of computing devices (e.g., devices that may include a computer processor, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular manner) to achieve desired system performance.

[0030] The processor of a computing device may include hardware components such as a micro processing unit (MPU) or central processing unit (CPU), cache memory, data bus, etc. The computing device may also include software components such as an operating system and applications that perform specific purposes.

[0031] However, this does not exclude the case where the computing device includes an integrated processor in which the medium, processor, and memory for implementing the present invention are integrated.

[0032] FIG. 3 is a schematic diagram illustrating a flow diagram for learning a video large-scale multi-modal model through iterative self-retrospective judgment, according to one embodiment of the present invention.

[0033] First, the learning device 100 inputs training video data and training query data in text format into an initial video large-scale multimodal model, and generates a training first visual context for the training video data using the initial video large-scale multimodal model, and generates a training first_1 response and a training first_2 response corresponding to the training query data by referring to the training video data, thereby generating a training first preference dataset including the training first visual context, the training first_1 response, the training first_2 response, the training video data, and the training query data (S210_1).

[0034] Regarding the S210_1 process, referring to (a) of Figure 4a, the learning device 100 may be capable of generating two different responses corresponding to the learning query, namely, learning response 1_1 and learning response 1_2, by setting a temperature hyperparameter related to the output sensitivity of the initial video large-scale multimodal model 300_1 to a specific temperature hyperparameter value that is equal to or greater than a predetermined threshold.

[0035] For example, if the value of a particular temperature hyperparameter is close to 0 (e.g., 0.1), the output sensitivity to the initial video large-scale multimodal model 300_1 will be low, and conversely, if the value of a particular temperature hyperparameter is close to 1 (e.g., 0.7 or 0.8), the output sensitivity to the initial video large-scale multimodal model 300_1 will be high.

[0036] Therefore, the learning device 100 may input the training video data and the training query data into the initial video massive multimodal model 300_1 and generate the training 1_1 response and the training 1_2 response using a specific temperature hyperparameter value using the initial video massive multimodal model 300_1. That is, when the value of the specific temperature hyperparameter is low, close to 0, the training 1_1 response and the training 1_2 response may be the same or similar responses, whereas when the value of the specific temperature hyperparameter is high, close to 1, the training 1_1 response and the training 1_2 response may be different from each other. For example, in the present invention, the learning device 100 may set the specific temperature hyperparameter value to 0.7 so that the initial video massive multimodal model 300_1 generates different training 1_1 responses and training 1_2 responses, but this is not limited to this.

[0037] In this case, the learning device 100 may be configured, but is not limited to, (i) to input the training video data and training query data into the initial video large scale multimodal model 300_1 only once, and use the initial video large scale multimodal model 300_1 to generate a training 1_1 response and a training 1_2 response that are different from each other, or (ii) to input the training video data and training query data into the initial video large scale multimodal model 300_1, and use the initial video large scale multimodal model 300_1 to generate a training 1_1 response, and then input the training video data and training query data into the initial video large scale multimodal model 300_1 again, and use the initial video large scale multimodal model 300_1 to generate a training 1_2 response that is different from the training 1_1 response.

[0038] In addition, the learning device 100 may be configured to use the initial video large scale multimodal model 300_1 to embed training video data and training query data in text format through an embedding layer included in the initial video large scale multimodal model 300_1 to generate a first embedding vector, and to generate a training first visual context, a training first_1 response, and a training first_2 response using the first embedding vector through a large scale language model included in the initial video large scale multimodal model 300_1, but is not limited to this.

[0039] Meanwhile, the learning device 100 can use the initial video large-scale multimodal model 300_1 to generate a first training visual context corresponding to the training video data. The first training visual context is data that describes what the training video data is or what state it is in. It can be used to determine preferences for the first training response and the second training response output from the initial video large-scale multimodal model 300_1 in the first iteration (i.e., processes S210_1 to S230_1 in FIG. 3), and can also be used to generate a second training visual context that is more specific than the first training visual context in the second iteration. Examples of such visual contexts will be described later with reference to FIG. 5.

[0040] In this manner, the learning device 100 generates the first training visual context, the first training response, and the first training response using the initial video large-scale multimodal model 300_1, and can generate a first training preference dataset 310_1 including the first training visual context, the first training response, the first training response, and the first training response, as well as the training video data and training query data used as input to the initial video large-scale multimodal model 300_1. In this case, the first training preference dataset 310_1 can be used as input data to be input to the initial video large-scale multimodal model 300_1 in a subsequent process described below.

[0041] Next, referring to Figure 3, the learning device 100 inputs the first training preference data set into an initial video large-scale multimodal model, and uses the initial video large-scale multimodal model to refer to the first training preference data set to determine one of the training 1_1 response and the training 1_2 response as the first training preference response, and determine the other response as the first training non-preference response, thereby generating first training preference feedback data (S220_1) including the first training preference response, the first training non-preference response, training video data, and training query data.

[0042] Referring to (b) of FIG. 4A regarding process S220_1, in process S210_1, initial video massive multimodal model 300_1 is used to generate a training first visual context and generate training response #1_1 and training response #1_2. However, in process S220_1, initial video massive multimodal model 300_1 may be used as a model to judge preferences for each of training response #1_1 and training response #1_2 by referring to training first preference dataset 310_1. In other words, unlike the conventional RLHF method, which uses a reward model to determine human preferences, in the present invention, initial video massive multimodal model 300_1 not only generates a visual context corresponding to video data and two different responses corresponding to query data, but also directly judges preferences for the two different responses. This reduces the amount of computation required for training compared to the conventional RLHF method using a reward model. This can also be applied to the subsequent process of FIG. 4B described below. In addition, the learning device 100 can further improve the accuracy of preference determination by referring to the first learning visual context when determining each of the first learning preference response and the first learning non-preference response using the initial video large-scale multimodal model 300_1.

[0043] Therefore, the learning device 100 can generate training first preference feedback data 320_1 including the training first preference response and the training first non-preference response determined by the initial video large-scale multimodal model 300_1 and the training video data and training query data included in the training first preference dataset 310_1, and the training first preference feedback data 320_1 can be used to train the initial video large-scale multimodal model 300_1.

[0044] Next, referring again to Figure 3, the learning device 100 can generate a first DPO loss for the first preference feedback data for learning using DPO (Direct Preference Optimization), and generate a first learned video large-scale multimodal model (S230_1) by updating the parameters of the initial video large-scale multimodal model using the first DPO loss.

[0045] 4a(c), regarding the S230_1 process, the learning device 100 can input the training video data and training query data included in the training first preference feedback data 320_1 to a reference video massive multimodal model 400 corresponding to the initial video massive multimodal model 300_1, and output a training reference preference response and a training reference non-preference response, respectively, using the reference video massive multimodal model 400. Furthermore, the learning device 100 can use DPO to generate a first DPO loss by referring to each of the training first preference response and first non-preference response, and each of the training reference preference response and training reference non-preference response, included in the training first preference feedback data 320_1. The reference video large-scale multimodal model 400 is a model that has been pre-trained in a supervised manner using training data for a specified task, and can be used as a base model for generating the first DPO loss, and the training reference preference response and the training reference non-preference response can each be used as GT (Ground Truth) for the training first preference response and the training first non-preference response, respectively.

[0046] Specifically, the learning device 100 may input the first learning preference response, the first learning non-preference response, the learning reference preference response, and the learning reference non-preference response to the loss layer 330, and through the loss layer 310, substitute reward modeling for the preference response by referring to the ratio of the first learning preference response to the learning reference preference response, and substitute reward modeling for the non-preference response by referring to the ratio of the first learning non-preference response to the learning reference non-preference response, thereby generating the first DPO loss.

[0047] Therefore, when the first DPO loss is generated through the loss layer 330, the learning device 100 can generate a first trained video massive multimodal model by updating the parameters of the initial video massive multimodal model 300_1 using the first DPO loss.

[0048] As described above, once the first iteration is completed through the above process and the first trained video large-scale multimodal model is generated, the learning device 100 can perform the second to n-th iterations by repeating the same / similar process as the first iteration. To simply describe the execution of the second to n-th iterations, a variable k can be introduced, where k may be an integer that increases from 2 to n. Accordingly, the process in the k-th iteration can be described as follows.

[0049] Referring to FIG. 3, the learning device 100 inputs the (k-1)th training visual context, training video data, and training query data into the (k-1)th trained video large-scale multimodal model, and uses the (k-1)th trained video large-scale multimodal model to generate a k-th training visual context for the training video data by referring to the (k-1)th training visual context and the training video data, and to generate a k_1st training response and a k_2nd training response corresponding to the training query data by referring to the training video data, thereby generating a k-th training preference dataset (S210_k) including the k-th training visual context, the k_1st training response, the k_2nd training response, the training video data, and the training query data.

[0050] Regarding the S210_k process, referring to (a) of Figure 4b, in the first iteration (specifically, the (a) process of Figure 4a), only the training video data and training query data were input into the initial video massive multimodal model 300_1, but in the kth iteration, the learning device 100 not only inputs the same training video data and training query data as in the first iteration, but also inputs the (k-1)th training visual context obtained in the previous iteration into the (k-1)th trained video massive multimodal model 300_k, which is different from the process in the first iteration.

[0051] In other words, when the learning device 100 generates the kth training visual context through the (k-1)th training video large-scale multimodal model 300_k in the kth iteration, it may refer to the (k-1)th training visual context in the previous iteration to assist in generating more specific and rich explanations for the training video data.

[0052] Also, similar to the first iteration, the learning device 100 can make the training k_2 response generated through the (k-1)th video large-scale multimodal model 300_k a response that is different from the training k_1 response.

[0053] Specifically, the learning device 100 sets the temperature hyperparameter of the (k-1)th trained video large-scale multimodal model to a specific temperature hyperparameter value, inputs training video data and training query data into the (k-1)th trained video large-scale multimodal model, and uses the (k-1)th trained video large-scale multimodal model to generate the training k_1th response and the training k_2th response using the specific temperature hyperparameter value.

[0054] In this case, the learning device 100 can (i) input the training video data and training query data into the (k-1)th trained video large scale multimodal model only once, and use the (k-1)th trained video large scale multimodal model to generate a training k_1th response and a training k_2th response that are different from each other, or (ii) input the training video data and training query data into the (k-1)th trained video large scale multimodal model, and use the (k-1)th trained video large scale multimodal model to generate a training k_1th response, and then input the training video data and training query data again into the (k-1)th trained video large scale multimodal model, and use the (k-1)th trained video large scale multimodal model to generate a training k_2th response that is different from the training k_1th response, but is not limited to this.

[0055] The learning device 100 may also be configured to use the (k-1)th trained video large-scale multimodal model 300_k to embed training video data and training query data through an embedding layer to generate a kth embedding vector, and to use the kth embedding vector through a large-scale language model to generate a kth training visual context, the k_1st training response, and the k_2nd training response.

[0056] In this manner, the learning device 100 generates a k-th training visual context, a k-th training response, and a k-th training response using the (k-1)th trained video large-scale multimodal model 300_k. Then, the learning device 100 can generate a k-th training preference dataset 310_k, including the k-th training visual context, the k-th training response, and the k-th training response, as well as the training video data and training query data used as input to the (k-1)th trained video large-scale multimodal model 300_k. In this case, the k-th training preference dataset 310_k can be used as input data to be input to the (k-1)th trained video large-scale multimodal model 300_k in a subsequent process described below. An example of the k-th training preference dataset 310_k will be described with reference to FIG. 5 .

[0057] FIG. 5 is a diagram that schematically illustrates an example of a training preference dataset for training a video large-scale multi-modal model according to one embodiment of the present invention.

[0058] First, it can be seen that (a) in Figure 5 shows examples of training image data and training query data, (b) shows an example of a training kth visual context, and (c) shows examples of a training k_1th response and a training k_2th response.

[0059] In this case, the same training video data and training query data are repeatedly used from the first iteration to the nth iteration, the kth training visual context is generated by the (k-1)th trained video large-scale multimodal model by referring to the (k-1)th training visual context and the training video data, and the k_1th training response and the k_2nd training response may each be generated by referring to the training video data and the training query data.

[0060] Furthermore, when the k_1st training response (e.g., y1) is checked, some of the content may be underlined, indicating that the content is incorrect information unrelated to the training video data, and when the k_2nd training response (e.g., y2) is checked, some of the content may be bolded, indicating that the content is correct information whose basis is confirmed in the training video data, but this is not limited to this. For example, when the k_1st training response and the k_2nd training response are generated in this manner, in a subsequent process described below, the learning device 100 can determine the k_1st training response as a non-preferred response and the k_2nd training response as a preferred response using the (k-1)th trained video large-scale multimodal model 300_k.

[0061] Next, referring again to Figure 3, the learning device 100 inputs the kth training preference data set into the (k-1)th trained video large-scale multimodal model, and, using the (k-1)th trained video large-scale multimodal model, determines one of the training k_1st response and the training k_2nd response as the training kth preference response by referring to the kth training preference data set, and determines the other response as the training kth non-preference response, thereby generating (S220_k) the kth training preference feedback data including the training kth preference response, the training kth non-preference response, the training video data, and the training query data.

[0062] 4b(b) for the S220_k process, in the S210_k process, the (k-1)th trained video large-scale multimodal model 300_k was used to generate the k-th training visual context and generate the k-th training response and the k-th training response. However, in the S220_k process, the (k-1)th trained video large-scale multimodal model 300_k may be used as a model to determine the preference for each of the k-th training response and the k-th training response by referring to the k-th training preference dataset 310_k. Furthermore, the learning device 100 may refer to the k-th training visual context when determining each of the k-th training preference response and the k-th training non-preference response using the (k-1)th trained video large-scale multimodal model 300_k, thereby further improving the accuracy of preference determination.

[0063] Therefore, the learning device 100 can generate training kth preference feedback data 320_k including the training kth preference response and the training kth non-preference response determined by the (k-1)th trained video large-scale multimodal model 300_k, and the training video data and training query data included in the training kth preference dataset 310_k, and the training kth preference feedback data 320_k can be used to train the (k-1)th trained video large-scale multimodal model 300_k.

[0064] Next, referring again to Figure 3, the learning device 100 can use DPO to generate the kth DPO loss for the kth learning preference feedback data, and use the kth DPO loss to update the parameters of the (k-1)th learned video large-scale multimodal model, thereby generating the kth learned video large-scale multimodal model (S230_k).

[0065] 4b(c) for the S230_k process, the learning device 100 can input the training video data and training query data included in the k-th training preference feedback data 320_k to the reference video massive multimodal model 400 corresponding to the (k-1)th trained video massive multimodal model 300_k, and output a training reference preference response and a training reference non-preference response, respectively, using the reference video massive multimodal model 400. Furthermore, the learning device 100 can use the DPO to generate the k-th DPO loss by referring to the k-th training preference response and the k-th training non-preference response, respectively, and the training reference preference response and the training reference non-preference response, respectively, included in the k-th training preference feedback data 320_k. The reference video large-scale multimodal model 400 is a model that has been pre-trained in a supervised manner using training data for a specified task, and can be used as a base model for generating the kth DPO loss, and the training reference preference response and training reference non-preference response can be used as GTs for the training kth preference response and training kth non-preference response, respectively.

[0066] Specifically, the learning device 100 may input the kth training preference response, the kth training non-preference response, the training reference preference response, and the training reference non-preference response to the loss layer 330, and through the loss layer 330, substitute reward modeling for the preference response by referring to the ratio of the kth training preference response to the training reference preference response, and substitute reward modeling for the non-preference response by referring to the ratio of the kth training non-preference response to the training reference non-preference response, thereby generating the kth DPO loss.

[0067] In this case, the kth DPO loss can be expressed by the following formula:

number

[0068] In the above formula,

number

number

[0069] Therefore, when the kth DPO loss is generated through the loss layer 330, the learning device 100 can generate the kth trained video massive multimodal model by updating the parameters of the (k-1)th trained video massive multimodal model 300_k using the kth DPO loss.

[0070] Meanwhile, the results of inputting the same video data and query data into a video massive multimodal model trained according to the process of the present invention and a conventional video massive multimodal model and comparing the differences in the responses generated are described with reference to Figure 6.

[0071] Figure 6 is a diagram showing an example of response results generated by inputting the same data into a trained video massive multimodal model of the present invention and a conventional video massive multimodal model after a predetermined number of iterative training steps have been completed for the video massive multimodal model according to one embodiment of the present invention.

[0072] Referring to (a) of Figure 6, the top row shows video data in which an athlete is seen performing a long jump, and query data asking what the athlete is doing in the video. The video data and query data may be input to the trained video massive multimodal model of the present invention and a conventional video massive multimodal model, respectively.

[0073] Therefore, referring to (b) of Figure 6, when the iteration is applied five times, the response results generated by each of the trained video large-scale multimodal model of the present invention and the conventional video large-scale multimodal model are examined. It can be seen that the response generated through the conventional video large-scale multimodal model contains some content shown in bold, i.e., content that is based on the video data, but also contains a considerable amount of content shown in underline, i.e., content that is unrelated to the video data, whereas the response generated through the video large-scale multimodal model of the present invention mainly contains content shown in bold, i.e., content that is based on the video data, and does not contain any unnecessary content.

[0074] Furthermore, it can be seen that when nine iterations are applied, the responses generated by the conventional video large scale multimodal model contain a lot of underlined content, i.e., content unrelated to the video data, whereas the responses generated by the video large scale multimodal model of the present invention contain more specifically bolded content, i.e., content based on the video data. In other words, when the video large scale multimodal model generates a response corresponding to query data by referring to video data, the learning method of the present invention allows the generated responses to be more specifically described as the iterations increase, while avoiding the inclusion of content unrelated to the video data.

[0075] Meanwhile, the advantages of a method for learning a video large-scale multimodal model through iterative self-retrospective judgment according to an embodiment of the present invention will be described below. [Table 1]

[0076] The above experimental results compare the performance difference between our method (Iterative Self-Retrospective Judgment, i-SRT) and related state-of-the-art techniques using an in-domain zero-shot video question answering (VAQ) dataset.

[0077] Specifically, the learning methods that showed the best performance in both accuracy and score for each of the Activity Net-QA, VIDAL-QA, and WebVid-QA datasets are displayed in bold, confirming that the performance of the i-SRT of the present invention is the best. [Table 2]

[0078] The above experimental results compare the performance difference between our method (i-SRT) and related state-of-the-art techniques using an out-domain zero-shot video question-answering dataset.

[0079] Specifically, the learning methods that showed the best performance in both accuracy and score for each of the MSVD-QA, MSRVTT-QA, TGIF-QA, and SSV2-QA datasets are displayed in bold, confirming that the performance of the i-SRT of the present invention is the best.

[0080] The above-described embodiments of the present invention may be implemented in the form of program instructions that can be executed by various computer components and stored on a computer-readable storage medium. The computer-readable storage medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions stored on the computer-readable storage medium may be specially designed and constructed for the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine language code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processes according to the present invention, or vice versa.

[0081] The present invention has been described above based on specific details such as specific components and limited examples and drawings, but this is merely provided to facilitate a more comprehensive understanding of the present invention, and the present invention is not limited to the above examples. A person having ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from such descriptions.

[0082] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and all modifications equivalent to or equivalent to the scope of the claims, as well as the scope of the claims, are within the spirit of the present invention.

Claims

1. A method for learning a large-scale multimodal model of video through iterative self-retrospective judgment, (a) a learning device (i) inputs training video data and text-format training query data into an initial video massively multimodal model, and generates a training first visual context for the training video data using the initial video massively multimodal model, and generates a training first_1 response and a training first_2 response corresponding to the training query data by referring to the training video data (the training first_2 response is a response different from the training first_1 response), thereby generating the training first visual context, the training first_1 response, the training first_2 response, the training video data, and the training query data; (ii) inputting the first training preference dataset into the initial video massively multimodal model, and determining, with the initial video massively multimodal model, one of the first training response_1 and the first training response_2 as a first training preference response by referring to the first training preference dataset, and determining the other response as a first training non-preference response, thereby generating first training preference feedback data including the first training preference response, the first training non-preference response, the training video data, and the training query data; and (iii) using Direct Pitch Order (DPO) to generate first training preference feedback data including the first training preference response, the first training non-preference response, the training video data, and the training query data. generating a first DPO loss for the first training preference feedback data using Preference Optimization (PPO), and updating parameters of the initial video massive multimodal model using the first DPO loss to generate a first trained video massive multimodal model; and (b) a learning device (i) inputs a (k-1)th training visual context (where k is an integer increasing from 2 to n, and n is an integer equal to or greater than 2), the training video data, and the training query data into a (k-1)th trained video massive multimodal model, and generates a k-th training visual context for the training video data by referring to the (k-1)th training visual context and the training video data using the (k-1)th trained video massive multimodal model, and generates a k-th training response and a k-th training response corresponding to the training query data by referring to the training video data (the k-1)th training response and the k-2) (the training response is a response different from the k-1)th training response), and generates a k-th training preference data including the k-th training visual context, the k-1st training response, the k-2nd training response, the training video data, and the training query data; (ii) inputting the k-th training preference dataset into the (k-1) trained video massive multimodal model, and using the (k-1) trained video massive multimodal model, determining one of the training k_1st response and the training k_2nd response as the training k-th preference response by referring to the k-th training preference dataset, and determining the other response as the k-th training non-preference response, thereby generating training k-th preference feedback data including the training k-th preference response, the training k-th non-preference response, the training video data, and the training query data; (iii) generating a k-th DPO loss for the training k-th preference feedback data using the DPO, and updating parameters of the (k-1) trained video massive multimodal model using the k-th DPO loss, thereby generating a k-th trained video massive multimodal model; A method comprising:

2. In the step (a), In (iii), the learning device uses the DPO to create a first DPO loss by referring to each of the first training preference response and the first training non-preference response in the first training preference feedback data, and each of the training reference preference response and the training reference non-preference response output from a reference video large scale multimodal model corresponding to the initial video large scale multimodal model by inputting the training video data and the training query data into the reference video large scale multimodal model; In the step (b), The method of claim 1, wherein in (iii), the learning device uses the DPO to generate the kDPO loss by referring to each of the training k-th preference response and the training k-th non-preference response in the training k-th preference feedback data and each of the training reference preference response and the training reference non-preference response output from the reference video large-scale multimodal model by inputting the training video data and the training query data into the reference video large-scale multimodal model.

3. The method of claim 2 , wherein the reference video large-scale multi-modal model is a supervised trained model and is a base model for generating the first through kth DPO losses.

4. In the step (a), the learning device sets a temperature hyperparameter of the initial video massively multimodal model to a specific temperature hyperparameter value equal to or greater than a preset threshold, inputs the training video data and the training query data into the initial video massively multimodal model, and uses the initial video massively multimodal model to generate the training first_1 response and the training first_2 response, respectively, using the specific temperature hyperparameter value; In the step (b), 2. The method of claim 1 , wherein the learning device sets the temperature hyperparameter of the (k-1) trained video massively multimodal model to the specific temperature hyperparameter value, inputs the training video data and the training query data to the (k-1) trained video massively multimodal model, and uses the (k-1) trained video massively multimodal model to generate the training k_1th response and the training k_2th response using the specific temperature hyperparameter value.

5. In the step (a), the learning device uses the initial video large-scale multimodal model to embed the training video data and the training query data in text format through an embedding layer to generate a first embedding vector, and uses the first embedding vector through a large-scale language model to generate the first training visual context, the first training response_1, and the first training response_2; In the step (b), 2. The method of claim 1, wherein the learning device uses the (k-1)th learned video large-scale multimodal model to embed the training video data and the training query data through the embedding layer to generate a k-th embedding vector, and uses the k-th embedding vector through the large-scale language model to generate the k-th training visual context, the k_1st training response, and the k_2nd training response.

6. A learning device for learning a large-scale multimodal model of video through iterative self-retrospective judgment, at least one memory for storing instructions; and at least one processor configured to execute the instructions; The processor (I) (i) inputs training video data and text-based training query data into an initial video massive multimodal model, and generates a training first visual context for the training video data using the initial video massive multimodal model, and generates a training first_1 response and a training first_2 response corresponding to the training query data by referring to the training video data (the training first_2 response is a response different from the training first_1 response), thereby generating the training first visual context, the training first_1 response, the training first_2 response, the training video data, and the training query data. (ii) inputting the first training preference dataset into the initial video massively multimodal model, and determining, with reference to the first training preference dataset, one of the first training response_1 and the first training response_2 as a first training preference response and determining the other response as a first training non-preference response, thereby generating first training preference feedback data including the first training preference response, the first training non-preference response, the training video data, and the training query data; and (iii) using Direct Pitch Order (DPO) to generate first training preference feedback data including the first training preference response, the first training non-preference response, the training video data, and the training query data. generating a first DPO loss for the first training preference feedback data using Preference Optimization (PPO), and updating parameters of the initial video massive multimodal model using the first DPO loss to generate a first trained video massive multimodal model;and (II) (i) inputting a (k-1)th training visual context (where k is an integer increasing from 2 to n, and n is an integer equal to or greater than 2), the training video data, and the training query data into a (k-1)th trained video massive multimodal model, and using the (k-1)th trained video massive multimodal model to generate a k-th training visual context for the training video data by referring to the (k-1)th training visual context and the training video data, and to generate a k_1st training response and a k_2nd training response corresponding to the training query data by referring to the training video data (the training k_2nd training response is a response different from the k_1st training response), thereby obtaining a k-th training preference dataset including the k-th training visual context, the k_1st training response, the k_2nd training response, the training video data, and the training query data. (ii) inputting the k-th training preference dataset into the (k-1) trained video massive multimodal model, and using the (k-1) trained video massive multimodal model, determining one of the training k_1st response and the training k_2nd response as the training k-th preference response by referring to the k-th training preference dataset and determining the other response as the training k-th non-preference response, thereby generating training k-th preference feedback data including the training k-th preference response, the training k-th non-preference response, the training video data, and the training query data; and (iii) generating a k-th DPO loss for the training k-th preference feedback data using the DPO, and generating a k-th trained video massive multimodal model by updating parameters of the (k-1) trained video massive multimodal model using the k-th DPO loss.

7. the processor: In (iii) of the process (I), using the DPO, the first DPO loss is generated by referring to each of the first training preference response and the first training non-preference response in the first training preference feedback data, and each of the training reference preference response and the training reference non-preference response output from a reference video large-scale multimodal model corresponding to the initial video large-scale multimodal model by inputting the training video data and the training query data into the reference video large-scale multimodal model; The learning device of claim 6, wherein in (iii) of the (II) process, the kth DPO loss is generated by using the DPO to refer to each of the kth training preference response and the kth training non-preference response in the kth training preference feedback data, and each of the training reference preference response and the training reference non-preference response output from the reference video large-scale multimodal model by inputting the training video data and the training query data into the reference video large-scale multimodal model.

8. The learning device according to claim 7 , wherein the reference video large-scale multi-modal model is a supervised trained model and is a base model for generating the first DPO loss through the kth DPO loss.

9. the processor: In the process (I), setting a temperature hyperparameter of the initial video massively multimodal model to a specific temperature hyperparameter value equal to or greater than a preset threshold, inputting the training video data and the training query data into the initial video massively multimodal model, and using the initial video massively multimodal model to generate the training first_1 response and the training first_2 response, respectively, using the specific temperature hyperparameter value; In the process (II), The learning device of claim 6, wherein the temperature hyperparameter of the (k-1) trained video massively multimodal model is set to the specific temperature hyperparameter value, the training video data and the training query data are input to the (k-1) trained video massively multimodal model, and the (k-1) trained video massively multimodal model is used to generate the training k_1th response and the training k_2th response using the specific temperature hyperparameter value.

10. the processor: In the process (I), Using the initial video large-scale multimodal model, embed the training video data and the training query data in text format through an embedding layer to generate a first embedding vector, and using the first embedding vector through a large-scale language model to generate the first training visual context, the first training response_1 and the first training response_2; In the process (II), The learning device of claim 6, wherein the learning device uses the (k-1)th learned video large-scale multimodal model to embed the training video data and the training query data through the embedding layer to generate a k-th embedding vector, and uses the k-th embedding vector through the large-scale language model to generate the k-th training visual context, the k_1st training response, and the k_2nd training response.

Images