Method for pre-training a visual-language transformer and an artificial intelligence system including a visual-language transformer pre-trained thereby.

Abstract

The present invention relates to a method and system for augmenting large, uncurated datasets through image augmentation and / or masking, and for pre-training vision language transformers by reflecting misaligned information between augmented images and text through a knowledge distillation framework, thereby reducing data processing overhead and enabling rapid learning of simplified vision language transformers.

Description

[Technical Field]

[0001] This invention relates to a method for pre-training a transformer with visual and linguistic data, and to an artificial intelligence system including a visual transformer pre-trained therethrough. [Background technology]

[0002] In recent years, AI-based computer vision processing technology has been rapidly developing, with the emergence of pre-trained vision-language models (VLPs) that are used on large-scale general domain data.

[0003] In particular, as described in prior art 1, vision transformers trained on large image-text datasets using techniques such as global self-attention and contrastive language-image pretraining have represented a groundbreaking advance in downstream tasks, including diverse and complex visual tasks.

[0004] However, fully training global self-attention, which is primarily driven by visual transformers, requires a large dataset and suffers from the problem of excessive data processing overhead.

[0005] To secure such large datasets, many methods are used to augment language data and / or visual data to obtain diverse data, such as randomly applying rotation, flipping, resizing, cropping, hue adjustment, enlargement, cutting, and Gaussian noise to existing images.

[0006] During the image augmentation process described above, particularly when randomly enlarging, shrinking, or cropping specific areas, a misalignment problem can occur where the existing image being augmented and the matching text do not match.

[0007] Furthermore, when a visual-language transformer is pre-trained in the same way as existing methods, based on pairs of pre-enhanced images and matched texts, a problem arises in that the final performance of the pre-trained visual-language transformer is degraded.

[0008] To overcome these problems, prior art 2 proposed a technique that introduces an additional external module to detect misalignment via an object sensor and, if misalignment is detected, correct the text via a summary extractor. Prior art 3 proposed a technique to match alignment during pre-training via station embedding. However, using such external modules increases the amount of data processing required, which places a heavy burden on pre-training. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, on 26 Feb 2021 [Patent Document 2] Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, and Chunhua Shen. Pyramidclip: Hierarchical feature alignment for vision-language model pretraining. arXiv preprint arXiv:2204.14095, 2022. 1, 2, 3 [Patent Document 3] Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for Contrastive language-Image pretraining. In Advances in Neural Information Processing Systems, 2022. 1, 2, 3, 5, 6, 7 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The visual-language transformer pretraining method according to the present invention attempts to secure pretraining data for multiple image-text pairs by randomly enlarging or masking multiple image data necessary for pretraining.

[0011] Furthermore, the visual-language transformer pre-training method according to the present invention aims to learn the visual-language transformer by utilizing misalignment of image-text pairs that occurs during random image enlargement or masking as useful information.

[0012] Furthermore, the present invention aims to develop an artificial intelligence system that can effectively perform various visual tasks by utilizing a pre-trained visual-language transformer in this manner. [Means for solving the problem]

[0013] The vision - language transformer pre - training method according to an embodiment of the present invention proposes a method for pre - training a vision - language transformer by randomly enhancing an image, intentionally inducing a mismatch between the enhanced image and text, and using misalignment as useful information.

[0014] Specifically, the vision - language transformer pre - training method according to an embodiment of the present invention proposes a misaligned image - text pre - training method using knowledge distillation (Misalign, Contrast then Distill: MCD, hereinafter referred to as "MCD pre - training method") that can utilize the misalignment of enhanced image - text pairs as useful information for pre - training.

[0015] Specifically, an embodiment of the present invention provides a method for pretraining a visual-language transformer on a computing system comprising memory and a processor, using a dataset of multiple original image-text pairs, comprising the steps of: generating a plurality of augmented images by randomly augmenting the original images; inputting text matched to the original images into a text encoder and outputting a text feature vector representation; inputting the original images and the plurality of augmented images into a teacher image encoder and outputting a first image feature vector representation; inputting the original images and the plurality of augmented images into a student image encoder and outputting a second image feature vector representation; generating a first alignment matrix for the text feature vector representation and the first image feature vector representation; learning the first alignment matrix so that the text feature vector representation and the first image feature vector representation are aligned to have similarity by positive and negative mapping relationships between image-text pairs; and performing knowledge distillation on the student image encoder by aligning the second alignment matrix to predict the output of the learned first alignment matrix.

[0016] In this case, the step of inputting to the teacher image encoder and outputting a first image feature vector representation may include the step of outputting a first image feature vector representation via the original image and a plurality of augmented images through a plurality of self-attention layers and a feed-forward network layer.

[0017] Furthermore, the step of inputting to the student image encoder and outputting a second image feature vector representation may include the step of inputting the original image and a plurality of augmented images to a token scarcity layer based on the output values ​​of a plurality of self-attention layers and performing token scarcity.

[0018] Furthermore, the step of learning the first alignment matrix so that the text feature vector representation and the first image feature vector representation are aligned may include the steps of determining positive feature vector representation pairs and negative feature vector representation pairs of the text feature vector representation and the first image feature vector representation based on the mapping relationship between the original image-text pair and the enhanced image-text pair, and learning the teacher image encoder with a loss function that causes the distance between the positive feature vector representation pairs to decrease and the distance between the negative feature vector representation pairs to increase for similarity alignment.

[0019] Furthermore, the step of training the encoder using the loss function may include the step of applying a momentum stop gradient to the teacher image encoder to block backpropagation during training for similarity sorting using the loss function.

[0020] Furthermore, the step of performing knowledge distillation on the student image encoder may include the step of performing knowledge distillation such that the output value of the first alignment matrix by similarity alignment is predicted by the second alignment metric.

[0021] Furthermore, the step of performing knowledge distillation on the second alignment matrix may include a step of blocking backpropagation to the text encoder during the knowledge distillation.

[0022] Furthermore, the step of performing knowledge distillation on the second alignment matrix may include the step of performing knowledge distillation such that the parameters of the second alignment matrix follow the parameters of the first alignment matrix.

[0023] Furthermore, the step of knowledge distillation such that the parameters of the second aligned matrix follow the parameters of the first aligned matrix may include the step of updating the parameters of the second aligned matrix to the parameters of the first aligned matrix using an exponential moving average (EMA).

[0024] Furthermore, the step of performing knowledge distillation on the second alignment matrix may include defining a loss function that reflects misalignment information between the augmented image and the text via the distance between the first image feature vector representation and the text feature vector representation and the distance between the second image feature vector representation and the text feature vector representation, and then performing knowledge distillation.

[0025] Furthermore, the step of defining a loss function based on the distance and performing knowledge distillation may include calculating a first Euclidean distance between the original image feature vector representation and the text feature vector representation output by the student image encoder, a second Euclidean distance between the augmented image feature vector representation and the text feature vector representation output by the student image encoder, and calculating a first log ratio obtained by calculating the ratio of the first Euclidean distance and the second Euclidean distance on a logarithmic scale.

[0026] Furthermore, the step of defining a loss function based on the distance and performing knowledge distillation may include calculating a third Euclidean distance between the original image feature vector representation and the text feature vector representation output by the teacher image encoder, a fourth Euclidean distance between the enhanced image feature vector representation and the text feature vector representation output by the teacher image encoder, and calculating a second log ratio obtained by calculating the ratio of the third Euclidean distance and the fourth Euclidean distance on a logarithmic scale.

[0027] Furthermore, the step of defining a loss function based on the distance and performing knowledge distillation may further include the step of defining the difference between the first log ratio and the second log ratio as a loss function for aligning the second alignment metric to approximate the first alignment metric and performing knowledge distillation.

[0028] At this time, the first alignment matrix and the second alignment matrix are aligned The loss function for this purpose can be defined as the following equation 3.

[0029]

number

[0030] Furthermore, the method may further include a step of performing an update via a stop gradient to prevent backpropagation between the teacher image encoder and the text encoder.

[0031] Furthermore, a pre-trained visual-language transformer in a computing system comprising a memory and a processor according to an embodiment of the present invention includes a text encoder that receives text and outputs a text feature vector representation, a teacher image encoder that receives an original image and a plurality of augmented images enhanced from the original image and outputs a first image feature vector representation, and a student image encoder that receives the original image and a plurality of augmented images enhanced from the original image and outputs a second image feature vector representation. The teacher image encoder and the text encoder learn a first alignment matrix by similarity alignment of the text feature vector representation and the first image feature vector representation, and the student image encoder undergoes knowledge distillation so that a second alignment matrix for the text feature vector representation and the second image feature vector representation predicts an output value in the first alignment matrix.

[0032] Furthermore, the present invention aims to provide an artificial intelligence system including a visual-language transformer that can perform visual tasks with improved performance by securing multiple sets of data necessary for pre-training via the image enhancement method, and efficiently pre-training the visual-language transformer with the multiple misaligned image-text pairs secured. [Effects of the Invention]

[0033] The visual-language transformer pre-training method according to an embodiment of the present invention expands multiple misaligned image-text pairs into pre-training data by randomly augmenting images, making it easy to secure multiple sets of pre-training data containing diverse and useful information from multiple perspectives.

[0034] Furthermore, the visual-language transformer pre-training method according to the embodiment of the present invention can provide a visual-language transformer with improved performance through an MCD pre-training method that can learn misalignment of augmented image-text pairs as useful information.

[0035] Furthermore, an artificial intelligence system including a pre-trained visual-language transformer according to an embodiment of the present invention can effectively perform visual tasks such as image classification, object detection, image segmentation, image captioning, image analysis, and optical character recognition by utilizing the visual-language transformer learned from the pre-training data containing diverse and useful information from multiple perspectives. [Brief explanation of the drawing]

[0036] [Figure 1] An example block diagram of a computing system that performs the MCD pre-training method according to an embodiment of the present invention is shown. [Figure 2] An example block diagram of a computing device that pre-trains a visual-language transformer using an MCD pre-training method according to an embodiment of the present invention and executes the pre-trained visual-language transformer is illustrated. [Figure 3]An MCD pre-training method according to an embodiment of the present invention is demonstrated, and an example of a block diagram in other aspects for a computing device that runs a pre-trained visual-language transformer is illustrated. [Figure 4] A conceptual framework of the MCD pre-training method according to an embodiment of the present invention is shown. [Figure 5] This shows the meta-architecture of the framework for the MCD pre-training method according to an embodiment of the present invention. [Modes for carrying out the invention]

[0037] The present invention can be modified in various ways and has various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, and the methods for achieving them, will become clear when referred to the embodiments described in detail below, along with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be realized in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a restrictive sense but to distinguish one component from another. Also, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, terms such as "includes" or "has" mean that the features or components described in the specification exist, and do not preclude the possibility of adding one or more other features or components. Also, in the drawings, for illustrative purposes, the size of components, etc., may be exaggerated or reduced. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for illustrative purposes, and the present invention is not necessarily limited to what is illustrated.

[0038] Figure 1 illustrates an example block diagram of a computing system that performs the MCD pre-training method according to an embodiment of the present invention.

[0039] As shown in Figure 1, a computing system 1000 according to one embodiment of the present invention comprises a user computing device 110, a training computing system 150, and a server computing system 130, and each device and system is connected to communicate via a network 170.

[0040] According to various embodiments of the present invention, 1) a user computing device 110 can locally pre-train a visual-language transformer 120 and run an application including the trained visual-language transformer 120; 2) a server computing system 130 communicating with the user computing device 110 can pre-train a visual-language transformer 120 or / and 140 and provide the user computing device 110 with an application including the visual-language transformer 120 or / and 140 directly or in the form of a web service; and 3) the user computing device 110 and the server computing system 130 can cooperate with each other to pre-train a visual-language transformer 120 or / and 140, or run the pre-trained visual-language transformer 120 or / and 140 to provide various application services.

[0041] Furthermore, according to various embodiments of the present invention, the user computing device 110 and / or the server computing system 130 can train the model 120 through interaction with a training computing system 150 which is communicatively connected via a network 170. In this case, the training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0042] In other words, the visual-language transformer pretraining method according to the embodiment allows 1) a user computing device 110 to pretrain the visual-language transformer 120 locally and directly, 2) a server computing system 130 and the user computing device 110 to interact with each other via a network for pretraining, and 3) another training computing system 150 to pretrain the visual-language transformer using various training and learning techniques.

[0043] Furthermore, this can also be implemented by having the training computing system 150 transmit and provide / / update pre-trained visual-language transformers 120 and / or 140 to the user computing device 110 and / or server computing system 130 via a network.

[0044] In some embodiments, the training computing system 150 may be part of the server computing system 130 or part of the user computing device 110.

[0045] Furthermore, the present invention, a method and system for pre-training visual-language transformers, can be included in applications that perform various downstream tasks, such as fine-tuning pre-trained visual-language transformers.

[0046] The user computing device 110 may include smartphones, mobile phones, digital broadcasting devices, PDAs (personal digital assistants), PMPs (portable multimedia players), desktops, wearable devices, embedded computing devices, and / or tablet PCs, as well as all other types of computing devices.

[0047] Such a user computing device 110 comprises at least one processor 111 and memory 112. Here, the processor 111 can consist of at least one or more electrically connected processors from among a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuits (ASICs), digital signal processors (DSSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for functional execution.

[0048] The memory 112 may include one or more non-temporary / temporary computer-readable storage media and combinations thereof, such as RAM, ROM, EEPROM, EPROM, flash memory devices, and magnetic disks, and may include web storage of a server that performs memory storage functions over the internet. Such memory 112 can store data 113 and instruction words 114 necessary for at least one of the processors 111 to pre-train the visual-language transformer 120 or to perform operations such as executing an application that includes the pre-trained visual-language transformer 120.

[0049] In one embodiment, the user computing device 110 can store at least one or more machine learning models (e.g., a visual-language transformer 120).

[0050] More specifically, the visual-language transformer 120 in one embodiment may be composed of various machine learning models such as multiple neural networks (e.g., deep neural networks), or other types of machine learning models including nonlinear and / or linear models, or combinations thereof.

[0051] Furthermore, the neural network may include at least one of the following: feed-forward neural networks, cyclic neural networks (e.g., long-short-term memory cyclic neural networks), convolutional neural networks, and / or other forms of neural networks.

[0052] In one embodiment, the user computing device 110 receives at least one or more visual-language transformers 120 from the server computing system 130 via the network 170, stores them in memory, and then the processor 111 executes the stored visual-language transformers 120 to run applications having various visual-based tasks.

[0053] In other embodiments, the server computing system 130 may include at least one machine learning model (e.g., a visual-language transformer 140) and operate via the visual-language transformer 140, providing the user with an artificial intelligence system that performs various tasks using the visual-language transformer 140 in conjunction with the user computing device 110 by communicating data related to the user computing device 110. For example, the user computing device 110 may perform a visual task including the visual-language transformer 140 in a manner in which the server computing system 130 uses the visual-language transformer 140 via the web to provide output for user input. Alternatively, the visual-language transformers 120 and / or 140 can be implemented in a manner in which at least a portion of the visual-language transformers 120 and / or 140 are executed on the user computing device 110, and the remainder are executed on the server computing system 130.

[0054] Furthermore, the user computing device 110 may include at least one input component that senses user input. For example, the user input component may include a touch sensor (e.g., a touchscreen and / or touchpad) that senses touch from the user's input medium (e.g., a finger or stylus), an image sensor that senses user motion input, a microphone that senses user voice input, buttons, a mouse, and / or a keyboard. The user input component may also include an interface and an external controller (e.g., a mouse, keyboard) if it receives input to an external controller via an interface.

[0055] The server computing system 130 comprises at least one processor 131 and memory 132. Here, the processor 131 can consist of at least one or more electrically connected processors from among a central processing unit (CPU), graphics processing unit (GPU), ASICs (application-specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for functional execution.

[0056] The memory 132 may include one or more non-temporary / temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. Such a memory 132 can store data 133 and instruction words 134 necessary for the processor 131 to pre-train the visual-language transformer 140 or to perform various visual tasks (e.g., image detection, classification, segmentation, etc.) using the visual-language transformer 140.

[0057] In one embodiment, the server computing system 130 can be implemented by including at least one computing device. For example, the server computing system 130 can be implemented so that multiple computing devices operate in a sequential computing architecture, a parallel computing architecture, or a combination thereof. The server computing system 130 can also include multiple computing devices connected by a network.

[0058] Furthermore, the server computing system 130 can house at least one or more visual-language transformers 140. For example, the server computing system 130 may include a neural network and / or other multi-layer nonlinear model as a visual-language transformer 140. Exemplary neural networks may include feedforward neural networks, deep neural networks, circular neural networks, and convolutional neural networks.

[0059] The training computing system 150 comprises at least one processor 151 and memory 152. Here, the processor 151 can consist of at least one or more electrically connected processors from among a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuits (ASICs), digital signal processors (DSSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for functional execution.

[0060] The memory 152 may include one or more non-temporary / temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. Such a memory 152 can store data 153 and instruction words 154 necessary for the processor 151 to learn the visual-language transformer.

[0061] For example, the training computing system 150 may include a model trainer 160 that pre-trains the visual-language transformers 120 and / or 140 stored in the user computing device 110 and / or the server computing system 130 using various training or learning techniques, such as backward propagation of errors (according to the framework shown in Figure 5).

[0062] For example, the model trainer 160 can update one or more parameters of the visual-language transformer 120 or / 140 in a backpropagation manner based on a defined loss function.

[0063] In some implementations, error backpropagation may include truncated backpropagation through time. The model trainer 160 may perform several generalization techniques (e.g., weight reduction, dropout, knowledge distillation, etc.) to improve the generalization ability of the trained visual-language transformers 120 and / or 140.

[0064] In particular, the model trainer 160 can train the visual-language transformers 120 and / or 140 based on a series of training data. The training data may include data in multiple different formats (multi-modal), such as images, audio samples, and text. Examples of image types that may be used may include general RGB images, video frames, LiDAR point clouds, X-ray images, computed tomography scans, superspectroscopic images, and / or various other forms of images.

[0065] Such trainer data and input data for downstream tasks can be provided by the user computing device 110 and / or the server computing system 130. When the training computing device trains the visual-language transformer 120 on specific data from the user computing device 110, the visual-language transformer 120 can be characterized into a personalized model.

[0066] The model trainer 160 includes computer logic that is utilized to provide the desired functionality. The model trainer 160 can be implemented with hardware, firmware, and / or software that control a general-purpose processor. For example, in one embodiment, the model trainer 160 includes a program file stored in a storage device, which is loaded into memory and can be executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instruction words, etc., stored in a tangible computer-readable storage medium such as a RAM hard disk or an optical or magnetic medium.

[0067] Network 170 includes, but is not limited to, 3GPP (3rd Generation Partnership Project) networks, LTE (Long Term Evolution) networks, WiMAX (World Interoperability for Microwave Access) networks, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth® networks, satellite broadcasting networks, analog broadcasting networks, and / or DMB (Digital Multimedia Broadcasting) networks.

[0068] In general, communication over network 170 can be carried out using any type of wired and / or wireless connection and via various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, SSL, SSL).

[0069] Figure 2 illustrates an example block diagram of a computing device that pre-trains a visual-language transformer using a knowledge distillation framework according to an embodiment of the present invention and runs the pre-trained visual-language transformer.

[0070] As shown in Figure 2, the computing device 100 included in the user computing device 110, the server computing system 130, and the training computing system 150 contains multiple applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more visual-language transformers. For example, an application may include a visual test application (e.g., detection, classification, segmentation, etc.) and text messaging applications, email applications, typography applications, virtual keyboard applications, browser applications, chatbot applications, etc. that include such visual tests.

[0071] In one embodiment, the computing device 100 may include a model trainer 160 for pre-training a visual-language transformer, storing and operating the pre-trained visual-language transformer to perform various visual tests using the visual-language transformer on input data.

[0072] Each application of the computing device 100 can communicate with several other components of the computing device 100, such as one or more sensors, context managers, device state components, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0073] Figure 3 illustrates an example of a block diagram in other aspects of a computing device 200 that pre-trains a visual-language transformer via a knowledge distillation framework according to an embodiment of the present invention and runs the pre-trained visual-language transformer.

[0074] As shown in Figure 3, the computing device 200 includes multiple applications (e.g., Application 1 to Application N). Each application can communicate with the central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a writing application, a virtual keyboard application, a browser application, and so on. In one embodiment, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0075] The central intelligence layer can include multiple visual-language transformers. For example, as shown in Figure 3, at least a portion of each visual-language transformer can be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single visual-language transformer. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer can be implemented within or separately from the operational structure of computing device 200.

[0076] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data storage facility for the computing device 200. As shown in Figure 3, the central device data layer can communicate with several other components of the computing device 200, such as one or more sensors, context managers, device state components, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0077] The technologies described herein refer not only to servers, databases, software applications, and other computer infrastructure systems, but also to actions taken and information transmitted to or from said systems. The inherent flexibility of computer infrastructure systems, etc., will be recognized as allowing for a wide range of possible configurations, combinations, and divisions of work and functionality between and from components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications can be implemented in a single system or in a distributed system across multiple systems. Distributed components can operate sequentially or in parallel.

[0078] The process by which such a computing system 1000 pre-trains a visual-language transformer using a knowledge distillation framework with a dataset augmented through random image expansion will be described in detail below with reference to Figures 4 to 6.

[0079] In this invention, the visual-language transformer refers to a visual-language-based artificial intelligence model (VLP) that has been pre-trained on a large-scale dataset to co-represent data of two heterogeneous forms: visual-language (image-text) pairs.

[0080] A visual-language transformer according to such an embodiment may include a single-stream model that transforms input data in which images and text are combined, and a dual (multi)-stream model that processes image-text through separate image encoders and text encoders.

[0081] In the following embodiments, for the sake of convenience, we will describe a visual-language transformer having a dual-stream architecture that is pre-trained on a dataset in which image-text is matched as a control target.

[0082] The visual-language transformer pretraining method according to the embodiment can facilitate pretraining of a dataset of control image-text pairs using a knowledge distilled encoder based on a plurality of augmented image-text pairs secured via image random augmentation.

[0083] As shown in Figures 4 and 5, the visual-language transformer pre-training architecture according to this embodiment comprises a text encoder 10, a teacher image encoder 20, and a student image encoder 30. The student image encoder 30 and the teacher image encoder 20 may include a multi-head self-attention layer and a feed-forward network. The student image encoder 30 may further include a token scarcity layer.

[0084] Here, the image-text dataset for pre-training is an uncurated dataset, meaning data that has not undergone tasks such as labeling or captioning.

[0085] In the embodiment, in order to clearly verify the efficiency of the visual-language transformer pre-training method according to this embodiment, at least one of the large open-source datasets CC (Conceptual Captions) 3M, YFCC (Yahoo Flickr Creative Commons) 15M, and YFCC 15M, 88M may be included as the dataset for pre-training.

[0086] Furthermore, this embodiment may include zero-shot image-text data in Flickr30K and / or MS-COCO as a downstream dataset to verify the performance of the pre-trained visual-language transformer.

[0087] In particular, the computing system 1000 according to the embodiment of the present invention can prepare a plurality of image-text pairs, each consisting of an image data (hereinafter referred to as "image") and text data (hereinafter referred to as "text") which is a label for the image, as a pre-training dataset.

[0088] Furthermore, the computing system 1000 can generate augmented images by randomly augmenting the original images to increase the diversity of the dataset and improve the generalization ability of the visual-language transformer.

[0089] The computing system 1000 according to this embodiment can generate multiple augmented images by applying, in particular, random image scaling (e.g., random cropping, random rotation, random flipping, color jittering, and / or random grayscale) during image augmentation in order to intentionally induce misalignment between the augmented image and the text.

[0090] Furthermore, the computing system 1000 can generate additional augmented images by applying a method of masking random areas of the image.

[0091] Thus, additional augmented images generated through random image enhancement can cause serious defects in the text pairs of the original images. A brief theoretical explanation for this is presented below.

[0092] First, we formulate the text feature vector T, the original image feature vector I, and the augmented image feature vector I' into a Markov chain T → I → I', which means that I' depends only on the original image I.

[0093] According to the data processing inequality theory, which states that data processing cannot increase the amount of information, the following formula can be derived.

[0094]

number

[0095] Here, |(·) represents mutual information, and the two pieces of information are the same only if image I and augmented image I' are captured identically and contain the same information for text T.

[0096] The computing system 1000 according to this embodiment aims to propose a novel method for utilizing the information imbalance between such original image-text pairs and augmented image-text pairs as learning information.

[0097] Figure 4 conceptually illustrates the framework of an MCD pre-training method according to an embodiment of the present invention.

[0098] As shown in Figure 4, the computing system 1000 according to the embodiment can pre-train the image encoders 20, 30 and the text encoder 10 in the following way: 1) As described above, generate multiple augmented images by augmenting the original image, and generate original image-text pairs and augmented image-text pairs; 2) Input the original image-text pairs and the multiple augmented image-text pairs into the teacher image encoder 20, the student image encoder 30, and the text encoder 10, respectively, and output feature vector representations for the image-text pairs; 3) Project the output image feature representation vector and the text feature representation vector using contrastive objectives to obtain the distance between the feature representation vectors; and 4) Distill the knowledge of the teacher image encoder 20 into the student image encoder 30 based on the obtained distance.

[0099] The following section will explain the MCD pre-training process in more detail.

[0100] First, the computing system 1000 can generate multiple augmented images by randomly scaling and / or masking the original image.

[0101] Next, the MCD pre-training method framework, as shown in Figure 4, comprises a momentum teacher image encoder 20, a student image encoder 30, and a text encoder 10, and the knowledge of the teacher image encoder 20 can be distilled into the student image encoder 30 based on a stop gradient.

[0102] Here, the momentum teacher image encoder 20 is a teacher image encoder 20 in which the model's parameters are updated more slowly over time, allowing the student image encoder 30 to be trained stably.

[0103] Furthermore, the student image encoder 30 can be a machine learning model that is relatively simple (e.g., has fewer parameters) compared to the teacher image encoder 20, which is trained to mimic the operation of the teacher image encoder 20.

[0104] Such a teacher-student image encoder 30 is trained to convert the features of the original image and the augmented image into feature representations, and can output an image feature vector representation of the input image.

[0105] Here, the feature vector representation refers to a vector that represents the features of an image object in n dimensions. It is a feature vector obtained by combining various features of a transformation object in a format that can be processed by a machine learning model and transforming it into a structured form, which can be called embedding.

[0106] Finally, the text encoder 10 is an encoder that, upon receiving text, outputs a text feature vector representation T. More specifically, the text encoder 10 is trained to convert text features into feature representations and can output a text feature vector representation T for the input text.

[0107] The computing system 1000 can obtain a first image feature vector representation, which includes the original image feature vector representation bar I and the augmented image feature vector representation bar I′, from the original image and the augmented image, respectively, using the teacher image encoder 20.

[0108] Furthermore, the computing system 1000 can input the original image and the enhanced image into the student image encoder 30 to obtain a second image feature vector representation that includes the original image feature vector representation I and the enhanced image feature vector representation I'.

[0109] The computing system 1000 can then input the text matched to the original image into the text encoder 10 to obtain a text feature vector representation T.

[0110] Next, the computing system 1000 can generate a first alignment matrix bar A by mapping the text feature vector representation T with the first image feature vector representation output from the teacher image encoder 20 using pre-matched positive pairs and unmatched negative pairs.

[0111] The computing system 1000 can then train the teacher image encoder 20 and the text encoder 10 using a similarity-based sorting method (for example, InfoNCE loss, a method that trains to maximize the similarity of positive pairs and minimize the similarity of negative pairs) to sort the first sorting matrix bar A using a positive / negative criterion that has been pre-mapped to the output text feature vector representation T and the first image feature vector representation.

[0112] In this process, the computing system 1000 can train the first alignment matrix bar A, composed of first image feature vector representations, for similarity alignment. The teacher image encoder 20 can be a Momentum Teacher with Stop Gradient momentum model, and therefore, backpropagation (sg) to the teacher image encoder 20 can be blocked during similarity alignment learning.

[0113] Subsequently, the computing system 1000 can be trained by a loss function such that the spatial distance between positive feature vector representations decreases and the spatial distance between negative feature vector representations increases for similarity alignment.

[0114] That is, score(f(x), f(x + )) / Positive samples >> score(f(x),f(x - Contrastive learning can be performed by defining a loss function that results in negative samples.

[0115] For example, as mentioned above, the computing system 1000 can train the teacher image encoder 20 by comparative learning of the first alignment matrix bar A so as to apply the loss function InfoNCE Loss to the similarity metric.

[0116] The computing system 1000 can then input the original image and the enhanced image into the student image encoder 30 and output a second image feature vector representation.

[0117] In this case, the student image encoder 30 can also accelerate pre-training by reconstructing patch tokens including a token scarcity layer. However, the token scarcity layer may be omitted.

[0118] More specifically, the student image encoder 30 calculates the attention value between images (self-attention) and can discard tokens below a predetermined threshold based on the calculated attention value between each image.

[0119] For example, the student image encoder 30 can discard inattentive tokens at a fixed rate (1-κ) based on the attention value between each patch of the 4th, 7th, and 10th converter layers of the self-attention layer, where κ is the token retention rate.

[0120] The computing system 1000 can then generate a second alignment matrix A by mapping the text feature vector representation T with the second image feature vector representation using pre-matched positive pairs and unmatched negative pairs.

[0121] Next, unlike existing knowledge distillation methods, the computing system 1000 can perform knowledge distillation so that the second aligned matrix A predicts the output value of the first aligned matrix bar A, which is aligned by similarity mapping.

[0122] In other words, the computing system 1000 can perform knowledge distillation by training the student image encoder 30 so that the second alignment matrix A is aligned by soft-aligning the first alignment matrix bar A.

[0123] In this case, the text encoder 10 and the teacher image encoder 20 can be a momentum model with a stop gradient that prevents the backpropagation (sg) of knowledge distillation to the text encoder 10.

[0124] More specifically, the computing system 1000 can learn the parameters of the student image encoder 30 and perform knowledge distillation so that the second alignment matrix A is aligned by the first alignment matrix bar A.

[0125] The computing system 1000 can then update the parameters of the teacher image encoder 20 to an exponential moving average (EMA) based on the parameters of the student image encoder 30.

[0126] In this case, the computing system 1000 can perform knowledge distillation by defining a loss function based on the distance between the image feature vector representation and the text feature vector representation T, in order to reflect misalignment information between the augmented image and the text, as described above.

[0127] More specifically, the computing system 1000 can calculate a first Euclidean distance between the original image feature vector representation I and the text feature vector representation T output by the student image encoder 30, and a second Euclidean distance between the enhanced image feature vector representation I' and the text feature vector representation T output by the student image encoder 30. It can then calculate a first ratio between the first and second Euclidean distances, and a logarithmic scale can be applied to this ratio. That is, the first ratio can be calculated using a logarithmic scale to obtain the first logarithmic ratio.

[0128] Furthermore, the computing system 1000 can calculate a third Euclidean distance between the original image feature vector representation bar I and the text feature vector representation T output by the teacher image encoder 20, and a fourth Euclidean distance between the enhanced image feature vector representation bar I′ and the text feature vector representation T output by the teacher image encoder 20, and calculate a second ratio between the third Euclidean distance and the fourth Euclidean distance, at which time a logarithmic scale can be applied. That is, the second ratio can be calculated using a logarithmic scale to calculate the second logarithmic ratio.

[0129] The computing system 1000 can then train an encoder by defining the difference between the first log ratio and the second log ratio as a loss function for aligning the second alignment metric to approximate the first alignment matrix bar A.

[0130] The following section will explain in detail the calculation process for the above pre-training using specific mathematical formulas.

[0131] Specifically, a function for the momentum teacher image encoder 20 with a stopping gradient, and a function A￣ representing the first alignment matrix A′ for the momentum text encoder 10 with a stopping gradient. ij This is a function A that represents the second sorted matrix A. ij This can be defined as shown in the following equation 2.

[0132]

Number

[0133] Here, sg is the stop gradient, and I￣ j | = f￣(x I j ) and I j | = f(x I j ) are the image feature representations for the j-th image obtained using the teacher image encoder 20 and the student image encoder 30, respectively, and T i = f T (x T i ) is the text feature representation T for the i-th text, A ∈ R N×N￣ is the alignment matrix for the image feature representation and the text feature representation, N is the patch size of the image-text pair, and sim means the function for the cosine similarity.

[0134] And, as described above, the loss of the first alignment matrix bar A can be obtained using the InfoNCE loss.

[0135] And, as described above, a loss function can be defined and pre-trained based on the distances between the original image feature vector representations I, bar I and the enhanced image feature vector representations I′, barI′, and the text feature vector representation T, and this process will be described through Equations 3 to 6.

[0136] And, the computing system 1000 can define a loss function for aligning the first alignment matrix bar A and the second alignment matrix A based on the InfoNCE loss.

[0137]

Number

[0138] Here, l i represents the align loss, and D(V1, V2) represents the Euclidean distance between the V1 vector and the V2 vector, which can be calculated via cosine similarity.

[0139] Therefore, D(I i ,T j ) is the first Euclidean distance, and D(I' i ,T j ) is the second Euclidean distance, and D(I￣ j ,T i ) is the third Euclidean distance, and D(I￣' i ,T j ) is the fourth Euclidean distance.

[0140] Specifically, in the embodiment, the original image feature vector I i and the text feature vector T j This is an L2-normalized vector, and the Euclidean distance is the cosine similarity function D(I i ,T j )=2(1-A ij It can be calculated via ).

[0141] Next, the computing system 1000 can progressively distill the knowledge of the teacher image encoder 20 into the student image encoder 30 based on the loss function described above.

[0142] More specifically, as described above, the computing system 1000 performs knowledge distillation to predict that the second sorting matrix and the first sorting matrix will match.

[0143] More specifically, distillation loss is defined as the KL divergence for each row and column between the first aligned matrix bar A and the second aligned matrix A. More specifically, when distillation loss is defined as the KL divergence for each row and column between the first aligned matrix bar A and the second aligned matrix A, if σ is the softmax function, then D is the KL divergence between the first aligned matrix bar A and the second aligned matrix A. KL (A￣||A) can be expressed as shown in equation 4 below, and the distillation loss can be calculated using equation 4.

[0144]

number

[0145] Here, Ldistill, which is the overall distillation loss, is the average of the KL losses for the row and column vectors of the parameters of the first and second aligned matrices, respectively, and can therefore be defined as shown in Equation 5 below.

[0146]

number

[0147] Then, in order to accelerate the training of the student image encoder 30 and to balance between knowledge distillation training and teacher image encoder 20 training, the final loss Lstudent of the student image encoder 30 can be calculated as number 6 based on Ldistill and InfoNCE loss Lclip(A).

[0148]

number

[0149] Here, λ is a parameter that balances the KL divergence loss and the InfoNCE loss, and in the embodiment, it is set based on the exponential moving average (ema).

[0150] Therefore, the final loss L of the MCD pre-training can be calculated as shown in Equation 7.

[0151]

number

[0152] As mentioned above, the teacher image encoder 20 and the text encoder 10 can update the parameter values ​​of encoders 10 and 20 via a stop gradient to prevent backpropagation.

[0153] For more details, see q fI and q f￣I This refers to the parameters of the student encoder and the momentum teacher image encoder 20, respectively, and q in the t-th step (t) f￣I The update can be performed by the following number 8.

[0154]

number

[0155] The experimental results showed that the most efficient training could be performed when m was 0.994.

[0156] The following describes how to compare the effects of a visual-language transformer learned through MCD pre-training according to an embodiment of the present invention with existing technologies.

[0157] The artificial intelligence system including the visual-language transformer of the present invention can perform visual tasks such as image classification, segmentation, object detection, image generation, automatic caption generation, image search, and image description with relatively higher accuracy compared to existing transformers.

[0158] Table 1 below compares the zero-shot image classification performance of an MCD model pre-trained on the YFCC15M dataset, which includes 11 downstream datasets, using the MCD pre-training method, with that of a visual-language transformer model trained on the YFCC15M dataset using existing techniques. In this table, whether or not additional oversight other than contrast loss was performed on image-text pairs is expressed as S: augmentation inter-SSL, E: text augmentation, N: nearest neighbor, L: masked language modeling, and I: augmentation information X encoded by an additional embedding layer.

[0159] [Table 1]

[0160] As can be seen from the table above, the MCD model that only uses enhanced SSL performs better than existing technologies in 9 out of 11 downstream datasets, and the average performance has also improved considerably.

[0161] Therefore, the computing system 1000 can run various applications, including a visual-language transformer with excellent performance for such visual tasks, and perform various artificial intelligence tasks.

[0162] Furthermore, this framework, which utilizes token scarcity and knowledge distillation for pre-training of contrasting language images, could be extended and applied at the level of a typical engineer to pre-train additional formats such as audio.

[0163] The embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specifically designed and configured for the present invention, or may be publicly known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. Hardware devices can be modified into one or more software modules to perform the processing according to the present invention, and vice versa. [Industrial applicability]

[0164] This invention provides a method for pre-training a visual transformer via a computing system and is a fundamental invention for performing tasks such as visual tasks using the pre-trained visual transformer, and therefore has industrial applicability.

Claims

1. A method for pre-training a visual-language transformer on a dataset of multiple original image-text pairs in a computing system comprising memory and a processor, The steps include generating multiple enhanced images by randomly enhancing the original image, The steps include inputting text matched to the original image into a text encoder and outputting a text feature vector representation, The steps include inputting the original image and multiple augmented images into a teacher image encoder to output a first image feature vector representation, The steps include inputting the original image and multiple augmented images into a student image encoder to output a second image feature vector representation, A step of generating a first alignment matrix for the text feature vector representation and the first image feature vector representation, A step of learning the first alignment matrix such that the text feature vector representation and the first image feature vector representation are aligned such that they have similarity based on the positive and negative mapping relationship between image-text pairs, The process includes the step of performing knowledge distillation on a student image encoder by aligning the second alignment matrix to predict the output of the learned first alignment matrix, Pre-training methods for visual-language transformers.

2. The step of inputting to the aforementioned teacher image encoder and outputting a first image feature vector representation is: The process includes the step of outputting a first image feature vector representation of the original image and multiple augmented images via multiple self-attention layers and a feed-forward network layer. The method for pre-training a visual-language transformer according to claim 1.

3. The step of inputting to the aforementioned student image encoder and outputting a second image feature vector representation is: The process includes inputting the original image and multiple augmented images into a token dilution layer based on the output values ​​of multiple self-attention layers to perform token dilution. The method for pre-training a visual-language transformer according to claim 2.

4. The step of learning the first alignment matrix so that the text feature vector representation and the first image feature vector representation are aligned is: The steps include determining a positive feature vector representation pair and a negative feature vector representation pair between the text feature vector representation and the first image feature vector representation based on the mapping relationship between the original image-text pair and the enhanced image-text pair, The process includes the step of training a teacher image encoder with a loss function that causes the distance between pairs of positive feature vector representations to decrease and the distance between pairs of negative feature vector representations to increase in order to achieve similarity alignment. The method for pre-training a visual-language transformer according to claim 1.

5. The step of training the encoder using the loss function is: The loss function includes the step of applying a momentum stop gradient to the teacher image encoder to block backpropagation during learning for similarity sorting, The method for pre-training a visual-language transformer according to claim 4.

6. The step of performing knowledge distillation on the aforementioned student image encoder is: The process includes a step of knowledge distillation such that the output value of the first alignment matrix obtained by the similarity alignment is predicted by the second alignment metric, The method for pre-training a visual-language transformer according to claim 5.

7. The step of performing knowledge distillation on the second alignment matrix is: The knowledge distillation includes a step of blocking backpropagation to the text encoder, The method for pre-training a visual-language transformer according to claim 6.

8. The step of performing knowledge distillation on the second alignment matrix is: The process includes the step of knowledge distillation such that the parameters of the second alignment matrix follow the parameters of the first alignment matrix. The method for pre-training a visual-language transformer according to claim 7.

9. The step of knowledge distillation such that the parameters of the second alignment matrix follow the parameters of the first alignment matrix is: The procedure includes the step of updating the parameters of the second aligned matrix with the parameters of the first aligned matrix using an exponential moving average (EMA). The method for pre-training a visual-language transformer according to claim 8.

10. The step of performing knowledge distillation on the second alignment matrix is: The step of knowledge distillation includes defining a loss function that reflects misalignment information between the augmented image and the text via the distance between the first image feature vector representation and the text feature vector representation and the distance between the second image feature vector representation and the text feature vector representation, The method for pre-training a visual-language transformer according to claim 1.

11. The step of defining a loss function based on the aforementioned distance and performing knowledge distillation is, The process includes the steps of calculating a first Euclidean distance between the original image feature vector representation and the text feature vector representation output by the student image encoder, a second Euclidean distance between the augmented image feature vector representation and the text feature vector representation output by the student image encoder, and a first log ratio calculated on a logarithmic scale from the ratio of the first Euclidean distance to the second Euclidean distance. The method for pre-training a visual-language transformer according to claim 10.

12. The step of defining a loss function based on the aforementioned distance and performing knowledge distillation is, The process includes the steps of calculating a third Euclidean distance between the original image feature vector representation and the text feature vector representation output by the teacher image encoder, a fourth Euclidean distance between the enhanced image feature vector representation and the text feature vector representation output by the teacher image encoder, and a second log ratio calculated on a logarithmic scale from the ratio of the third Euclidean distance to the fourth Euclidean distance. The method for pre-training a visual-language transformer according to claim 11.

13. The step of defining a loss function based on the aforementioned distance and performing knowledge distillation is, The process further includes the step of performing knowledge distillation, where the difference between the first log ratio and the second log ratio is defined as a loss function for aligning the second alignment metric to approximate the first alignment metric. The method for pre-training a visual-language transformer according to claim 12.

14. The loss function for aligning the first and second alignment matrices is defined as the following equation 3: The method for pre-training a visual-language transformer according to claim 13. [Math 3]

15. The step further includes performing an update via a stop gradient to prevent backpropagation between the teacher image encoder and the text encoder. The method for pre-training a visual-language transformer according to claim 10.

16. Performing a visual test including a pre-trained visual-language transformer according to claim 1, Artificial intelligence system.

17. A pre-trained visual-language transformer comprising a computing system with memory and a processor, A text encoder that receives text and outputs a text feature vector representation, A teacher image encoder receives an original image and a plurality of enhanced images obtained from the original image and outputs a first image feature vector representation, A student image encoder that receives the original image and a plurality of enhanced images obtained from the original image and outputs a second image feature vector representation, The teacher image encoder and the text encoder learn a first alignment matrix by similarity alignment of the text feature vector representation and the first image feature vector representation. The student image encoder is pre-trained, knowledge distilled so that a second alignment matrix for the text feature vector representation and the second image feature vector representation predicts output values ​​in the first alignment matrix. A visual-language transformer.

Images