relative separation of contrastive learning

Abstract

Methods, systems, and devices, including computer programs encoded on computer storage media, for training neural networks through contrastive learning. In particular, contrastive learning is modified to use relative margins to adjust the contribution of training pairs to optimization.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 541,254, filed on September 28, 2023, the entire contents of which are incorporated herein by reference. Background Technology

[0003] This manual relates to using machine learning models to process data.

[0004] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received input. Some machine learning models are parametric models, and they generate outputs based on the received input and the values ​​of the model parameters.

[0005] Some machine learning models are deep models, which employ multiple layers to generate outputs from received inputs. For example, a deep neural network is such a deep machine learning model: it includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification describes techniques for using contrastive learning to train dual encoder machine learning models to perform downstream tasks such as detection tasks (e.g., text-based object detection in an image, audio-based object detection in an image, etc.) or generation tasks (e.g., image generation based on audio data, document generation based on text prompts, audio generation based on audio prompts, image generation based on text prompts, caption generation based on an image, audio generation based on an image, etc.).

[0007] The subject matter described herein may be implemented in specific embodiments in order to achieve one or more of the following advantages.

[0008] The system described in this specification can perform contrastive pre-training, that is, the system can "contrarianly" pre-train a first encoder neural network and a second encoder neural network of a machine learning model to learn representations of a first input modality (e.g., an image) and a second input modality (e.g., text) in a shared embedding space based on a contrastive loss, such that embeddings of semantically similar first and second modalities tend to be closer in the embedding space. The contrastive loss encourages highlighting the alignment of actual pairs of first and second input modalities relative to the alignment of incorrect pairs, thereby creating a separation between them. Contrarian pre-training of the first encoder neural network and the second encoder neural network can significantly improve their downstream performance on tasks where available training data may be insufficient.

[0009] Conventional contrastive pre-training methods are limited by vanishing gradients. For input pairs with high separation, the gradient of the pair decays rapidly, resulting in high-separation pairs contributing little to the optimization. To address this issue, the system described in this paper applies a margin to training pairs with higher separation to rebalance gradient strength. This system uses relative alignment scores as a separation metric and incorporates a linear margin function to adaptively increase the pair's contribution to the optimization during training. Therefore, applying a margin to higher-separation pairs improves training accuracy compared to conventional techniques, as more higher-separation pairs contribute to the optimization. This can improve the performance of downstream tasks, especially in few-shot scenarios.

[0010] The performance of conventional contrastive pre-training methods can be limited by noisy data; for example, web-scale data may include both incorrect and hard examples. When performing contrastive learning, conventional training systems can focus on examples with low separation, which can be beneficial when all training data is accurate. However, when training examples include incorrect data, incorrect pairings can contribute to optimization, while gradients from correct pairings vanish, thus detrimentaling model performance. As described in this specification, using relative margins to adjust the contribution of pairs to optimization allows the system to significantly improve performance on zero-shot applications when using web-scale training examples (e.g., when there are not enough accurately labeled training examples available, or when the training examples include a large amount of noisy data).

[0011] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0012] Figure 1 This is a block diagram of the example system.

[0013] Figure 2 This is a flowchart of an example process for training an encoder neural network.

[0014] Figure 3 An example of the distribution of the input training pairs and their corresponding gradients is shown.

[0015] Figure 4 An example of gradient distributions trained using different intervals is shown.

[0016] Figure 5 Examples of the performance of models trained using different intervals are shown.

[0017] Figure 6 Another example of the performance of a model trained using different intervals is shown.

[0018] In the various figures, the same reference numerals and names indicate the same elements. Detailed Implementation

[0019] Figure 1 Example system 100 is shown. System 100 is an example of a system implemented as a computer program on one or more computers located in one or more locations, in which the systems, components and techniques described below are implemented.

[0020] System 100 is configured to perform downstream tasks. Examples of downstream tasks typically include language modeling, image captioning, visual question answering, open-vocabulary recognition, and cross-modal retrieval. Specifically, system 100 is configured to generate task output 124, which includes one or more second inputs 114 most relevant to the first input 112.

[0021] Generally speaking, the second input 114 can be any kind of data item in various modalities, such as text documents, images, videos, audio signals, or multimodal data items that include two or more modalities, such as text, images, audio, or video.

[0022] Similarly, the first input 112 can be any of the data in any of the various modalities, such as a text query, an image query, a video query, an audio query, or a multimodal query that includes two or more modalities, such as text, images, audio, or video.

[0023] When the downstream task is a language modeling task, the input is a text segment representing the already written text, and the output is a text segment representing the next part of the input text segment.

[0024] When the downstream task is an image captioning task, the input is an image, and the output is a text caption describing the content of the image.

[0025] When the downstream task is a visual question answering task, the input is an image and a text segment describing a question about that image, and the output is a text segment describing the answer to the question.

[0026] When the downstream task is an open vocabulary recognition task, the input is an image, and the output is text describing the coordinates of one or more regions in the image and the corresponding label for each of the one or more regions.

[0027] When the downstream task is a cross-modal retrieval task, the input is a query in one modality (e.g., audio, video, text, etc.), and the output is data items in a different modality consistent with the query. For example, the query could be a text / audio query requesting images / videos with specified content (e.g., images containing a specific object, images containing a specific scene, etc.), and the output could be one or more images containing the specified content and / or related to the specified content.

[0028] Figure 1 The diagram illustrates the use of both a first encoder neural network 106 and a second encoder neural network 108 to perform a downstream task. However, in practice, using two encoder neural networks is not necessary. In some cases, the downstream task can use only the first encoder neural network 106 or only the second encoder neural network 108. Furthermore, the encoder neural network used to perform the downstream task can be further trained before being used for the downstream task.

[0029] System 100 includes a training system 102 and a downstream task execution system 104.

[0030] After training, the downstream task execution system 104 is configured to generate task output 124 in response to the first input 112.

[0031] The downstream task execution system 104 includes a first encoder neural network 106 configured to generate a first embedding 116 by processing a first input 112 and a second encoder neural network 108 configured to generate a second embedding 118 by processing each of the second inputs 114.

[0032] Specifically, system 104 uses second encoder 108 to generate a plurality of second embeddings 118, each second embedding corresponding to a corresponding second input in second input 114. For example, system 104 may use second encoder 108 to generate second embeddings 118 offline after training is complete and before system 104 processes a new first input.

[0033] The second embedding 118 can be an ordered set of numerical values ​​(e.g., a vector or matrix representing floating-point or other numerical values ​​of the target data item 114).

[0034] Each second embedding 118 is typically an embedding within a specific embedding space. An "embedding space" is a space of embeddings with a specified dimension, such as a space of vectors with a specified number of entries.

[0035] The second encoder 108 can be any suitable neural network that can map data items of a specific type to an embedded one. For example, the second encoder 108 can be a Transformer, a convolutional neural network, a visual Transformer, or a recurrent neural network.

[0036] System 104 stores the second embedding 118 in a data structure configured to allow the second embedding 118 to be searched. For example, the data structure could be an index.

[0037] The system can then receive the first input 112. Specifically, the first input 112 can be a new input submitted by the user of the system. For example, the user can submit the first input by entering the first input 112 into the user interface.

[0038] In some examples, the first input can be text input used for general retrieval tasks. For example, the text input could be "Picture of a Fish". The task output could be an image depicting a fish.

[0039] In some other examples, the first input can be an image input for the captioning task. For example, the image input could be a photo of a dog. The task output could be a caption that reads "Dog".

[0040] In some other examples, the first input can be text input for the audio generation task. For example, the text input could be "sad melody in a minor key". The task output could be an audio output of the melody in a minor key.

[0041] In some other examples, the first input can be text input for the video generation task. For example, the text input could be "fish swimming". The task output could be a video output depicting fish swimming.

[0042] In some other examples, the first input can be text input for a text generation task. For example, the text input could be "the history of marbles". The task output could be a text document describing the history of marbles.

[0043] In some other examples, the first input can be a query for a relatively specialized retrieval task targeting a specific relevant output, such as whether a data item is positive or negative, or the length / size of a data item. For example, the query could be "Positive Review of Donuts" or "Long Description of Donuts".

[0044] The system can generate a first embedding 116 by processing the first input 112 using the first encoder 106.

[0045] The first embedding 116 may be an ordered set of numerical values ​​(e.g., a vector or matrix representing the floating-point or other numerical values ​​of the first input 112) that has the same dimension as the second embedding, i.e., is located in the same embedding space as the second embedding.

[0046] The first encoder 106 can be any suitable neural network that can map the first input to the embedded information. For example, the first encoder 106 can be a Transformer, a convolutional neural network, a visual Transformer, or a recurrent neural network.

[0047] As will be described below, the first encoder 106 can be jointly pre-trained with the term encoder 108 (e.g., through contrastive learning).

[0048] Based on the first embedding 116, the system 100 can select one or more second embeddings 118 corresponding to one or more associated second inputs 114.

[0049] Specifically, the system can perform a search to identify one or more second embeddings 118 that are closest to the first embedding 116 based on a similarity metric such as cosine similarity or Euclidean distance. For example, the system can perform a k-nearest neighbor search or an approximate k-nearest neighbor search on the second embeddings 118 to find the second embedding 118 that is closest to the first embedding 116.

[0050] The system can then generate data output 124, including one or more corresponding related second inputs 114 for a specific task. For example, the system can provide task output 124 to be presented to a user or to another system that has submitted the first input. In some examples, generating data output 124 includes inputting the first embedding and / or one or more second embeddings into a further neural network model that processes the first embedding and one or more second embeddings to generate task output 124.

[0051] Before using the encoder neural network to generate task output, the training system 102 trains the encoder neural network through contrastive learning.

[0052] More specifically, the training system uses training pairs 128 from the set of training data 126 to train the neural network.

[0053] Each training pair 128 includes a first training input for the first modality and a second training input for the second modality (i.e., the second input that has been determined to be associated with the first training input).

[0054] Unlike conventional systems, training system 102 modifies the contrastive loss function used by system 102 for training by adjusting the relative intervals of the contributions of training pairs to optimization.

[0055] Conventional contrastive pre-training methods are limited by vanishing gradients. For input training pairs with high separation, the gradient of the pair decays rapidly, resulting in high-separation pairs contributing little to the optimization. This system applies a margin to training pairs with even higher separation to rebalance gradient strength. The system uses relative alignment scores as a separation metric and incorporates a linear margin function to adaptively increase the pair's contribution to optimization during training.

[0056] More specifically, system 102 performs training of the neural network by executing multiple iterations of the training process.

[0057] In each iteration of the training process, system 102 obtains a batch of training pairs 128. For example, system 102 can sample a batch from training data 126.

[0058] Then, system 102 uses the batch training pair 128 to update the parameters of the neural network using the gradient of the contrastive loss function.

[0059] The training process will be described in more detail below.

[0060] Figure 2 This is a flowchart of an example process 200 for training a first encoder neural network and a second encoder neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately configured according to this specification (e.g., Figure 1 The training system 102) can execute process 200.

[0061] The system can repeat process 200 on different batches of training pairs to jointly train two encoder neural networks.

[0062] The system obtains a batch of training pairs, each training pair including a first training input and a second training input (step 202).

[0063] The system processes the first training input in the training pair through the first encoder neural network based on the current value of the first encoder neural network parameters to generate a corresponding first embedding for each training first input (step 204).

[0064] The system processes each training second input in each training pair through the second encoder neural network based on the current value of the second encoder neural network parameters to generate a corresponding second embedding for each training second input (step 206).

[0065] The system determines multiple positive similarity scores (step 208). Generally, each positive similarity score corresponds to one training pair in the training pair and measures the similarity between the first embedding of the first input in the training pair and the second embedding of the second training input in the training pair. In other words, the system can generate a corresponding positive similarity score for each training pair, which measures the similarity between the embedding of the first training input in the training pair and the embedding of the second training input in the training pair. For example, the positive similarity score could be the dot product or cosine similarity between the embeddings of the first training input in the training pair and the embeddings of the second training input in the training pair.

[0066] The system determines multiple negative similarity scores (step 210).

[0067] Each negative similarity score corresponds to a first training input and a second input that is not in the same training pair as the first training input. Each negative similarity score measures the similarity between the first embedding of the corresponding first training input and the second embedding of the corresponding second input.

[0068] For example, a negative similarity score could be the dot product or cosine similarity between the embedding of the first training input and the embedding of the second input.

[0069] The system determines the relative interval for each positive similarity score based on (i) the positive similarity score and (ii) the average of the positive and negative similarity scores (step 212). The system can determine the relative interval by subtracting the average of the positive and negative similarity scores from the positive similarity score.

[0070] As an example, relative intervals can be represented as follows: , Where N represents the number of training pairs. This represents the positive similarity score, and when hour, This indicates the negative similarity score.

[0071] The system determines an adjusted positive similarity score for each positive similarity score based on the positive similarity score and the relative interval between the positive similarity scores (step 214). The system can determine the adjusted positive similarity score by subtracting the output of a monotonically increasing function of the relative interval from the positive similarity score. In some examples, the monotonically increasing function can be a linear function. In some cases, the linear function is (i) the product of a scaling factor and (ii) the relative interval from the positive similarity scores.

[0072] As an example, the adjusted positive similarity score can be represented as follows: , in This represents the adjusted positive similarity score, and (Relative Interval) represents the output of a monotonically increasing function with a relative interval.

[0073] As an example, when the monotonically increasing function is the following linear function: , in This represents the scaling factor.

[0074] The system trains a first encoder neural network on a contrastive loss function applied to (i) the adjusted positive similarity score and (ii) the negative similarity score (step 314).

[0075] In some implementations, the system trains a second encoder neural network on a contrastive loss function applied to (i) adjusted positive similarity scores and (ii) negative similarity scores.

[0076] Generally speaking, contrastive loss functions encourage positive similarity scores to reflect higher similarity compared to negative similarity scores.

[0077] The contrastive loss function can be any contrastive loss function, such as those used by Minderer, Matthias, and others in "Simple open-vocabulary object detection" (…). European Conference on Computer Vision (Cham: Springer Nature Switzerland, 2022), or Chen, Ting et al., in "A simple framework for contrastive learning of visual representations". (International Conference on Machine Learning) (PMLR, 2020), or Khosla, Pranay et al. in “Supervised contrastive learning” ( Advances in Neural Information Processing Systems The contrastive loss functions used in Volume 33 (2020): 18661-18673) are described in this specification. The systems described herein may use any of these contrastive loss functions, modified to use an adjusted positive similarity score instead of the original positive similarity score as the positive version in the loss.

[0078] As an example, the contrastive loss can be represented as follows: .

[0079] This system can train neural networks using the gradient of the contrastive loss. That is, the system can compute the gradient of the contrastive loss relative to the parameters of the first encoder neural network and the second encoder neural network, and then apply an optimizer (e.g., stochastic gradient descent optimizer, Adam optimizer, AdamW optimizer, etc.) to the gradient to update the parameters of the two neural networks.

[0080] Alternatively, in addition to the contrastive loss term as described above, the loss function may also include one or more additional terms, such as a regularization term.

[0081] After training the neural network, the system or another inference system can use the first encoder neural network, the second encoder neural network, or at least a portion of both to perform downstream tasks.

[0082] For example, the system can fine-tune a task neural network for training data used in a downstream task, the task neural network including a first encoder neural network, a second encoder neural network, or a portion thereof. For instance, the task neural network may include a first encoder neural network and one or more additional layers that generate output for the downstream task. As another example, the system can further fine-tune the first encoder, the second encoder, or both before using the neural network to perform the downstream task.

[0083] Figure 3 Example 300 illustrates the distribution of input training pairs and their corresponding gradients. This example shows three low-separation training pairs 302, 304, and 306, and three high-separation training pairs 308, 310, and 312. Each training pair includes an image and an associated description. High-separation training pairs are high-quality training pairs (i.e., correctly trained pairs). Low-separation training pairs are low-quality (i.e., difficult examples) or incorrect training pairs (i.e., irrelevant examples).

[0084] The system can comparatively pre-train a first encoder neural network and a second encoder neural network of a machine learning model to learn representations of a first input modality (e.g., an image) and a second input modality (e.g., text) in a shared embedding space based on a contrastive loss, such that embeddings of semantically similar first and second modalities tend to be closer in the embedding space. The contrastive loss encourages highlighting the alignment of actual pairs of first and second input modalities relative to the alignment of incorrect pairs, thereby creating a separation between them.

[0085] Low-discretion training pairs include both incorrect and difficult examples. For instance, training pair 306 is an incorrect example that includes an image of King Leopold II and the associated description read "this is not King Leopold II." As another example, training pair 304 is a difficult example that includes a group of people riding bicycles near a body of water and the associated description read "with the group near Carlsbad, ca."

[0086] High-separation training pairs include correct and simple examples. For example, training pair 308 includes an image of a chocolate calculator and an associated description read as "chocolate calculator".

[0087] The performance of pre-trained methods can be limited by noisy data; for example, web-scale data may include both incorrect and hard examples. Training the system can focus on examples 302, 304, and 306, which have low separation, potentially beneficial when all training data is accurate and low separation is only needed for hard examples. However, when training examples include incorrect data, incorrect pairings can contribute to optimization, while gradients from correct pairings vanish, thus harming model performance.

[0088] This example includes a graph 316 showing the distribution of the gradients of the contrastive loss function 318. The contrastive pre-training method is limited by gradient vanishing. For input pairs with high separation, the gradient of the pair decays rapidly, resulting in high-separation pairs contributing little to the optimization. This learning focuses on training pairs with separation close to zero. Applying a relative interval to higher-separation training pairs allows the training system to adaptively increase the contribution of training pairs to the optimization during training. This improves training accuracy because more higher-separation pairs can contribute to the optimization.

[0089] Figure 4 Example 400 shows the distribution of gradients trained using a scaling factor. Example 400 shows the distribution trained without scaling factor 402, the distribution trained with scaling factor 0.2 404, and the distribution trained with scaling factor 0.4 406.

[0090] The training system can apply an interval to training pairs with higher separation to rebalance gradient strength during contrastive training. The system can use relative alignment scores as a separation metric and incorporate a linear margin function to adaptively increase the contribution of pairs to optimization during training. The distribution trained with a scaling factor of 0.4 (406) is less skewed (i.e., has a longer left tail) than the distribution trained with a scaling factor of 0.2 (404), which in turn is less skewed than the distribution trained without a scaling factor of 402. This allows training examples with high separation to contribute more to optimization during training.

[0091] Using relative intervals to adjust the contribution of pairs to optimization allows the system to significantly improve performance on zero-shot applications when using web-scale training examples (e.g., when there are not enough accurately labeled training examples available, or when the training examples include a large amount of noisy data).

[0092] Figure 5 Example 500 shows the performance of models trained with different intervals. This example illustrates the performance of the model on a zero-sample image-text retrieval task when trained with various scaling factors (including cases where no interval is used (i.e., the scaling factor is 0)). The best performance for each task emerges when a non-zero scaling factor is used during training, and therefore when a relative interval is applied.

[0093] Figure 6 Another example 600 illustrates the performance of models trained with different intervals. This example demonstrates the performance of the model on a zero-shot image classification task when trained using various scaling factors, including cases where no interval is used (i.e., a scaling factor of 0). The best performance for each task emerges when a non-zero scaling factor is used during training, and therefore when a relative interval is applied.

[0094] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform that operation or action.

[0095] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible storage medium, which may be non-transitory, for execution by a data processing device or to control the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals—e.g., machine-generated electrical, optical, or electromagnetic signals—to generate artificially generated propagation signals to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0096] The term "data processing device" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0097] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0098] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0099] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0100] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to or from them, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0101] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0102] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Similarly, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0103] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0104] Machine learning models can be implemented and deployed using machine learning frameworks (such as TensorFlow or Jax frameworks).

[0105] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0106] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by computer programs executed on respective computers that establish a client-server relationship between them. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0107] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from that claimed combination may be removed, and the claimed combination may involve sub-combinations or variations thereof.

[0108] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0109] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method executed by one or more computers for training a first encoder neural network and a second encoder neural network, the first encoder neural network having first encoder neural network parameters and configured to process a first input to generate a first embedding of the first input in an embedding space, the second encoder neural network having second encoder neural network parameters and configured to process a second input to generate a second embedding of the second input in the embedding space, the method comprising: Obtain a batch of training pairs, each training pair including the first input of the first modality and the second input of the second modality; Based on the current values ​​of the parameters of the first encoder neural network, each first input in each training pair is processed by the first encoder neural network to generate a corresponding first embedding for each first input; Each second input is processed by the second encoder neural network based on the current value of the second encoder neural network parameters to generate a corresponding second embedding for each second input; Multiple positive similarity scores are determined, each positive similarity score corresponding to one of the training pairs, and the similarity between the first embedding in the training pair and the second embedding in the training pair is measured; Multiple negative similarity scores are determined, each negative similarity score corresponds to a corresponding first training first input and a corresponding second input that is not in the same training pair as the corresponding first training first input, and the similarity between the first embedding of the corresponding first training input and the second embedding of the corresponding second input is measured; For each positive similarity score, a relative interval is determined based on (i) the positive similarity score and (ii) the average of the positive similarity score and the negative similarity score; For each positive similarity score, an adjusted positive similarity score is determined based on that positive similarity score and the relative interval for that positive similarity score; as well as The first encoder neural network is trained on the contrastive loss function applied to (i) the adjusted positive similarity score and (ii) the negative similarity score.

2. The method of claim 1, further comprising: The second encoder neural network is trained on the contrastive loss function applied to (i) the adjusted positive similarity score and (ii) the negative similarity score.

3. The method of any of the preceding claims, wherein the first encoder neural network is an image encoder neural network, and the first input of the first modality is an input image.

4. The method of any of the preceding claims, wherein the second encoder neural network is a text encoder neural network, and the second input of the second modality is an input text segment.

5. The method of any of the preceding claims, wherein the first encoder neural network and the second encoder neural network are jointly trained.

6. The method of any of the preceding claims, wherein the positive similarity score and the negative similarity score measure cosine similarity.

7. The method of any of the preceding claims, wherein determining the relative interval for each positive similarity score based on (i) the positive similarity score and (ii) the average of the positive similarity score and the negative similarity score comprises: Subtract the average of the positive and negative similarity scores from the positive similarity score.

8. The method of any preceding claim, wherein determining an adjusted positive similarity score for each positive similarity score based on the positive similarity score and a relative interval for the positive similarity score comprises: Subtract the output of a monotonically increasing function of the relative interval from the positive similarity score from the positive similarity score.

9. The method of claim 8, wherein the monotonically increasing function is a linear function.

10. The method of claim 9, wherein the linear function is the product of (i) the scaling factor and (ii) the relative interval.

11. The method of any of the preceding claims, wherein the method further comprises: Downstream tasks are performed using the first encoder neural network, the second encoder neural network, or at least a portion of both.

12. The method of claim 11, wherein the method further comprises: The task neural network is fine-tuned on training data used for the downstream task, the task neural network including at least a portion of the first encoder neural network, the second encoder neural network, or both.

13. A computer-implemented method, comprising: The first input is processed using a first encoder neural network to generate a first embedding, wherein the first input includes input data of a first modality; One or more second embeddings among a plurality of second embeddings are identified based on a similarity metric to the first embedding, each second embedding corresponding to a corresponding second input among a plurality of second inputs, wherein each second embedding is generated from the corresponding second input using a second embedding neural network, and wherein the second input includes input data of a second modality; and The task output is generated at least in part based on the one or more second embeddings. The first encoder neural network and the second encoder neural network have been trained using the method described in any of the preceding claims.

14. A system comprising: One or more computers; as well as One or more storage devices communicatively coupled to one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of a corresponding method as described in any one of claims 1 to 13.

15. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method as claimed in any one of claims 1 to 13.

Images