Learning unpaired multi-modal feature matching for semi-supervised learning

Abstract

A computer-implemented method for learning multimodal feature matching is provided. The method includes training an image encoder to obtain an encoded image. The method also includes training a common classifier on the encoded image using labeled images. The method further includes training a text encoder using learned text embeddings and their corresponding labels while keeping the common classifier in a fixed configuration. The text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image.

Description

Background Technology

[0001] This invention generally relates to machine learning, and more specifically to learning unpaired multimodal feature matching for semi-supervised learning.

[0002] Generating one data modality from another is an important function in many machine learning applications.

[0003] Typically, applications involve two or more data modalities, where for each modality there are a few labeled samples and many unlabeled samples. The goal is to use the labeled samples to learn a common mapping between the modalities.

[0004] In “Text to Image Generative Model using Constrained Embedding Space Mapping”, IEEE International Workshop On Machine Learning For Signal Processing, 2017, by Subhajit Chaudhury et al. and “Conditional generation of multi-modal data using constrained embedding space mapping”, International Conference on Machine Learning (ICML) workshop on Implicit Generative Models, 2017, by Subhajit Chaudhury et al., deterministic mapping schemes are used, where they are first computed to force the latent space to be equal during training. Because the latent space is deterministic, deterministic mapping schemes cannot model different variations in multimodal distributions. Furthermore, deterministic mapping schemes are prone to overfitting because they do not provide a measure of maximum decoupling between the embeddings and the common latent space. Additionally, they use paired training data.

[0005] In “Multimodal deep learning,” in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 689–696, by Jiquan Ngiam et al., a deep learning framework is proposed that uses a restricted Boltzmann machine and a deep belief network to learn effective features of audio and video modalities. However, they require two modalities to infer the latent space, which restricts the conditional generation of data from one modality to another.

[0006] In “Generating images from captions with attention,” ComputingResearch Repository (CoRR), Vol. abs / 1511.02793, 2015, by Elman Mansimov et al., it is shown that using attention-based models to generate images from text captions results in higher quality samples. However, they cannot produce bidirectional multimodal data distributions.

[0007] In “Generative adversarial text to image synthesis,” in Proceedings of the 33rd International Conference on Machine Learning, Vol. 48. 2016, ICML'16, pp. 1060–1069, JMLR.org, by Scott Reed et al., deep convolutional generative adversarial networks (GANs) were proposed that combine natural language and image embeddings to produce synthetically generated images. However, they are only capable of generating text from images, and not the other way around.

[0008] In the workshop "Joint Multimodal Learning with Deep Generative Models," International Conference on Learning Representations (ICLR) 2017, April 24–26, 2017, Toulon, France, by Masahiro Suzuki et al., joint distributed learning was proposed. This approach creates a common latent space by sharing data modalities, directly applying variational inference to the data modalities. However, their method cannot be directly used for conditionally independent inference. Furthermore, their method requires more network parameters, uses more data for training, and must rely on adversarial models used to train natural images. Summary of the Invention

[0009] According to various aspects of the present invention, a computer-implemented method for learning multimodal feature matching is provided. The method includes training an image encoder to obtain an encoded image. The method further includes training a common classifier on the encoded image using labeled images. The method also includes training a text encoder using learned text embeddings and their corresponding labels while keeping the common classifier in a fixed configuration. The text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image. The matching of the distance of the predicted text embeddings to the fitted Gaussian distribution on the encoded image forces unlabeled images to have soft clustering scores for each category, thereby utilizing a small number of labeled images, which leads to improved multimodal matching performance using a large amount of data.

[0010] In this embodiment, the text encoder is trained to simultaneously optimize the cross-entropy of the common classifier and the KL divergence between the fitted Gaussian distribution in the image domain and the predicted text embedding. In this way, the distributions of the latent representations in both the image and text embeddings can match the same distributions in a category-based manner that enables cross-modal generation and classification.

[0011] In this embodiment, a common classifier is trained without paired data. In this way, a classifier trained on image samples can be used to distinguish text embedding samples using a fixed classifier. This allows for the implicit distribution of text embeddings in a latent space without using paired image and text samples where applications are found in many real-world use cases.

[0012] In this embodiment, cross-entropy loss is used to train a common classifier. In this way, the common classifier can learn common text and image latent space representations that distinguish similar classification tasks, automatically aligning the embeddings of the two modalities into the same distribution in a categorical manner.

[0013] According to other aspects of the invention, a computer program product for learning multimodal feature matching is provided. The computer program product includes a non-transient computer-readable storage medium having program instructions embodied therein. The program instructions are executable by a computer to cause the computer to perform a method. The method includes training an image encoder to obtain an encoded image. The method further includes training a common classifier on the encoded image using labeled images. The method further includes training a text encoder using learned text embeddings and corresponding labels of the learned text embeddings, while keeping the common classifier in a fixed configuration. The text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image.

[0014] According to other aspects of the invention, a computer processing system for learning multimodal feature matching is provided. The computer processing system includes a memory device for storing program code. The computer processing system also includes a processor device operatively coupled to the memory device for running the program code to train an image encoder to obtain an encoded image. The processor device further runs program code to train a common classifier on the encoded image using labeled images. The processor device further runs program code to train a text encoder by using learned text embeddings and corresponding labels for the learned text embeddings while keeping the common classifier in a fixed configuration. The text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image.

[0015] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments of the invention, which is read in conjunction with the accompanying drawings. Attached Figure Description

[0016] The following description will provide details of preferred embodiments with reference to the following figures, in which:

[0017] Figure 1 This is a block diagram illustrating an exemplary computing device according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart illustrating an exemplary method for learning unpaired multimodal feature matching for improved semi-supervised learning according to an embodiment of the present invention;

[0019] Figure 3This is a flowchart illustrating another exemplary method for learning unpaired multimodal feature matching for improved semi-supervised learning according to an embodiment of the present invention;

[0020] Figure 4 This is a block diagram illustrating an exemplary scenario involving an image generator utilizing triplet loss according to an embodiment of the present invention;

[0021] Figure 5 The illustration shows an embodiment of the present invention. Figure 3 A block diagram of a portion of the method's block;

[0022] Figure 6 The illustration shows an embodiment of the present invention. Figure 3 The block diagram of another part of the method's block;

[0023] Figure 7 The illustration shows an embodiment of the present invention. Figure 3 The block diagram of another part of the method;

[0024] Figure 8 This is a high-level block diagram illustrating an exemplary learning process of a general text encoder with a common classifier according to an embodiment of the present invention.

[0025] Figure 9 This is a block diagram illustrating an illustrative cloud computing environment with one or more cloud computing nodes according to an embodiment of the present invention, wherein a local computing device used by a cloud consumer communicates with the cloud computing nodes; and

[0026] Figure 10 This is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment according to an embodiment of the present invention. Detailed Implementation

[0027] Embodiments of the present invention relate to learning unpaired multimodal feature matching for semi-supervised learning.

[0028] Therefore, current methods for multimodal feature matching suffer from numerous shortcomings as described above.

[0029] Advantageously, embodiments of the invention overcome the aforementioned drawbacks by initially using unpaired image and text samples. The invention uses only a small number of labeled samples from each category in both modalities and learns multimodal features by using a common classifier and distribution learning based on KL divergence. Therefore, even based on unpaired samples, the invention proposes learning robust shared embeddings between the two modalities using less training data, enabling cross-modal tasks such as generation and captioning.

[0030] Figure 1This is a block diagram illustrating an exemplary computing device 100 according to an embodiment of the present invention. The computing device 100 is configured to learn unpaired multimodal feature matching for semi-supervised learning.

[0031] The computing device 100 can be embodied as any type of computing or computer device capable of performing the functions described herein, including but not limited to computers, servers, rack-based servers, blade servers, workstations, desktop computers, laptop computers, notebook computers, tablet computers, mobile computing devices, wearable computing devices, network devices, web devices, distributed computing systems, processor-based systems, and / or consumer electronics devices. Additionally or alternatively, the computing device 100 can be implemented as one or more computer rails, memory rails, or other racks, rails, computer shelves, or other components of a physically separate computing device. Figure 1 As shown, computing device 100 illustratively includes processor 110, input / output subsystem 120, memory 130, data storage device 140, and communication subsystem 150, and / or other components and devices common in servers or similar computing devices. Of course, in other embodiments, computing device 100 may include other or additional components, such as those typically found in server computers (e.g., various input / output devices). Additionally, in some embodiments, one or more illustrative components may be incorporated into another component or otherwise formed part of another component. For example, in some embodiments, memory 130 or a portion thereof may be incorporated into processor 110.

[0032] Processor 110 can be implemented as any type of processor capable of performing the functions described herein. Processor 110 can be implemented as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single-core or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0033] Memory 130 can be implemented as any type of volatile or non-volatile memory or data storage device capable of performing the functions described herein. In operation, memory 130 can store various data and software used during the operation of computing device 100, such as operating systems, applications, programs, libraries, and drivers. Memory 130 is communicatively coupled to processor 110 via I / O subsystem 120, which can be embodied as circuitry and / or components to facilitate input / output operations with processor 110, memory 130, and other components of computing device 100. For example, I / O subsystem 120 can be implemented as or otherwise include a memory controller hub, input / output control hub, platform controller hub, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, optical fibers, printed circuit board traces, etc.) and / or other components and subsystems to facilitate input / output operations. In some embodiments, the I / O subsystem 120 may form part of a system-on-a-chip (SOC) and be integrated onto a single integrated circuit chip along with the processor 110, memory 130, and other components of the computing device 100.

[0034] Data storage device 140 can be embodied as one or more devices of any type configured for short-term or long-term data storage, such as memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. Data storage device 140 can store program code for learning unpaired multimodal feature matching for semi-supervised learning. Communication subsystem 150 of computing device 100 can be embodied as any network interface controller or other communication circuitry, device, or combination thereof capable of enabling communication between computing device 100 and other remote devices via a network. Communication subsystem 150 can be configured to use any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to achieve such communication.

[0035] As shown in the figure, the computing device 100 may also include one or more peripheral devices 160. Peripheral devices 160 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 160 may include a display, touchscreen, graphics circuitry, keyboard, mouse, speaker system, microphone, network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0036] Of course, the computing device 100 may also include other elements (not shown) that are readily apparent to those skilled in the art, and some elements may be omitted. For example, as will be readily understood by those skilled in the art, various other input and / or output devices may be included depending on the specific implementation of the computing device 100. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memories, etc., may be utilized in various configurations. Additionally, in another embodiment, a cloud configuration may be used (e.g., see...). Figure 9-10 Given the teachings of the invention provided herein, these and other variations of the processing system 100 will readily occur to those skilled in the art.

[0037] As used herein, the terms "hardware processor subsystem" or "hardware processor" can refer to a processor, memory (including RAM, cache(s), etc.), software (including memory management software), or a combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuitry, processing circuitry, instruction execution devices, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories that may be on-board or off-board, or may be dedicated to use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0038] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0039] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0040] According to embodiments of the present invention, these and other variations of the hardware processor subsystem are also contemplated.

[0041] The following Figure 2 and 3 The method of the invention is described at different levels of detail. In particular, Figure 2Method 200 generally describes embodiments of the present invention, while Figure 3 This is a further detailed description of embodiments of the present invention. Method 200 can be considered as a general embodiment of the present invention, while method 300 can be considered as a more specific embodiment of the present invention.

[0042] Figure 2 This is a flowchart illustrating an exemplary method 200 for learning unpaired multimodal feature matches for improved semi-supervised learning according to an embodiment of the present invention.

[0043] In box 205, the image encoder is trained to obtain the encoded image.

[0044] In box 210, a common classifier is trained on the encoded image using labeled images. In one embodiment, the common classifier can be trained without paired data; that is, a classifier trained on image samples can be used as a discriminator on text samples, with fixed classifier weights to implicitly align the latent representations of two modalities with similar distributions. In one embodiment, cross-entropy loss is used to train the common classifier.

[0045] In box 215, while keeping the common classifier in a fixed configuration, a text encoder is trained using the learned text embeddings and their corresponding labels. The text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image. In one embodiment, random adjectives may be added to each of the corresponding labels used for variations in the text distribution.

[0046] At box 220, the output is the trained encoded image, the trained common classifier, and the trained text encoder.

[0047] Figure 3 This is a flowchart illustrating another exemplary method 300 for learning unpaired multimodal feature matching for improved semi-supervised learning according to an embodiment of the present invention.

[0048] In box 305, the input image is received.

[0049] In box 310, a learned image generator is trained using triplet loss in the latent image space. The triplet loss uses labeled images: Image generators implemented by autoencoders or GANs use labeled images. .

[0050] In block 315, a multivariate Gaussian fit is applied to the labeled embedding. .

[0051] In box 320, receive the input text.

[0052] In box 325, pre-trained text embeddings are extracted by applying a pre-trained text embedding model to the input text. That is, the input text is transformed into a representative numerical vector using a pre-trained text embedding model.

[0053] In box 330, the pre-trained text embeddings are mapped to the common space by the text encoder. The term "common space" refers to the space where image and text samples coexist.

[0054] In box 335, use labeled samples. and Learning a common classifier. In an embodiment, block 335 (i.e., learning the common classifier) ​​may involve loss0.

[0055] In block 340, use the sample. and Minimize the Kullback Liebler (KL) divergence between the fitted Gaussian and the mapped word embeddings. In one embodiment, box 340 (i.e., minimizing the KL divergence) may involve a loss1. .

[0056] In box 345, semi-supervised learning is performed on the common embedding space. The total loss of the semi-supervised learning is calculated as follows:

[0057]

[0058] Learning is performed through the following steps: Block 310 (Image Generator + Triple Loss) -> Block 335 (Common Classifier) ​​-> Block 330 (Text Encoder).

[0059] In box 350, an input image is received for captioning with an output text description, or an input text string is received for retrieving the corresponding output image.

[0060] In box 355, control of a hardware object (e.g., a vehicle such as a car) is initiated in response to an output caption or image. For example, control could be initiated in response to a label exported for an image indicating an impending collision, such as steering, braking, or acceleration of the vehicle.

[0061] In one embodiment, the method is performed by a text captioning system that adds captions to the input image using an output text description. Other exemplary applications to which this invention can be applied include, but are not limited to, adding captions to images based on small datasets or image generation, which can be applied to commercial systems for image retrieval, recommendation systems, etc.

[0062] Figure 4This is a block diagram illustrating an exemplary scenario 400 involving an image generator utilizing triplet loss according to an embodiment of the present invention.

[0063] The triplet loss groups similar images together and separates different images. Note that there is essentially equal spacing on the left side of the two 5s and 8s, while on the right side, the 5s are more closely spaced than the 8s.

[0064] Figure 5 The illustration shows an embodiment of the present invention. Figure 3 A block diagram of part of box 335 of method 300.

[0065] This corresponds to the image encoder training step after the input image passes through encoder 510 and decoder (common classifier) ​​520, where decoder 520 implements a triplet loss that "push" similar categories together and "push" different categories apart.

[0066] Once the image encoder 510 were trained, classifier The 520 is then trained on the encoded image output to obtain classification in the image encoding space 515. Classifier model The following cross-entropy loss method can be used for training:

[0067]

[0068] Figure 6 The illustration shows an embodiment of the present invention. Figure 3 The other part of the block diagram of method 300 is shown in box 335.

[0069] This is the text encoder training step, where a classifier that classifies various types of categories maps the input text to clusters after it has been converted into text embeddings.

[0070] Note that the decoder (common classifier) ​​520 is fixed, similar to a GAN architecture. Only the text encoder 610 is trained using cross-entropy (CE) loss, while the decoder 520 is not trained (it is fixed).

[0071] if It is the input character. It is an embedding function (similar to word2vec or GloVe). If the classifier (decoder) is frozen (520), then the encoder model is trained using the following CE loss. :

[0072]

[0073] Figure 7The illustration is based on an embodiment of the present invention. Figure 3 The block diagram of method 300 is another part of box 335.

[0074] The text encoder 510 is trained to correctly classify the learned text embeddings using an image classifier (fixed weights) 520, and also uses the following to match the distance of the predicted text embeddings to the fitted Gaussian distribution 730:

[0075]

[0076] The foregoing content involves three sets of terms. The first set of terms ( This corresponds to the minimum Gaussian distribution. The second term set ( ) corresponds to the cross-entropy loss of a fixed classifier. The third set ( This corresponds to the fitting of a multivariate Gaussian loss. Two losses are optimized simultaneously (for the second and third term sets).

[0077] Figure 8 This is a high-level block diagram illustrating an exemplary learning 800 of a general text encoder with a common classifier according to an embodiment of the present invention.

[0078] exist Figure 8 In this process, image encoder 510 learns latent representations (or embeddings) for image samples as part of a common latent space 805 using triplet loss on small labeled samples. Next, a common classifier 520 is trained on the image embeddings to classify samples for each class. Subsequently, the image embeddings for each class are fitted in a class-specific multivariate Gaussian distribution 730. Finally, variational text encoder 610 is trained on a frozen-weight version of the common classifier using cross-loss, and additionally trained using KL divergence loss to fit the class-specific multivariate Gaussian distribution.

[0079] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0080] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0081] The characteristics are as follows:

[0082] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0083] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0084] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0085] Rapid Flexibility: In some cases, the ability to scale outwards and inwards quickly and flexibly can be provided. For consumers, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0086] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the providers and consumers of the services being utilized.

[0087] The service model is as follows:

[0088] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0089] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0090] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0091] The deployment model is as follows:

[0092] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party and can exist inside or outside a building.

[0093] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0094] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0095] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursting for load balancing between clouds).

[0096] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0097] Now for reference Figure 9 The figure depicts an illustrative cloud computing environment 950. As shown, the cloud computing environment 950 includes one or more cloud computing nodes 910 to which local computing devices used by cloud consumers can communicate, such as personal digital assistants (PDAs) or cellular phones 954A, desktop computers 954B, laptop computers 954C, and / or automotive computer systems 954N. Nodes 910 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 950 to provide cloud consumers with infrastructure, platforms, and / or software-as-a-service that eliminates the need for them to maintain resources on their local computing devices. It should be understood that... Figure 9The types of computing devices 954A-N shown are for illustrative purposes only, and computing node 910 and cloud computing environment 950 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0098] Now for reference Figure 10 This demonstrates the 950 (cloud computing environment) Figure 9 This provides a set of functional abstractions. It should be understood beforehand that... Figure 10 The components, layers, and functions shown are for illustrative purposes only, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0099] The hardware and software layer 1060 includes hardware and software components. Examples of hardware components include: a host 1061; a server 1062 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1063; a blade server 1064; a storage device 1065; and a network and networking component 1066. In some embodiments, the software components include network application server software 1067 and database software 1068.

[0100] The virtualization layer 1070 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1071; virtual storage 1072; virtual network 1073, including virtual private network; virtual application and operating system 1074; and virtual client 1075.

[0101] In one example, management layer 1080 may provide the functionality described below: Resource Provisioning 1081 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 1082 provides cost tracking when utilizing resources in the cloud computing environment, as well as billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 1083 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 1084 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 1085 provides pre-scheduling and procurement of cloud resources, where future needs are anticipated according to the SLA.

[0102] Workload layer 1090 provides examples of functionalities that can leverage cloud computing environments. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1091; software development and lifecycle management 1092; virtual classroom education delivery 1093; data analysis and processing 1094; transaction processing 1095; and learning unpaired multimodal feature matching for semi-supervised learning 1096.

[0103] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0104] Computer-readable storage media can be tangible devices capable of retaining and storing instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0105] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0106] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as SMALLTALK, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from the computer-readable program instructions.

[0107] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0111] References to "one embodiment" or "embodiment" and other variations of the invention in this specification mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout the specification, as well as any other variations, do not necessarily refer to the same embodiment.

[0112] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of” is intended to cover the selection of only the first listed option (A), or only the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to many of the listed items, as will be apparent to those skilled in the art and related fields.

[0113] Preferred embodiments of the systems and methods have been described (these are intended to be illustrative and not limiting), and it is noted that modifications and variations can be made by those skilled in the art based on the foregoing teachings. Therefore, it should be understood that changes can be made to the specific embodiments disclosed, and these changes are within the scope of the invention as summarized by the appended claims. Thus, aspects of the invention have been described in the details and features required by patent law, and the claimed and patentable contents are set forth in the appended claims.

Claims

1. A computer-implemented method for learning multimodal feature matching, comprising: An image encoder is trained using a triplet loss that pushes apart similar image sets and dissimilar images to obtain encoded images; Text embeddings with corresponding labels are learned by training a common classifier on coded images using labeled images; and A text encoder is trained while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image.

2. The computer-implemented method of claim 1 further includes training the common classifier by using the labeled image with the image encoder and the labeled text with the text encoder.

3. The computer-implemented method of claim 1, wherein the text encoder is trained to simultaneously optimize the cross-entropy of the common classifier and the KL divergence between the fitted Gaussian distribution and the predicted text embedding.

4. The computer-implemented method according to claim 1, wherein, The common classifier was trained without paired data.

5. The computer-implemented method according to claim 1, wherein, The common classifier is trained using cross-entropy loss.

6. The computer-implemented method according to claim 1, wherein, The total loss is calculated as the sum of the loss corresponding to the common classifier and the result of multiplying the hyperparameters by the loss corresponding to the image encoder.

7. The computer-implemented method of claim 1 further comprises minimizing the Kullback-Liebler divergence between the fitted Gaussian distribution and the learned text embedding using the corresponding labels.

8. The computer-implemented method of claim 7 further includes performing semi-supervised learning in a common embedding space.

9. The computer-implemented method according to claim 1, wherein, The text encoder maps pre-trained text embeddings together with the image embeddings to a common latent representation to enable cross-modal tasks.

10. The computer-implemented method of claim 1, further comprising extracting the text embedding by means of a pre-trained text embedding model applied to the training text.

11. The computer-implemented method of claim 1, wherein the method is performed by a text captioning system that adds captions to an input image using an output text description.

12. The computer-implemented method of claim 11, further comprising controlling the vehicle to avoid a collision in response to at least one indication of an impending collision in the output text description.

13. The computer-implemented method according to claim 1, wherein, The triplet loss pushes similar encoded images together and separates dissimilar encoded images.

14. The computer-implemented method of claim 1, wherein training the text encoder further comprises using the common classifier to map the learned text embeddings to sample clusters to classify the learned text embeddings into corresponding categories among a plurality of categories.

15. The computer-implemented method of claim 1, wherein the text encoder is trained such that the cross-entropy loss and multivariate Gaussian loss of the classifier in the fixed configuration are simultaneously optimized.

16. The computer-implemented method of claim 1, further comprising adding random adjectives to each of the corresponding labels for text distribution variation.

17. A computer program product for learning multimodal feature matching, the computer program product comprising a non-transient computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a computer to cause the computer to perform a method, the method comprising: An image encoder is trained using a triplet loss that pushes apart similar image sets and dissimilar images to obtain encoded images; Text embeddings with corresponding labels are learned by training a common classifier on coded images using labeled images; and A text encoder is trained while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image.

18. The computer program product of claim 17, wherein the method further comprises training the common classifier by using the labeled image with the image encoder and using the labeled text with the text encoder.

19. The computer program product of claim 17, wherein the text encoder is trained to simultaneously optimize the cross-entropy of the common classifier and the KL divergence between the fitted Gaussian distribution and the predicted text embedding.

20. The computer program product according to claim 17, wherein, The common classifier was trained without paired data.

21. The computer program product according to claim 17, wherein, The common classifier is trained using cross-entropy loss.

22. The computer program product according to claim 17, wherein, The total loss is calculated as the sum of the loss corresponding to the common classifier and the result of multiplying the hyperparameters by the loss corresponding to the image encoder.

23. The computer program product of claim 17, wherein the method further comprises minimizing the Kullback-Liebler divergence between the fitted Gaussian distribution and the learned text embedding using the corresponding labels.

24. The computer program product of claim 17, wherein, The method is performed by a text captioning system that adds captions to the input image using output text descriptions.

25. A computer processing system for learning multimodal feature matching, the system comprising: Memory devices used to store program code; as well as A processor device, operatively coupled to the memory device, is used to run the program code to train an image encoder to obtain an encoded image by using a triplet loss that pushes apart sets of similar images and dissimilar images. Text embeddings with corresponding labels are learned by training a common classifier on encoded images using labeled images; and A text encoder is trained while keeping the common classifier in a fixed configuration by using the learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match the distance of the predicted text embeddings encoded by the text encoder to a fitted Gaussian distribution on the encoded image.

Images