Self-supervised learning using model extension

Abstract

A method for providing a neural network system includes performing contrastive learning on a neural network to generate a trained neural network system. Performing contrastive learning includes performing a first model augmentation on a first encoder of the neural network system to generate a first embedding of samples, performing a second model augmentation on the first encoder to generate a second embedding of samples, and optimizing the first encoder using a contrast loss based on the first embedding and the second embedding. The trained neural network system is provided to perform a task.

Description

[Technical Field]

[0001] This application claims priority to U.S. Non-Provisional Patent Application No. 17 / 579,377, filed January 19, 2022, U.S. Non-Provisional Patent Application No. 63 / 252,375, filed October 5, 2021, and U.S. Provisional Patent Application No. 63 / 230,474, filed August 6, 2021, which are incorporated by reference herein in their entireties.

[0002] The present disclosure relates generally to neural networks, and more particularly to contrastive self-supervised learning (SSL) using machine learning systems and model extension. [Background technology]

[0003] Sequential recommendation in machine learning aims to predict future items in a sequence, and one of the key parts is characterizing the item relationships in the sequence. Traditional sequence modeling in machine learning is sometimes used to validate the superiority of transforms, such as self-attention mechanisms, in uncovering item correlations in sequences. For example, a transformer can be used to infer the embedded sequence at a given position by a weighted aggregation of item embeddings, where the weights are learned via self-attention.

[0004] However, the problems of data sparsity and noise in sequences impair the performance of neural network models (also called models) in sequential recommendation. The former hinders performance due to insufficient training, as the complex structure of sequential models requires a dense corpus to be properly trained. The latter also hinders the model's recommendation ability, as noisy item sequences cannot reveal actual item correlations.

[0005] Therefore, it would be advantageous to develop systems and methods for improved sequential recommendations. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a simplified diagram of a computing device according to some embodiments described herein.

[0007] [Figure 2] FIG. 1 is a simplified diagram of a method for performing contrastive learning using model augmentation, according to some embodiments described herein.

[0008] [Figure 3] FIG. 1 is a simplified diagram illustrating an exemplary control self-supervised learning system according to some embodiments described herein.

[0009] [Figure 4] FIG. 1 is a simplified diagram of a method for performing model expansion for contrastive learning, according to some embodiments described herein.

[0010] [Figure 5A] FIG. 1 is a simplified diagram of an example neuron masking module for implementing model expansion using neuron masking, according to some embodiments described herein.

[0011] [Figure 5B] 1 illustrates another exemplary contrastive learning system with model expansion using neuron masking, according to some embodiments described herein.

[0012] [Figure 6A] 1 illustrates an example layer dropping module for implementing model expansion using layer dropping, according to some embodiments described herein.

[0013] [Figure 6B]10 illustrates another exemplary contrastive learning model with model augmentation using layer dropping, according to some embodiments described herein.

[0014] [Figure 7] 1 illustrates another exemplary contrastive learning system with model augmentation using neuron masking and layer dropping, according to some embodiments described herein.

[0015] [Figure 8] 1 illustrates an example encoder completion module for implementing model extension using encoder completion, according to some embodiments described herein.

[0016] [Figure 9] FIG. 1 is a simplified diagram of a computing device that implements contrastive learning with model expansion according to some embodiments described herein.

[0017] In the figures, elements with the same number have the same or similar function. DETAILED DESCRIPTION OF THE INVENTION

[0018] As used herein, the term "network" may include any artificial intelligence network or system, neural network or system, and / or any hardware or software-based framework including any training or learning model implemented therein or with it.

[0019] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0020] 1 is a simplified diagram of a computing device 100, according to some embodiments. As shown in FIG. 1, computing device 100 includes a processor 110 coupled to memory 120. The operation of computing device 100 is controlled by processor 110. Also, while computing device 100 is shown with only one processor 110, it is understood that processor 110 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits, graphics processing units (GPUs), etc. within computing device 100. Computing device 100 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0021] Memory 120 may be used to store software executed by computing device 100 and / or one or more data structures used during operation of computing device 100. Memory 120 may include one or more types of machine-readable media. Some common forms of machine-readable media may include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or computer.

[0022] Processor 110 and / or memory 120 may be located in any suitable physical location. In some embodiments, processor 110 and / or memory 120 may be implemented on the same board, the same package (e.g., a system-in-package), the same chip (e.g., a system-on-chip), etc. In some embodiments, processor 110 and / or memory 120 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, processor 110 and / or memory 120 may be located in one or more data centers and / or cloud computing facilities.

[0023] As shown, memory 120 includes neural network model 130, which may be used to implement and / or emulate neural network systems and models described further herein and / or to implement any of the methods described further herein. In some examples, neural network module 130 may be used to translate structured text. In some examples, neural network module 130 may also handle the iterative training and / or evaluation of a translation system or model used to translate structured text. In some examples, memory 120 may include a non-transitory, tangible, machine-readable medium including executable code that, when operated by one or more processors (e.g., processor 110), may cause the one or more processors to perform contrastive learning using the augmentation methods described in further detail herein. In some examples, neural network module 130 may be implemented using hardware, software, and / or a combination of hardware and software. As shown, computing device 100 receives input 140 provided to neural network module 130, which then generates output 150.

[0024] As described above, sequential recommendation aims to predict the next item in a user's behavior, which can be solved by characterizing item relationships in a sequence. To address the issues of data sparsity and noise in sequences, a self-supervised learning (SSL) paradigm may be used to improve performance, which uses contrastive learning between positive and negative views of a sequence. While various methods may construct views by employing perspective augmentation of data, such data augmentation has various problems. For example, it may be difficult to devise an optimal data augmentation method. Furthermore, data augmentation methods may destroy sequence correlations. Furthermore, such data augmentation may not incorporate comprehensive self-supervised signals. To address these issues, a system and method for contrastive SSL using model augmentation is described below.

[0025] 2 and 3, FIG. 2 is a simplified diagram of a method 200 for performing contrastive SSL using model extension, and FIG. 3 is an exemplary neural network system 300 for performing method 200. One or more of the processes of method 200 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, can cause the one or more processors to perform one or more of the processes of method 200. In some embodiments, method 200 may correspond to a method for determining a neural network model to be used by neural network module 130 to perform training and / or inference using neural network models for various tasks. In the examples described below, a sequential recommendation task is used, but method 200 may be used for a variety of tasks, including, for example, sequential recommendation, image recognition, and language modeling.

[0026] In various embodiments, method 200 implements model augmentation for constructing view pairs for contrastive learning, e.g., as a complement to data augmentation methods. Furthermore, both single-level and multi-level model augmentation methods for constructing view pairs are described. In one example, a multi-level model augmentation method may include various multi-level model augmentation methods, such as a neuron masking method, a layer dropping method, and an encoder completion method. In another example, a single-level model augmentation method may include a single-level model augmentation, and the type of model augmentation may be determined based on the specific task. By using model augmentation, method 200 improves performance (e.g., for sequential recommendation or other tasks) by constructing views for contrastive SSL using the model augmentation.

[0027] Method 200 begins at block 201, where control training is performed on a neural network model using one or more batches of training data, where each batch may include one or more original samples. In the following description, an exemplary neural network model for sequential recommendation is used, where the original samples are also referred to as original sequences. Blocks 202 through 212 may be performed for each original sequence in the training batch.

[0028] In block 202, an original sequence from training data is provided. Referring to the example of FIG. 3, the sequence s u An original sequence 302, denoted as , is received, where the users and set of items are denoted as U and V, respectively. Each user u receives a sequence of items s in time order. u =[v1,…,v t ,…,v |su| ], where v t Let denote the items that user u has interacted with at time t, and |su| is the total number of items. The sequential recommendation problem may be formulated as follows:

number

[0029] The method 200 may proceed to block 204, where a first data augmentation is performed on the original sequence to generate a first augmented sequence. In some examples, the first data augmentation is optional. Various data augmentation techniques may be used, including, for example, truncation, masking, reordering, insertion, substitution, and / or combinations thereof. Referring to the example of FIG. 3, performing the first data augmentation on the original sequence 302 generates an augmented sequence 304. The first augmented sequence 304 is provided to an input of an encoder 306.

[0030] The method 200 may proceed to block 206, where a first model augmentation is performed on an encoder (e.g., encoder 306 of FIG. 3) to augment the encoder and generate a first embedding (e.g., embedding 310) of a first augmented sequence (e.g., augmented sequence 304 of FIG. 3). As shown in the example of FIG. 3, the model augmentation module 307 performs the model augmentation on the encoder to provide an augmented encoder that generates an output embedding. In some embodiments, concatenation (e.g., by concatenation module 308) may be performed on the output of the augmented encoder to generate the embedding 310.

[0031] The method 200 may proceed to block 208, where a second data extension is performed on the original sequence to generate a second extended sequence. In some examples, the second data extension is optional. In some examples, the second data extension is different from the first data extension, and the second extended sequence is different from the first extended sequence. Referring to the example of FIG. 3, performing the second data extension on the original sequence 302 generates an extended sequence 312. The second extended sequence 312 is provided to an input of the encoder 306.

[0032] Method 200 may proceed to block 210, where a second model augmentation is performed on an encoder (e.g., encoder 306 of FIG. 3) to generate a second embedding (e.g., embedding 316 of FIG. 3) of a second augmented sequence (e.g., augmented sequence 312 of FIG. 3). As shown in the example of FIG. 3, model augmentation module 307 performs model augmentation on encoder 306 to provide an augmented encoder that generates an output embedding. In some embodiments, concatenation (e.g., by concatenation module 308) may be performed on the output of the augmented encoder to generate embedding 316.

[0033] The method 200 may proceed to block 212, where an optimization process is performed on the encoder (e.g., encoder 306 of FIG. 3) using a contrastive loss based on the first embedding and the second embedding (e.g., embeddings 310 and 312 of FIG. 3). An example contrastive loss is provided as follows:

number

number

number

number

[0034] In various embodiments, for sequential recommendations, both SSL and next item prediction characterize item relationships within a sequence and may be combined to generate a final loss L to optimize the encoder. An example contrastive loss is provided as follows:

number

number

number

number

[0035] In block 216, the trained neural network generated by the contrastive learning in block 201 may be used to perform a task, such as a sequence recommendation task or any other suitable task. For example, the trained neural network may be used to generate next item predictions for an input sequence.

[0036] An exemplary model expansion method (e.g., for implementing blocks 206 and 210 of FIG. 2) is described with reference to FIGS. 4, 5A, 5B, 6A, 6B, 7, and 8. FIG. 4 illustrates an exemplary model expansion method 400. FIG. 5A shows an exemplary neuron masking module (also referred to as a neuron dropout module or dropout module) for implementing model expansion using neuron masking. FIG. 5B shows an exemplary contrastive training system including a neuron masking module for model expansion. FIG. 6A shows an exemplary layer dropping module for implementing model expansion using layer dropping. FIG. 6B shows an exemplary contrastive training system including a layer dropping module for model expansion. FIG. 7 shows an exemplary contrastive training system including a model expansion module with both neuron masking and layer dropping. FIG. 8 shows an exemplary encoder completion module for implementing model expansion using encoder completion.

[0037] Referring to FIG. 4, an exemplary model expansion method 400 is shown, which may use different model expansion methods at one or more levels. The method 400 may proceed to block 402, where neuron masking is performed. Referring to FIG. 5A, an exemplary neuron masking module 500 of an encoder (e.g., encoder 306 of FIG. 3) is shown. The neuron masking module 500 may receive hidden embeddings 502 (also called input embeddings) and generate output embeddings 504 for the next layer via a feed-forward network (FFN) 506. In some embodiments, during training, the neuron mask may randomly mask partial neurons in each FFN layer 506 based on their respective masking probabilities p. A larger value of p results in stronger embedding perturbations. Thus, by applying different masking probabilities within the encoder 306 (e.g., with or without data augmentation, data augmentation sequences 304 and 312 of FIG. 3), a pair of different views (e.g., embeddings 310 and 316 of FIG. 3) is generated from one and the same original sequence from the model's perspective. In some embodiments, during each batch of training, masked neurons are randomly selected, providing comprehensive contrastive learning for model extension. In some embodiments, different probability values ​​may be utilized for different FFN layers of the encoder. In some embodiments, the neuron masking probability is the same for different FFN layers of the encoder. Additionally or alternatively, the neuron masking method 400 may be applied to any neural layer in a model of a neural network system to inject more perturbations.

[0038] 5B, an exemplary contrastive training system 550 includes a model expansion module that includes a neuron masking module 500. The contrastive training system 550 is substantially similar to the contrastive training system 300 of FIG. 3, except for the differences described below. In the example of FIG. 5B, during training, the neuron masking module 500 performs neuron masking twice by randomly masking partial neurons in one or more layers of the original encoder 306 to generate two different views of the same original sequence 302. Contrastive training is performed using the two different views.

[0039] In block 404, layer dropping is performed. In various embodiments, dropping partial layers of a neural network model (e.g., encoder 306 in FIG. 3 ) reduces the depth and complexity of the neural network model. In some embodiments, only shallow embeddings for users and items may be required. In some embodiments, embeddings in shallow and deep layers are both important for reflecting comprehensive information of the data. Therefore, randomly dropping some layers during training may serve as a method of normalization. Model expansion using layer dropping may enable contrastive learning between embeddings with different layer depths. In one example, contrastive learning is achieved between shallow and deep embeddings, thereby providing enhancement to models that contrast only between deep features, for example.

[0040] In some embodiments, the layers within the original encoder are dropped. In some of these embodiments, dropping layers, particularly the necessary layers within the original encoder, may disrupt the original sequential correlation, and the views generated by dropping layers may not be positive pairs. Alternatively, in some embodiments, instead of operating on the original encoder, K FFN layers are stacked after the encoder, and M FFN layers are dropped during each training batch, where M and K are integers and M < K. In these embodiments, the layers of the original encoder are not dropped during layer dropping.

[0041] Referring to FIG. 6A, an exemplary layer dropping module 600 for implementing layer dropping is shown. The layer dropping module 600 may receive an embedding 602 (also referred to as an embedding) from an original encoder (e.g., encoder 306 of FIG. 3) and generate an output embedding 604. In various embodiments, the layer dropping module 600 may append K layers 606-1 to 606-K after the encoder 306 and randomly drop M layers (e.g., layer 606-2) during each training batch, where M may be an integer from 0 to K-1. In some embodiments, the same K appended FFN layers and M dropped FFN layers are applied to the encoder 306 for separate views. In other embodiments, different K's and M's are applied to the encoder 306 for separate views.

[0042] 6B , an exemplary contrastive training system 650 includes a model expansion module that includes a layer dropping module 600. The layer dropping module 600 may perform model expansion by appending multiple layers (e.g., multi-layer perceptron (MLP) layers, self-attention layers, residual layers, other suitable layers, and / or combinations thereof) to the sequence encoder 306. In the example of FIG. 6B , during training, the layer dropping module 600 performs layer dropping twice by appending one or more layers to the original encoder 306 and randomly dropping the appended layers to generate two different views of the same original sequence 302. Contrastive training is performed using the two different views.

[0043] Referring to FIG. 7, an exemplary contrastive training system 700 includes a model expansion module 702 that performs both neuron masking and layer dropping. The contrastive training system 700 is substantially similar to the contrastive training system 300 of FIG. 3, except for the differences described below. The model expansion module 702 may perform layer dropping by appending several layers (e.g., MLP layers or other suitable layers) to the sequence encoder 306 and randomly dropping M layers from a total of K append layers during training. Additionally, the model expansion module 702 may perform neuron masking to randomly mask partial neurons in one or more layers in the original encoder 306 and / or the append layers.

[0044] The method 400 may proceed to block 406, where neuron masking is performed.

[0045] In various embodiments, a single encoder may be used to generate embeddings for two views of a sequence during self-supervised learning. In some embodiments, using a single encoder may be effective in revealing complex sequential correlations, but contrasting a single encoder may result in embedding collapse issues for self-supervised learning. Furthermore, a single encoder may reflect item relationships only from a single perspective. For example, a transformer encoder employs attentative aggregation of item embeddings to infer sequence embeddings, while an RNN structure is more suitable for encoding direct item transitions. Therefore, in some embodiments, separate encoders may be used to generate views for contrastive learning, which may enable the model to learn comprehensive sequential relationships of items. However, in some embodiments, embeddings from two views of a sequence using different encoders may lead to a non-Siamese paradigm for self-supervised learning, which may be difficult to train and suffer from embedding collapse issues. Additionally, in instances where two separate encoders reveal significantly diverse sequential correlations, the embeddings may be very far from each other, resulting in poor views for contrastive learning. Furthermore, in some embodiments, two separate encoders may be optimized during the training phase, but combining them for inferring sequence embeddings to make recommendations can still be problematic.

[0046] The encoder completion methods described herein may address issues arising from using a single encoder or two separate encoders to generate views for contrastive training. In various embodiments, instead of contrastive training using a single encoder or two separate encoders, encoder completion uses a pre-trained encoder to complement model extension for the original encoder. Referring to FIG. 8 , an example encoder completion module 800 for performing encoder completion is shown. In the pre-training phase, an encoder 806 different from the encoder 306 is pre-trained on a next item prediction target to generate a pre-trained encoder 808. Then, during the contrastive self-supervised training phase, this pre-trained encoder 808 is utilized to generate another embedding 810 for the view. A combiner 812 combines the view embeddings 814 generated from the model encoder 306 with the view embeddings 810 from the pre-trained encoder 808. In some embodiments, this model extension is in one branch of the SSL paradigm. The embeddings 810 from the pre-trained encoder 306 may be rescaled by a hyperparameter γ (also referred to as a weight γ) before combining with (e.g., adding to) the embeddings 814 from the model encoder 808. A smaller value of the hyperparameter γ corresponds to fewer perturbations injected from the separate encoder 806. The output embeddings of the combiner 912 may be passed to the next layer via a feed-forward network (FFN) 812. By applying different weights γ to the outputs from the pre-trained encoder 808, two different views of the same sequence are provided for contrastive learning.

[0047] In some embodiments, the parameters of this pre-trained encoder 808 are fixed during the control self-supervised training. In these embodiments, there is no optimization for this pre-trained encoder 808 during the control self-supervised training. Furthermore, during the inference phase, it is no longer necessary to consider both encoders 306 and 808; only the model encoder 306 is used.

[0048] Referring to FIG. 9 , an exemplary computing device 900 that may be used to implement contrastive learning with model extension according to some embodiments described herein is illustrated. As shown in FIG. 9 , the computing device 900 includes a processor 910 coupled to a memory 920. The operation of the computing device 900 is controlled by the processor 910. Also, while the computing device 900 is shown with only one processor 910, it is understood that the processor 910 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits, graphics processing units (GPUs), etc. within the computing device 900. The computing device 900 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0049] Memory 920 may be used to store software executed by computing device 900 and / or one or more data structures used during operation of computing device 900. Memory 920 may include one or more types of machine-readable media. Some common forms of machine-readable media may include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or computer.

[0050] The processor 910 and / or memory 920 may be located in any suitable physical location. In some embodiments, the processor 910 and / or memory 920 may be implemented on the same board, the same package (e.g., a system-in-package), the same chip (e.g., a system-on-chip), etc. In some embodiments, the processor 910 and / or memory 920 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 910 and / or memory 920 may be located in one or more data centers and / or cloud computing facilities.

[0051] In some examples, memory 920 may include a non-transitory, tangible, machine-readable medium containing executable code that, when operated by one or more processors (e.g., processor 910), may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory 920 includes instructions for a neural network module 930 (e.g., neural network module 130 of FIG. 1 ), which may be used to implement and / or emulate systems and models and / or implement any of the methods described further herein. In some examples, neural network module 930 implements contrastive learning with model extension and may also be referred to as a contrastive learning with model extension module. The contrastive learning with model extension module 930 may receive input 940, such as, for example, an original sequence, via data interface 915. Data interface 915 may be either a user interface that receives input sequences uploaded by a user or a communications interface that may receive or retrieve previously stored sequences from a database. The contrastive learning with model extension module 930 may generate an output 950 in response to an input 940, such as a prediction of the next item for the input sequence.

[0052] In some embodiments, the contrastive learning with model augmentation module 930 may further include an encoder module 931 for providing an encoder, a neuron masking module 932 for performing neuron masking, a layer dropping module 933 for performing layer dropping, and an encoder completion module 934 for performing encoder completion.

[0053] Some examples of computing devices, such as computing devices 100, 400, may include non-transitory, tangible, machine-readable media that includes executable code that, when executed by one or more processors (e.g., processor 110), can cause the one or more processors to perform the processes of method 200. Some common forms of machine-readable media that can include the processes of the methods / systems described herein (e.g., the methods / systems of FIGS. 2-8) are, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0054] This description and the accompanying drawings, which illustrate aspects, embodiments, implementations, or applications of the invention, should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Like numbers in two or more figures represent the same or similar elements.

[0055] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous details are set forth to provide a thorough understanding of the embodiments. It will be apparent to one of ordinary skill in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative, but not limiting. Those skilled in the art may recognize other elements not specifically described herein that are within the scope and spirit of the present disclosure. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless otherwise specifically described or unless one or more features render the embodiment non-functional.

[0056] While exemplary embodiments have been shown and described, a wide range of modifications, variations, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention is to be limited only by the claims that follow, and it is appropriate that such claims be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A method for providing a neural network system, comprising: performing contrastive learning on the neural network system to generate a trained neural network system, wherein performing the contrastive learning includes: performing a first model augmentation on a first encoder of the neural network system to generate a first embedding of the samples; performing a second model extension on the first encoder to generate a second embedding of the samples; optimizing the first encoder using a contrast loss based on the first embedding and the second embedding; providing the trained neural network system to perform a task; Performing the first model augmentation includes: performing layer dropping by dropping one or more layers associated with the first encoder; performing the layer dropping step appending a plurality of append layers to the first encoder; and randomly dropping one or more of the plurality of append layers.

2. Performing the first model augmentation includes: The method of claim 1 , further comprising: performing neuron masking by randomly masking one or more neurons associated with the first encoder.

3. performing the neuronal masking The method of claim 2 , comprising randomly masking the one or more neurons in one or more layers associated with the first encoder based on a masking probability.

4. The method of claim 3 , wherein the same masking probability is applied to each layer.

5. The method of claim 3 , wherein different masking probabilities are applied to different layers.

6. The method of claim 2 , wherein the neuron masking is performed on an original layer of the first encoder or one of the append layers.

7. A method for providing a neural network system, comprising: performing contrastive learning on the neural network system to generate a trained neural network system, wherein performing the contrastive learning includes: performing a first model augmentation on a first encoder of the neural network system to generate a first embedding of the samples; performing a second model extension on the first encoder to generate a second embedding of the samples; optimizing the first encoder using a contrast loss based on the first embedding and the second embedding; providing the trained neural network system to perform a task; Performing the first model augmentation includes: performing encoder completion using a second encoder; performing the encoder interpolation pre-training the second encoder to provide a pre-trained encoder; providing, by the first encoder, a first intermediate embedding of the samples; providing, by the pre-trained encoder, a second intermediate embedding of the samples; and combining the first intermediate embedding and a weighted second intermediate embedding to generate the first embedding for contrastive training.

8. The method of claim 7 , wherein the first encoder and the second encoder are of different types.

9. The method of claim 8 , wherein the first encoder is a transformer-based encoder and the second encoder is a recurrent neural network (RNN)-based encoder.

10. 1. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions, the plurality of machine-readable instructions adapted, when executed by one or more processors, to cause the one or more processors to perform a method, the method comprising: performing contrastive learning on the neural network system to generate a trained neural network system, wherein performing the contrastive learning includes: performing a first model augmentation on a first encoder of the neural network system to generate a first embedding of the samples; performing a second model extension on the first encoder to generate a second embedding of the samples; optimizing the first encoder using a contrast loss based on the first embedding and the second embedding; providing the trained neural network system to perform a task; Performing the first model augmentation includes: performing layer dropping by dropping one or more layers associated with the first encoder; performing the layer dropping step appending a plurality of append layers to the first encoder; and randomly dropping one or more of the plurality of append layers.

11. Performing the first model augmentation includes:

11. The non-transitory machine-readable medium of claim 10, further comprising performing neuron masking by randomly masking one or more neurons associated with the first encoder.

12. performing the neuronal masking 12. The non-transitory machine-readable medium of claim 11, comprising randomly masking the one or more neurons of one or more layers associated with the first encoder based on a masking probability.

13. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions adapted, when executed by one or more processors, to cause the one or more processors to perform a method, the method comprising: performing contrastive learning on the neural network system to generate a trained neural network system, wherein performing the contrastive learning includes: performing a first model augmentation on a first encoder of the neural network system to generate a first embedding of the samples; performing a second model extension on the first encoder to generate a second embedding of the samples; optimizing the first encoder using a contrast loss based on the first embedding and the second embedding; providing the trained neural network system to perform a task; Performing the first model augmentation includes: performing encoder completion using a second encoder; performing the encoder interpolation pre-training the second encoder to provide a pre-trained encoder; providing, by the first encoder, a first intermediate embedding of the samples; providing, by the pre-trained encoder, a second intermediate embedding of the samples; and combining the first intermediate embedding and a weighted second intermediate embedding to generate the first embedding for contrastive training.

14. 1. A system comprising: a non-transient memory; one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method, the method comprising: performing contrastive learning on the neural network system to generate a trained neural network system, wherein performing the contrastive learning includes: performing a first model augmentation on a first encoder of the neural network system to generate a first embedding of the samples; performing a second model extension on the first encoder to generate a second embedding of the samples; optimizing the first encoder using a contrast loss based on the first embedding and the second embedding; providing the trained neural network system to perform a task; Performing the first model augmentation includes: performing layer dropping by dropping one or more layers associated with the first encoder; performing the layer dropping step appending a plurality of append layers to the first encoder; and randomly dropping one or more of the plurality of append layers.

15. Performing the first model augmentation includes: The system of claim 14 , further comprising: performing neuron masking by randomly masking one or more neurons associated with the first encoder.

16. performing the neuronal masking 16. The system of claim 15, comprising randomly masking the one or more neurons of one or more layers associated with the first encoder based on a masking probability.

17. A system comprising: a non-transient memory; one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method, the method comprising: performing contrastive learning on the neural network system to generate a trained neural network system, wherein performing the contrastive learning includes: performing a first model augmentation on a first encoder of the neural network system to generate a first embedding of the samples; performing a second model extension on the first encoder to generate a second embedding of the samples; optimizing the first encoder using a contrast loss based on the first embedding and the second embedding; providing the trained neural network system to perform a task; Performing the first model augmentation includes: performing encoder completion using a second encoder; performing the encoder interpolation pre-training the second encoder to provide a pre-trained encoder; providing, by the first encoder, a first intermediate embedding of the samples; providing, by the pre-trained encoder, a second intermediate embedding of the samples; combining the first intermediate embedding and a weighted second intermediate embedding to generate the first embedding for contrastive training.

Images