Friend Training: Method, Apparatus, and Computer Program for Learning from Models of Different but Related Tasks

Abstract

A method, apparatus, and non-transitory storage medium for training two or more cross-task neural network models based on two or more neural network tasks, the method, apparatus, and non-transitory storage medium including: mapping a first pseudo label based on a first model associated with a first task of the two or more neural network tasks and a second pseudo label based on a second model associated with a second task of the two or more neural network tasks into a same space; and calculating a matching score indicative of a cross-task matching between the first pseudo label and the second pseudo label based on the mapping. The method may further include selecting one or more cross-task pseudo labels based on the matching score and an accuracy rate associated with the first model and the second model; and training the two or more cross-task neural network models based on the one or more cross-task pseudo labels.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 17 / 941,417, filed September 9, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE DISCLOSURE

[0001] Embodiments of the present disclosure relate to language processing using artificial intelligence and machine learning. More specifically, embodiments of the present disclosure relate to training for different but related tasks. [Background technology]

[0003]

[0002] Many different machine learning algorithms, such as self-supervised learning, semi-supervised learning, and weakly supervised learning, aim to effectively use unlabeled data for training. As unlabeled data becomes increasingly available, the above supervised learning algorithms are attracting even greater interest. Self-training is a semi-supervised learning mechanism that aims to improve model performance by utilizing pseudo-labels in unlabeled data, and has been successfully applied in fields such as computer vision and natural language processing.

[0004] However, current self-training algorithms are primarily focused on a single task and often a single data set, and because they are applied in isolation, they do not take advantage of properties of inputs that are shared between related tasks.

[0005]

[0004] Therefore, there is a need for new methods, algorithms, systems, and devices for supervised learning that enable learning from different task types and analyze specific characteristics of inputs that are shared between related tasks. Summary of the Invention

[0006] According to an embodiment, there may be provided a method for training two or more cross-task neural network models based on two or more neural network tasks. The method may include: mapping a first pseudo-label based on a first model associated with a first task of the two or more neural network tasks and a second pseudo-label based on a second model associated with a second task of the two or more neural network tasks into the same space; calculating a matching score indicating a cross-task matching between the first pseudo-label and the second pseudo-label based on the mapping; selecting one or more cross-task pseudo-labels based on the matching score and accuracy rates associated with the first model and the second model; and training the two or more cross-task neural network models based on the one or more cross-task pseudo-labels.

[0007] According to an embodiment, there may be provided an apparatus for training two or more cross-task neural network models based on two or more neural network tasks, the apparatus may include at least one memory configured to store program code, and at least one processor configured to read the program code and operate according to instructions in the program code. The program code may include: mapping code configured to cause at least one processor to map a first pseudo label based on a first model associated with a first task of the two or more neural network tasks and a second pseudo label based on a second model associated with a second task of the two or more neural network tasks into the same space; computing code configured to cause the at least one processor to calculate a matching score indicative of a cross-task matching between the first pseudo label and the second pseudo label based on the mapping; selection code configured to cause the at least one processor to select one or more cross-task pseudo labels based on the matching score and accuracy rates associated with the first model and the second model; and training code configured to cause the at least one processor to train two or more cross-task neural network models based on the one or more cross-task pseudo labels.

[0008] According to an embodiment, a non-transitory computer-readable medium may be provided having instructions stored thereon that, when executed by at least one processor for training two or more cross-task neural network models based on two or more neural network tasks, may cause the at least one processor to map a first pseudo-label based on a first model associated with a first of the two or more neural network tasks and a second pseudo-label based on a second model associated with a second of the two or more neural network tasks into the same space, calculate a matching score indicative of a cross-task matching between the first pseudo-label and the second pseudo-label based on the mapping, select one or more cross-task pseudo-labels based on the matching score and accuracy rates associated with the first model and the second model, and train the two or more cross-task neural network models based on the one or more cross-task pseudo-labels. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram of an environment in which the methods, apparatus, and systems described herein may be implemented, according to an embodiment. [Figure 2] FIG. 2 is a block diagram of example components of one or more devices of FIG. 1. [Figure 3] FIG. 1 illustrates an exemplary method for training two or more cross-task neural network models based on two or more neural network tasks, according to an embodiment. [Figure 4] 1 shows a flowchart for training two or more cross-task neural network models based on two or more neural network tasks, according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010]

[0012] Embodiments of the present disclosure relate to methods, apparatus, and systems for training two or more cross-task neural network models based on two or more neural network tasks.

[0011]

[0013] As mentioned above, many different machine learning algorithms, such as self-supervised learning, semi-supervised learning, and weakly supervised learning, aim to effectively use unlabeled data for training. As unlabeled data becomes increasingly available, the supervised learning algorithms mentioned above are attracting even greater interest. Self-training is a semi-supervised learning mechanism that aims to improve model performance by utilizing pseudo-labels in unlabeled data and has been successfully applied to fields such as computer vision and natural language processing. However, current self-training algorithms primarily focus on a single task and often a single dataset. Because they are applied in isolation, current self-training algorithms do not exploit input characteristics shared across related tasks.

[0012]

[0014] Therefore, there is a need for new methods, algorithms, systems, and apparatus for supervised learning that enable learning from different task types and analyze specific characteristics of inputs that are shared between related tasks.Embodiments of the present disclosure leverage learning from different task types using a novel cross-task self-training framework, "friend training."

[0013]

[0015] Shared characteristics may include specific span boundaries in dependency parsing and phrase structure parsing, as well as some categories in sentiment analysis and emotion detection. As an example, two dialogue understanding tasks, conventional semantic role assignment (CSRL) and dialogue rewriting (DR), may have shared characteristics such as coreference and zero pronoun parsing. Supervision from such friend tasks (different but related tasks) can serve as a good benchmark for assessing the quality of pseudo-labels. Therefore, embodiments of the present disclosure focus on how to leverage cross-task supervision from related tasks in self-training.

[0014]

[0016] Compared with self-training in the related art, the friend training disclosed herein can leverage supervision from friend tasks for better pseudo-label selection. To use such supervision, this disclosure proposes two novel modules to incorporate supervision signals from friend tasks. The first module can include a translation matcher that can map pseudo-labels from different tasks for one instance into the same space and calculate a matching score that represents the degree of cross-task matching between models. The second module can include an extension selector that can leverage both the confidence and matching score of pseudo-labels from task-specific models to select high-quality pseudo-labels.

[0015]

[0017] In related technology, self-training aims to iteratively improve a model for a single task by using both labeled data and a large amount of unlabeled data. In each iteration, the self-training model can first assign pseudo-labels to the unlabeled data. Then, a set of unlabeled instances with pseudo-labels can be selected for training, along with information for better model generalization. The model can then be updated by minimizing the cross-entropy between the model's predictions and the labels on both the gold-labeled data and the pseudo-labeled data: TIFF0007821907000001.tif16170

[0016]

[0018] where the first term (i.e., the left term) is the loss for labeled data, the second term (i.e., the right term) is the loss for unlabeled data, λ is a coefficient to balance the first and second terms, N (N') is the number of instances, y (y') is the label, and p (p') can be the output probability of the model.

[0017]

[0019] In friend training with two tasks (two tasks are just examples, friend training can be extended to multiple models more than two), two classifiers f a and f b Let η be the expected accuracy rate. a and η b We can train on two different tasks using a labeled training set with the following two datasets: The two datasets can be created independently, and the prediction targets for the two tasks can be partially related through a pair of translation functions: TIFF0007821907000002.tif14170

[0018]

[0020] where Σ is the number of tasks All possible predictions for TIFF0007821907000003.tif6170 The set of possible sub-predictions can be reduced to TIFF0007821907000004.tif6170. The translation function is such that the expected confusion probability is It can be a general function, TIFF0007821907000005.tif11170. The translation function is deterministic and can always map a task's gold labels to the same translation.

[0019]

[0021] Both classifiers can make predictions on the unlabeled set U at iteration k. TIFF0007821907000006.tif7170 are selected as new training data based on the predictions from both classifiers translated by the translation function and some selection criterion, such as exact match. If exact match is used as the selection criterion, then in these instances, f a The probability of making an incorrect prediction is: TIFF0007821907000007.tif19170

[0020]

[0022] Since both classifiers can be very different due to different training data, models, prediction targets, etc., the two classifiers are very likely to be independent of each other. Under this condition, equation (4) becomes: TIFF0007821907000008.tif21170

[0021]

[0023] where: The file is TIFF0007821907000009.tif6170.

[0022]

[0024] The above formula shows that the quality of the selected instance is affected by the noisy translation. TIFF0007821907000010.tif6170 and the number of matching negative instances, E. By selecting a translation function with a sufficiently large codomain, Σ, When TIFF0007821907000011.tif5170 is minimized, the probability of an error instance being selected when two classifiers agree approaches 0. This also means that 1-η a is large, that is, f a This shows that even if the performance of is poor, the error rate of selected instances can be kept very low if the co-domain is large. As the dependency between two classifiers increases in training, the probability of an error instance also increases. If they are completely dependent on each other, Equation 4 can be expressed as 1-η a That is, conventional self-training.

[0023]

[0025] To reduce and keep the error rate low, two additional modules may be required: the first module can include a translation matcher that maps predictions from two models trained on different but related tasks into the same space and calculates a matching score; the second module can include an extension (instance) selector that considers both the matching score of the translated predictions and the model confidence to select instances with pseudo-labels for the classifier.

[0024]

[0026] Translation matcher - Given a model of two friend tasks, we find a pseudo-label {v} from the model. a ,v b} may be received. The translation matcher M can calculate a matching score m for a pair of pseudo labels, which represents the similarity of the pair in the translation space, with 1 being a perfect match. TIFF0007821907000012.tif8170

[0025]

[0027] The matching score can serve as a selection criterion for high-quality pseudo-labels using cross-task supervision.

[0026]

[0028] Extended selector - A pseudo label can be selected not only based on the similarity of the pseudo labels (i.e., matching score), but also based on other sources of information about the quality of the pseudo labels (e.g., model confidence metrics) to extend the matching score. The extended selector uses the confidence (also called accuracy) of the pseudo labels from a task-specific model (c a ,c b}) and matching scores can be considered. TIFF0007821907000013.tif8170

[0027]

[0029] where: TIFF0007821907000014.tif6170 can represent the results of pseudo-label selection. Therefore, instances with low matching scores but high confidence may also be selected as training data.

[0028]

[0030] An exemplary algorithm for friend training is disclosed below. TIFF0007821907000015.tif92170 Algorithm 1 - Friend training cross-task neural network model

[0029]

[0031] FIG. 1 is a diagram of an environment 100 in which the methods, apparatus, and systems described herein may be implemented, according to an embodiment.

[0030]

[0032] 1, environment 100 may include user devices 110, a platform 120, and a network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0031]

[0033] User device 110 includes one or more devices that can receive, generate, store, process, and / or provide information related to platform 120. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some implementations, user device 110 can receive information from and / or transmit information to platform 120.

[0032]

[0034] Platform 120 includes one or more devices as described elsewhere herein. In some implementations, platform 120 may include a cloud server or a collection of cloud servers. In some implementations, platform 120 may be designed to be modular so that software components can be swapped out. Thus, platform 120 may be easily and / or quickly reconfigured for different uses.

[0033]

[0035] In some implementations, as shown, platform 120 may be hosted in cloud computing environment 122. It should be noted that although the implementations described herein describe platform 120 as being hosted in cloud computing environment 122, in some implementations platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0034]

[0036] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 can provide services such as computation, software, data access, storage, etc. that do not require end-user (e.g., user device 110) knowledge of the physical location and configuration of the system(s) and / or device(s) that host platform 120. As shown, cloud computing environment 122 can include a collection of computing resources 124 (collectively referred to as “computing resources 124” and individually referred to as “computing resource 124”).

[0035]

[0037] Computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 124 may host platform 120. Cloud resources may include compute instances running on computing resources 124, storage devices provided on computing resources 124, data transfer devices provided by computing resources 124, etc. In some implementations, computing resources 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0036]

[0038] As further shown in FIG. 1, computing resources 124 include cloud resources such as one or more applications (“APPs”) 124-1, one or more virtual machines (“VMs”) 124-2, virtualized storage (“VSs”) 124-3, and one or more hypervisors (“HYPs”) 124-4.

[0037]

[0039] Application 124-1 includes one or more software applications that may be provided to or accessed by user device 110 and / or platform 120. Application 124-1 may eliminate the need to install and run software applications on user device 110. For example, application 124-1 may include software associated with platform 120 and / or any other software that may be provided via cloud computing environment 122. In some implementations, one application 124-1 may send information to or receive information from one or more other applications 124-1 via virtual machine 124-2.

[0038]

[0040] The virtual machine 124-2 comprises a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. The virtual machine 124-2 may be a system virtual machine or a process virtual machine, depending on the application and the degree to which the virtual machine 124-2 corresponds to an actual machine. A system virtual machine can provide a complete system platform that supports the execution of a complete operating system (“OS”). A process virtual machine can execute a single program and support a single process. In some implementations, the virtual machine 124-2 can run on behalf of a user (e.g., the user device 110) and manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-term data transfer.

[0039]

[0041] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization techniques within the storage systems or devices of computing resource 124. In some embodiments, within the context of a storage system, types of virtualization can include block virtualization and file virtualization. Block virtualization can refer to the abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed regardless of the physical storage or heterogeneous structure. This separation allows storage system administrators flexibility in how they manage storage for end users. File virtualization can remove the dependency between data accessed at the file level and where the file is physically stored. This can enable performance optimization of storage utilization, server consolidation, and / or nondisruptive file migration.

[0040]

[0042] The hypervisor 124-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer, such as the computing resource 124. The hypervisor 124-4 may present a virtual operating platform to the guest operating systems and manage the execution of the guest operating systems. Multiple instances of different operating systems may share virtualized hardware resources.

[0041]

[0043] Network 130 may include one or more wired and / or wireless networks. For example, network 130 may include a cellular network (e.g., a fifth-generation (5G) network, a long-term evolution (LTE) network, a third-generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, an optical fiber-based network, etc., and / or a combination of these or other types of networks.

[0042]

[0044] The number and arrangement of devices and networks shown in Figure 1 are provided as an example. In practice, there may be additional, fewer, different, or differently arranged devices and / or networks than those shown in Figure 1. Furthermore, two or more of the devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) in environment 100 may perform one or more functions described as being performed by another set of devices in environment 100.

[0043]

[0045] FIG. 2 is a block diagram of example components of one or more devices of FIG.

[0044]

[0046] The device 200 may correspond to the user device 110 and / or the platform 120. As shown in FIG. 2, the device 200 may include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270.

[0045]

[0047] Bus 210 includes components that enable communication between the components of device 200. Processor 220 is implemented in hardware, firmware, or a combination of hardware and software. Processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, processor 220 includes one or more processors that are programmable to perform certain functions. Memory 230 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions used by processor 220.

[0046]

[0048] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive.

[0047]

[0049] Input components 250 include components that enable device 200 to receive information, such as via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0048]

[0050] Communications interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 270 may enable device 200 to receive information from and / or provide information to other devices. For example, communications interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0049]

[0051] Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to processor 220 executing software instructions stored by a non-transitory computer-readable medium, such as memory 230 and / or storage component 240. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spanning multiple physical storage devices.

[0050]

[0052] Software instructions can be loaded into memory 230 and / or storage component 240 from other computer-readable media or from other devices via communications interface 270. When executed, the software instructions stored in memory 230 and / or storage component 240 can cause processor 220 to perform one or more of the processes described herein. Additionally, or alternatively, hardwired circuitry can be used in place of or in combination with software instructions to perform one or more of the processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0051]

[0053] The number and arrangement of components shown in Figure 2 are provided as an example. In practice, device 200 may include additional, fewer, different, or differently arranged components than those shown in Figure 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0052]

[0054] 3 is a diagram of an example process for training two or more cross-task neural network models based on two or more neural network tasks. As shown in FIG. 3, process 300 is an example process for friend training a CSRL model and a DR model.

[0053]

[0055] Using the examples of CSRL and DR as candidate tasks, embodiments can be described in detail. Experimental results of Friend training, unsupervised domain adaptation, and few-shot learning show that Friend training significantly outperforms both classical and state-of-the-art semi-supervised learning algorithms. It should be understood that the use of CSRL and DR as candidate tasks is merely exemplary. Other relevant tasks may be used instead, as appropriate.

[0054]

[0056] While both CSRL and DR utilize skills such as coreference and zero-pronoun analysis, the two tasks focus on different characteristics of dialogue utterances. CSRL can extract arguments of predicates within an utterance from the entire dialogue history. DR can rewrite the final turn of a dialogue into a context-free, fluent one by restoring all ellipsis and coreferences within the utterance.

[0055]

[0057] The unlabeled data 305 consists of N temporally ordered utterances {u1,...,u N}. t (e.g., u1310-1, u2310-2, etc.) and u t K predicates {pred1,…,pred K Given u, the CSRL parser 320 can predict spans from the dialogue as arguments to all predicates. t , the context {u1,...,u t-1}, we can rewrite it to be context-free. t-1} and the current utterance u t is a sequence of tokens {x1,…,x M} and encoded with BERT (or any suitable model) to get the contextualized embedding: TIFF0007821907000016.tif8170

[0056]

[0058] We use the same notation E for the output from the CSRL and DR encoders, even though the encoders do not share parameters.

[0057]

[0059] CSRL: Using contextualized embeddings, predicate-aware TIFF0007821907000017.tif6170 is generated by applying self-attention to E with predicate-aware masking, where the token can only attend to tokens within the same utterance and tokens from utterances containing the predicate. The predicate-aware representation can then be projected by a feedforward network to obtain a distribution of labels for each token. The labels can follow an appropriate labeling scheme, for example the BIO sequential labeling scheme, where BX and IX indicate that the token is the first token and an internal token of argument X, respectively, and where O means that the token does not belong to any argument. The output of the CSRL parser for K predicates is a set of {A1,…,A K}(or, for example, (pred k ,A K )325-k), where Set A k pred k can contain arguments.

[0058]

[0060] DR: DR can be considered as a sequence labeling task. Specifically, a binary classifier on E can be used to classify utterances u t It can decide whether to keep each token in the rewritten utterance. TIFF0007821907000018.tif9170

[0059]

[0061] A span of context tokens is predicted to be inserted before each token. In practice, two self-attention layers can be employed to calculate the probability that a context token is the start or end index of a span: TIFF0007821907000019.tif15170

[0060]

[0062] where: TIFF0007821907000020.tif7170x i x j Then, by applying argmax to P, we can get the start and end indices of each token's span: TIFF0007821907000021.tif16170

[0061]

[0063] TIFF0007821907000022.tif6170x m The probability that a span is inserted before TIFF0007821907000023.tif8170 Insertion may not be present. t The output of the dialogue rewriter for u t ' (e.g., u3'330).

[0062]

[0064] Translation Matching

[0065] The output (pseudo-labels) from the CSRL parser 320 and the dialogue rewriter 315 may be in the form of argument lists of predicates, and in the form of complete sentences rewritten from dialogue turns. To calculate the matching score between the pseudo-labels of the two tasks, the translation matcher 350 leverages a regular semantic role labeler 370 to match the pseudo-labels of the rewritten utterance u t From '330-t, {B1,…,B K}(For example, (pred k ,B K We can extract K arguments of the predicate, denoted as )375-k). TIFF0007821907000024.tif6170 (e.g., m1, m2, etc.) is A k and B k can be calculated based on the edit distance between: TIFF0007821907000025.tif13170

[0063]

[0066] Here, dist() can calculate the edit distance between two strings, and len() can return the length of a string. TIFF0007821907000026.tif5170 Set A of arguments in the given order k It is possible to indicate the concatenation of arguments in the utterance (e.g., the concatenation order of arguments: ARG0, ARG1, ARG2, ARG3, ARG4, ARGM-TMP, ARGM-LOC, ARGM-PRP) (an empty string may mean that no arguments are present). Furthermore, the rewritten utterance u t 'of TIFF0007821907000027.tif5170 could look like this: TIFF0007821907000028.tif9170

[0064]

[0067] Here, GM() may represent the geometric mean 380. The geometric mean 380 may be an example function used to calculate the matching score, although any suitable function may be used instead of and / or in addition to the geometric mean.

[0065]

[0068] Advanced Selectors

[0069] The advanced selector can select high-quality pseudo labels according to both the matching score and the confidence of each task model.

[0066]

[0070] CSRL allows us to calculate a confidence score for each predicate based on the output of the softmax layer. Specifically, By multiplying the probability of the token shown in TIFF0007821907000029.tif6170, predk The confidence of the argument is obtained. Next, pred k The geometric mean of the confidence scores of all arguments belonging to pred k is used as the confidence level of TIFF0007821907000030.tif6170 is calculated as follows: TIFF0007821907000031.tif7170

[0067]

[0071] Here, the hyperparameter α can strike a balance between the matching score and the confidence.

[0068]

[0072] In DR, the probability of inserting a span and the probability of deciding whether to keep a token are defined as b t It is written as u t ' can be multiplied to give the model reliability. TIFF0007821907000032.tif6170 is as follows: TIFF0007821907000033.tif10170

[0069]

[0073] Here, the larger the value of the hyperparameter β, the more importance is placed on the model reliability. To control the number and quality of the selected pseudo labels, s k and r t A threshold can be set to

[0070]

[0074] In the context of CSRL and DR, Tables 1 and 2 show a comparison of Friend training according to embodiments of the present disclosure with several classical or state-of-the-art baselines (e.g., DuConv to WeiboCSRL and / or Rewrite to Restoration). Friend training (also referred to as "FDT") achieves the best overall performance significantly outperforming the baselines in both unsupervised domain adaptation and few-shot learning scenarios, demonstrating the effectiveness of Friend training in various experimental situations for utilizing large-scale unlabeled corpora. TIFF0007821907000034.tif47170Table 1 - Unsupervised domain adaptation TIFF0007821907000035.tif55170Table 2 - Few-shot learning

[0071]

[0075] FIG. 4 is a diagram of an example process 400 for training two or more cross-task neural network models based on two or more neural network tasks, according to an embodiment.

[0072]

[0076] In operation 405, a first pseudo label based on a first model associated with a first one of the two or more neural network tasks may be mapped to the same space as a second pseudo label based on a second model associated with a second one of the two or more neural network tasks. As an example, a pseudo label from a CSRL task may be mapped to a pseudo label from a DR task. The mapping may indicate a measure of similarity between the first pseudo label and the second pseudo label.

[0073]

[0077] At operation 410, a matching score indicating a cross-task match between the first pseudo label and the second pseudo label may be calculated based on the mapping. In some embodiments, the matching score may be based on an edit distance between the first pseudo label and the second pseudo label. In some embodiments, the matching score is a value between zero and one.

[0074]

[0078] At operation 415, one or more cross-task pseudo labels may be selected based on the matching scores and accuracy rates associated with the first model and the second model. The selection of the one or more cross-task pseudo labels may be based on a threshold exact match criterion.

[0075]

[0079] In some embodiments, the accuracy rates associated with the first model and the second model can include a first accuracy rate based on a first function of each of the first pseudo labels associated with the first task (e.g., accuracy rate for the CSRL task). In some embodiments, the accuracy rates associated with the first model and the second model can also include a second accuracy rate based on a second function of each of the second pseudo labels associated with the second task (e.g., accuracy rate for the DR task).

[0076]

[0080] At operation 420, two or more cross-task neural network models may be trained based on one or more cross-task pseudo-labels. In some embodiments, two or more neural network tasks may have partially related prediction tasks. The partially related prediction tasks may be partially related via two or more respective translation functions, each of which includes a subset of possible sub-predictions of the two or more neural network tasks.

[0077]

[0081] According to some embodiments, operations 405-420 may be performed using a device configured to execute code, with each operation corresponding to code, such as received code, determined code, generated code, etc.

[0078]

[0082] Embodiments of the present disclosure also provide the flexibility to adjust the learning-based permutation, quantization, encoding, and decoding methods online or offline based on current data, and to support various types of learning-based quantization methods, including DNN-based or traditional model-based methods. The described method also provides a flexible and general framework that accommodates various DNN architectures and multiple quality metrics.

[0079]

[0083] The proposed methods may be used separately or combined in any order. Furthermore, each method (or embodiment) may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits) or may be implemented using software code (e.g., generate code, receive code, encode code, decode code, etc.). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0080]

[0084] This disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from practice of the embodiments.

[0081]

[0085] As used herein, the term component is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0082]

[0086] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the scope of the invention. Accordingly, the operation and behavior of the systems and / or methods are described herein without reference to specific software code. However, it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0083]

[0087] Although combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible embodiments. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible embodiments includes each dependent claim in combination with every other claim in the claim set.

[0084]

[0088] No element, act, or instruction used herein is construed as critical or essential unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Furthermore, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, terms such as "has," "have," and "having" are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise.

Claims

1. 1. A method for training two or more cross-task neural network models based on two or more neural network tasks, the method being executed by at least one processor, the method comprising: mapping a first pseudo-label based on a first model associated with a first task of the two or more neural network tasks and a second pseudo-label based on a second model associated with a second task of the two or more neural network tasks into the same space; calculating a matching score indicative of a cross-task matching between the first pseudo label and the second pseudo label based on the mapping; selecting one or more cross-task pseudo labels based on the matching scores and accuracy rates associated with the first model and the second model; training the two or more cross-task neural network models based on the one or more cross-task pseudo-labels; A method comprising:

2. The method of claim 1 , wherein the mapping indicates a measure of similarity between the first pseudo label and the second pseudo label.

3. The method of claim 1 , wherein the matching score is a value between 0 and 1.

4. The method of claim 1 , wherein the calculating the matching score is based on an edit distance between the first pseudo label and the second pseudo label.

5. The accuracy rates associated with the first model and the second model are: a first accuracy rate based on a first function for each of the first pseudo labels associated with the first task; and a second accuracy rate based on a second function for each of the second pseudo labels associated with the second task; The method of claim 1 , comprising:

6. The method of claim 1 , wherein the two or more neural network tasks have partially related prediction tasks.

7. 7. The method of claim 6, wherein the partially related prediction tasks are partially related via two or more respective translation functions, each of the two or more translation functions comprising a subset of possible sub-predictions of the two or more neural network tasks.

8. The method of claim 7 , wherein each of the two or more translation functions is deterministic.

9. 1. An apparatus for training two or more cross-task neural network models based on two or more neural network tasks, the apparatus comprising: at least one memory configured to store program code; at least one processor configured to read the program code and to act as directed by the program code; wherein the program code comprises: mapping code configured to cause the at least one processor to map a first pseudo-label based on a first model associated with a first task of the two or more neural network tasks and a second pseudo-label based on a second model associated with a second task of the two or more neural network tasks into the same space; computational code configured to cause the at least one processor to compute, based on the mapping, a matching score indicative of a cross-task matching between the first pseudo label and the second pseudo label; selection code configured to cause the at least one processor to select one or more cross-task pseudo labels based on the matching scores and accuracy rates associated with the first model and the second model; training code configured to cause the at least one processor to train the two or more cross-task neural network models based on the one or more cross-task pseudo-labels; 1. An apparatus comprising:

10. The apparatus of claim 9 , wherein the mapping indicates a measure of similarity between the first pseudo label and the second pseudo label.

11. The apparatus of claim 9 , wherein the matching score is a value between 0 and 1.

12. The apparatus of claim 9 , wherein calculating the matching score is based on an edit distance between the first pseudo label and the second pseudo label.

13. The accuracy rates associated with the first model and the second model are: a first accuracy rate based on a first function for each of the first pseudo labels associated with the first task; and a second accuracy rate based on a second function for each of the second pseudo labels associated with the second task; 10. The apparatus of claim 9, comprising:

14. The apparatus of claim 9 , wherein the two or more neural network tasks comprise partially related prediction tasks.

15. 15. The apparatus of claim 14, wherein the partially related prediction tasks are partially related via two or more respective translation functions, each of the two or more translation functions comprising a subset of possible sub-predictions of the two or more neural network tasks.

16. 1. A computer program comprising instructions, which when executed by at least one processor for training two or more cross-task neural network models based on two or more neural network tasks, cause the at least one processor to: mapping a first pseudo-label based on a first model associated with a first task of the two or more neural network tasks and a second pseudo-label based on a second model associated with a second task of the two or more neural network tasks into the same space; calculating a matching score indicative of a cross-task matching between the first pseudo label and the second pseudo label based on the mapping; selecting one or more cross-task pseudo labels based on the matching scores and accuracy rates associated with the first model and the second model; training the two or more cross-task neural network models based on the one or more cross-task pseudo-labels; Computer program.

17. 17. The computer program product of claim 16, wherein the two or more neural network tasks comprise partially related prediction tasks.

18. 20. The computer program product of claim 17, wherein the partially related prediction tasks are partially related via two or more respective translation functions, each of the two or more translation functions comprising a subset of possible sub-predictions of the two or more neural network tasks.

Images