A Reinforced Instance Transfer Learning Method Combined with Domain Adversarial Training

Abstract

The present invention discloses a method for strengthening instance transfer learning combined with domain adversarial training, and adopts a strengthening transfer learning model including a domain discrimination module, a transfer learning module and a data selection module to perform strengthening instance transfer learning, including: training the domain discrimination module and the transfer learning module to maximize and minimize the feature distance of two domain data respectively; making the transfer learning module learn domain invariant features by adversarial training; the data selection module selects data from the source domain data according to the output of the transfer learning module, and outputs it to the transfer learning module; the transfer learning module, the domain discrimination module and the data selection module are trained collaboratively, so that the strengthening transfer learning model can select useful data from the source domain data to help the strengthening transfer learning model learn the target domain data. The present invention can solve the negative transfer problem caused by the different distribution of target domain and source domain data in the traditional transfer learning process.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a reinforcement instance transfer learning method combined with domain adversarial training. Background Art

[0002] Deep neural networks have achieved great success in the fields of computer vision and natural language processing. Their excellent performance is highly dependent on sufficient training data. However, obtaining enough training data takes a lot of time and cost. Transfer learning is a classic method that uses data from a rich source domain to help learn in a scarce target domain. In recent years, transfer learning based on neural networks (also known as deep transfer learning) has achieved excellent performance in the fields of natural language processing and computer vision and has been widely used. However, due to the different data distributions in the source and target domains, ordinary deep transfer methods may have the problem of negative transfer.

[0003] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a reinforced instance transfer learning method combined with domain adversarial training for interpretation recognition, natural language inference and comment usefulness prediction tasks, which can solve the negative transfer problem caused by the different data distributions of the target domain and the source domain in the traditional transfer learning process, thereby enabling the reinforced instance transfer learning method to be effectively applied to interpretation recognition, natural language inference and comment usefulness prediction tasks.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention discloses a method for enhanced instance transfer learning combined with domain adversarial training, which uses an enhanced instance transfer learning model including a domain discrimination module, a transfer learning module and a data selection module to perform enhanced instance transfer learning, and includes the following steps:

[0007] S1: training the domain discrimination module to maximize the feature distance between the source domain data and the target domain data;

[0008] S2: training the transfer learning module to minimize the feature distance between the source domain data and the target domain data;

[0009] S3: The transfer learning module learns domain-invariant features through adversarial training of step S1 and step S2;

[0010] S4: outputting the output of the transfer learning module as a state signal of reinforcement learning to the data selection module, wherein the data selection module selects source domain data according to the state signal, and outputs the source domain data after data selection to the transfer learning module;

[0011] S5: The transfer learning module, the domain discrimination module and the data selection module perform collaborative training by executing steps S1 to S4, so that the reinforcement transfer learning model can select useful data from the source domain data to help the reinforcement transfer learning model learn the target domain data.

[0012] Preferably, the domain discrimination module includes a domain discriminator and a gradient reversal layer, and step S1 specifically comprises: training the domain discriminator through the gradient reversal layer to maximize the feature distance between the source domain data and the target domain data.

[0013] Furthermore, step S4 also includes: outputting the performance of the transfer learning module after data selection as a reward signal to the data selection module, and also using the reward signal to guide the learning of the data selection module during the process of the data selection module selecting source domain data.

[0014] Furthermore, in step S5, the transfer learning module, the domain discrimination module and the data selection module perform collaborative training by executing steps S1 to S4. Specifically, the transfer learning module, the domain discrimination module and the data selection module perform collaborative training on the data in batches by executing steps S1 to S4.

[0015] Preferably, the transfer learning module includes a feature extractor and two domain classifiers, the two domain classifiers are a source domain classifier and a target domain classifier, wherein the feature extractor is used to extract features of source domain data and target domain data and output them to the source domain classifier and the target domain classifier respectively, the source domain classifier is used to classify the source domain data, and the target domain classifier is used to classify the target domain data.

[0016] Preferably, the loss function in the adversarial training process of step S1 and step S2 is:

[0017]

[0018] In the formula, θ f represents the parameters of the feature extractor, represents the parameters of the source domain classifier, represents the parameters of the target domain classifier, θ d represents the parameters of the domain discriminator, ns Indicates the number of samples of source domain data, n t represents the number of samples in the target domain, D s Represents the source domain data, D t Represents the target domain data, represents the loss of the source domain classifier, represents the loss of the target domain classifier, represents the source domain classifier, represents the target domain classifier, G f represents the feature extractor, G d represents the domain discriminator, λ represents the hyperparameter, L d represents the loss of the domain discriminator, d i represents the domain label, where for the target domain data, d i =1, for source domain data, d i =0.

[0019] Preferably, the feature extractor adopts a TextCNN or DAM model, and the source domain classifier and the target domain classifier respectively adopt a fully connected layer.

[0020] Preferably, step S4 specifically includes: cascading the output of the feature extractor and the output of the target domain classifier as a state signal of reinforcement learning Output to the data selection module, the data selection module according to the state signal To get the actions taken on samples of source domain data According to the action The source domain data after data selection is obtained, and the source domain data after data selection is input into the transfer learning module; at the same time, the difference in performance of the transfer learning module after data selection is used as a reward signal r b Output to the data selection module, according to the reward signal r b The expected return is obtained, and maximizing the expected return is used as a goal to guide the learning of the data selection module.

[0021] Preferably, the data selection module selects the data according to the status signal To get the actions taken on samples of source domain data Specifically: the data selection module uses a strategy function To decide the action to take on a sample of source domain data The policy function for:

[0022]

[0023] In the formula, θ rRepresents the parameters of the strategy network used by the data selection module, including Wx and b k ; W k represents the weight matrix of the kth layer of the policy network, b k represents the bias term of the kth layer of the policy network, and g represents the ReLU function.

[0024] Preferably, the reward signal r b The expression is:

[0025]

[0026] In the formula, r b represents the reward signal corresponding to the b-th batch of samples of the source domain data, represents the performance of the transfer learning module after the data selection module selects the bth batch of samples of the source domain data, represents the performance of the transfer learning module before the data selection module selects the bth batch of samples of the source domain data, G f represents the feature extractor, represents the target domain classifier, x b Represents the bth batch of samples of the source domain data;

[0027] According to the reward signal r b Get the expected return J(θ r ):

[0028]

[0029] In the formula, represents the policy function of the policy network adopted by the data selection module, E represents the expectation, t represents the index value of the current batch, R t represents the cumulative return of the t-th batch of samples of the source domain data, where the cumulative return R t The expression is:

[0030]

[0031] Where T represents the number of batches of source domain data in a training round, γ represents the discount factor, and γ b represents the discount factor to the power of b, r t+b Represents the reward signal corresponding to the t+bth batch of samples in the source domain data.

[0032] Preferably, step S4 further comprises: updating the data selection module in a policy gradient-based manner to maximize the expected return, wherein the parameter θ of the policy network used by the data selection module is r The update formula is:

[0033]

[0034] In the formula, α represents the learning rate, Represents the gradient.

[0035] The present invention also discloses a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned enhanced instance transfer learning method are implemented.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows: the reinforced instance transfer learning method combined with domain adversarial training for interpretation recognition, natural language inference and comment usefulness prediction tasks proposed in the present invention, wherein the domain discrimination module and the transfer learning module perform adversarial training to learn domain invariant features, and also provide a good state representation for the data selection module based on reinforcement learning, wherein the domain discrimination module, the transfer learning module and the data selection module are independent modules, and adversarial training and reinforcement learning are performed simultaneously during the training process to improve the training efficiency of the model; at the same time, it also effectively alleviates the negative transfer problem in the transfer learning process, and can efficiently help applications and learning in low-resource fields, so that the reinforced instance transfer learning method can be effectively applied to interpretation recognition, natural language inference and comment usefulness prediction tasks.

[0037] In a further solution, the transfer learning module, domain discrimination module and data selection module in the present invention collaboratively train the data in batches, greatly improving the training efficiency of the model.

[0038] In a further scheme, the performance of the transfer learning module after data selection is used as a reward signal to guide the learning of the data selection module, thereby further improving the data selection ability of the data selection module.

[0039] In a further scheme, the transfer learning module includes a feature extractor and two domain classifiers, wherein the feature extractor is shared by the source domain and the target domain, and the two domain classifiers are unique to the two domains, respectively, so that the domain-invariant features (i.e., transferable features) and domain-specific features can be fully learned, thereby improving the migration effect; further, by cascading the output of the feature extractor and the output of the target domain classifier as the state signal of reinforcement learning, a better state representation can be provided for the reinforcement learning of the data selection module. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the architecture and flow of a reinforcement instance transfer learning model combined with domain adversarial training according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0042] In order to solve the problem of negative transfer that may exist in deep transfer methods, an effective solution is to use instance-based transfer learning to select useful data from the source domain to alleviate or avoid the negative transfer problem. Recently, some instance-based transfer learning works have applied reinforcement learning to the selection of source domain data. For example, some research works have proposed the Kernel Mean Matching (KMM) method to estimate instance weights. The goal is to make the weighted auxiliary domain data and the target domain data more similar in probability distribution. The distribution similarity measure is the mean difference between the weighted auxiliary data and the target data in the reproducible Hilbert space (RKHS). Some researchers use relative entropy KL (Pt(x)||w(x)Ps(x)) to measure the probability distribution difference. The goal is to estimate the instance weight so that the distribution similarity between the weighted auxiliary data and the target data is improved. The calculation of instance weights is reduced to estimating a mixed basis function model. Some research works have also proposed instance transfer learning methods based on AdaBoost to increase the instance weights that are beneficial to the target classification task and reduce the instance weights that are not beneficial to the target classification task. The generalization error upper bound of the model is derived based on the PAC theory. Some recent research works have achieved better results by co-training the data selection module based on reinforcement learning and the transfer learning module. However, the inventors found through research that the existing technology still has the following two problems:

[0043] (1) As a submodule of the data selection module, the transfer learning module updates the data selection strategy according to the final performance of the transfer learning module. The transfer learning module needs to be repeatedly trained multiple times to provide sufficient updates for the data selection module, resulting in reduced model training efficiency.

[0044] (2) The transfer learning module used in existing work is a simple fully connected network, which cannot fully learn domain-invariant features (i.e., transferable features) and domain-specific features, resulting in poor transfer effect. In addition, the transfer learning module cannot further provide a good state representation for the reinforcement learning module.

[0045] The present invention proposes a reinforcement transfer learning method combined with domain adversarial training to select useful data from the source domain to help the target domain. Figure 1As shown, the reinforcement transfer learning model of the present invention is composed of three modules: a domain discrimination module 10, a transfer learning module 20 and a data selection module 30 based on reinforcement learning. Among them, the domain discrimination module 10 includes a domain discriminator 11 and a gradient reversal layer 12. The domain discriminator 11 is trained by the gradient reversal layer 12 to maximize the feature distance between the source domain data and the target domain data. The transfer learning module 20 is trained to minimize the feature distance between the source domain data and the target domain data. Through this adversarial training method, the transfer learning module 20 learns domain invariant features (making it easier to transfer); the data selection module 30 uses the output of the transfer learning module 20 as the state signal of reinforcement learning, selects the source domain data, and inputs the source domain data after data selection into the transfer learning module 20, and uses the performance of the transfer learning module 20 after data selection as a reward signal to guide the data selection module 30 based on reinforcement learning to select data. In this way, the transfer learning module 20, the domain discriminator module 10 and the data selection module 30 cooperate with each other and train collaboratively in batches, so that the reinforcement transfer learning model can select useful data from the source domain data to help strengthen the learning of the transfer learning model in the target domain.

[0046] Combination Figure 1 As shown, the enhanced example transfer learning method combined with domain adversarial training proposed in the preferred embodiment of the present invention includes the following steps:

[0047] S1: training the domain discrimination module 10 to maximize the feature distance between the source domain data and the target domain data;

[0048] The domain discrimination module 10 includes a domain discriminator 11 and a gradient reversal layer 12. The gradient reversal layer 12 trains the domain discriminator 11 to maximize the feature distance between the source domain data and the target domain data.

[0049] S2: training the transfer learning module 20 to minimize the feature distance between the source domain data and the target domain data;

[0050] Among them, the transfer learning module 20 includes a feature extractor 21 and two domain classifiers, the two domain classifiers are a source domain classifier 22 and a target domain classifier 23, wherein the feature extractor 21 is used to extract features of source domain data and target domain data and output them to the source domain classifier 22 and the target domain classifier 23 respectively, the source domain classifier 22 is used to classify the source domain data, and the target domain classifier 23 is used to classify the target domain data.

[0051] S3: The transfer learning module 20 learns domain-invariant features through adversarial training of step S1 and step S2;

[0052] In a specific embodiment, given the source domain data Target area data Among them, n s The number of samples representing the source domain data, n t The number of samples representing the target domain data; and Represent samples and sample labels of source domain data respectively; and Represent the samples and sample labels of the target domain data respectively. The present invention hopes to use the source domain data D s Knowledge enhancement target domain data D t Performance (D s ≠D t ). Since the information of the source domain data is usually richer than that of the target domain data, the present invention can use the information of the source domain data with rich information to assist the learning of the target domain data.

[0053] The training process is divided into two stages: adversarial training (steps S1 to S3) and data selection using reinforcement learning (step S4). The following first introduces the adversarial training stage.

[0054] Theoretical research indicates that learning domain-invariant features in transfer learning can effectively improve the effect of transfer learning. Therefore, the present invention adopts adversarial training to learn domain-invariant features, thereby further reducing the difference in data distribution between domains. In the generative adversarial network GAN, the generator generates samples to confuse the discriminator, which is responsible for judging whether the sample is real or generated by the generator. In transfer learning, the feature extractor 21 plays the role of the generator in the generative adversarial network, and continuously learns the features of the domain data so that the domain discriminator 12 cannot distinguish between the two domains. The present invention trains the domain discriminator G d Distinguish the feature distribution differences between source domain data and target domain data, and train the feature extractor G f Extract domain-invariant features to confuse the domain discriminator; G d and G f These two units play a min-max game during the entire adversarial training process: the feature extractor G f Learn transferable features to confuse the domain discriminator G d , that is, maximize the domain discriminator G d The loss function of the domain discriminator G d Learn how to distinguish between the source domain and the target domain, that is, minimize the classification error of the domain discriminator.

[0055] Specifically, the domain discriminator 12 uses a gradient reversal layer 11 to implement adversarial loss. The gradient reversal layer 11 can be regarded as an identity transformation during forward propagation, and as multiplying the gradient by -λ (λ is a hyperparameter representing a negative constant) to change the direction of the gradient during reverse propagation. The gradient reversal layer has no parameter updates and can be regarded as a pseudo function Q(x) defined by the following two equations:

[0056] Q(x)=x

[0057]

[0058] Here, I represents the identity matrix. In this way, the domain-shared feature extractor can generate features that are difficult for the domain discriminator to distinguish, namely, domain-invariant features.

[0059] In this embodiment, the loss function in the adversarial training process is:

[0060]

[0061] Among them, θ f represents the parameters of the feature extractor, represents the parameters of the source domain classifier, represents the parameters of the target domain classifier, θ d represents the parameters of the domain discriminator, and Represent the loss of the source domain classifier and the target domain classifier, respectively, s Represents the source domain data, D t represents the target domain data, G f represents the feature extractor, G d represents the domain discriminator, represents the source domain classifier, represents the target domain classifier; λ is a hyperparameter, representing a negative constant, which can balance the learning of the domain discriminator and the two domain classifiers. d represents the loss of the domain discriminator, d i Represents the domain label (for the target domain, d i =1, for the source domain, d i =0).

[0062] S4: Outputting the output of the transfer learning module 20 as a state signal of reinforcement learning to the data selection module 30, the data selection module 30 performs data selection on the source domain data according to the state signal, and outputs the source domain data after data selection to the transfer learning module 20;

[0063] Specifically, the output of the feature extractor 21 and the output of the target domain classifier 23 are cascaded as the state signal of reinforcement learning Output to the data selection module 30, the data selection module 30 according to the state signal To get the actions taken on samples of source domain data According to the action The source domain data after data selection is obtained, and the source domain data after data selection is input into the transfer learning module 20; at the same time, the difference in performance of the transfer learning module 20 after data selection is used as the reward signal r b Output to the data selection module 30, according to the reward signal r b The expected return is obtained, and maximizing the expected return is used as a goal to guide the learning of the data selection module 30.

[0064] This step is the step of using reinforcement learning to select data. The reinforcement learning is described in detail below.

[0065] In this specific embodiment, the data selection module 30 is composed of a two-layer neural network. In this embodiment, the entire data selection process is modeled as a Markov decision process (MDP): the data selection module 30 based on reinforcement learning is used as an agent that continuously interacts with the environment. The agent specifically adopts a strategy network 31, and the action output by the strategy network 31 is used express: in, represents the action taken on the sample of the source domain data, b represents the batch index value, n represents the batch size (that is, the number of samples contained in a batch of source domain data), a i ∈{0,1} indicates whether to select the i-th sample in the source domain.

[0066] Combination Figure 1 As shown, the source domain data set to be selected is used Indicates that the output cascade of the shared feature extractor 21 and the target domain classifier 23 in the transfer learning module 20 is used as the state in reinforcement learning, and express: The samples of source domain data and target domain data are sent to the transfer learning module 20 for training, and the strategy network 31 is trained through the strategy function To decide the action to take on the source domain data sample And get a new source domain data set The new source domain data set and target domain data set Send it to the transfer learning module for training, and r bAs a reward signal to guide the learning of the data selector, the goal is to maximize the expected return. Figure 1 In the data selection module 30, a strategy network 31 is used to select source domain data sets. (indicated by the dark gray circle on the upper left) to obtain a new source domain data set (The dark gray circle in the lower left corner represents the retained data, and the light gray circle represents the discarded data), and then from the new source domain data set The source domain data after data selection is output to the transfer learning module 20.

[0067] Among them, state representation (state): The feature extractor learns domain-invariant features through adversarial training. For source domain data For example, the output of the feature extractor 21 and the output of the target domain classifier 23 are cascaded as the state representation

[0068] Action: a i Indicates whether the data selector will select the i-th sample from the source domain according to a i Can get action action It is determined by the policy function, which Calculated by:

[0069]

[0070] In the formula, θ r The parameters of the policy network used by the data selection module 30 include W k and b k ; W k represents the weight matrix of the kth layer of the policy network, b k represents the bias term of the kth layer of the policy network, and g represents the ReLU function.

[0071] Reward function: The goal of the data selection module 30 is to maximize the expected return:

[0072]

[0073] θ r Represents the parameters of the data selector, R t represents the cumulative return, E represents the expectation, t represents the index value of the current batch, and R t Represents the cumulative return of the t-th batch of samples of the source domain data. The goal of the data selection module is to learn the policy function To maximize the expected return. Different from the traditional reinforcement learning method, the present invention provides reward signals in batches to update the model.

[0074] The difference in performance of the transfer learning module before and after data selection is used as the reward for the action of the data selection module. For each batch, the reward function r b for:

[0075]

[0076] In the formula, r b represents the reward signal corresponding to the b-th batch of samples of the source domain data, It represents the performance of the transfer learning module after the data selection module selects the bth batch of samples of the source domain data. represents the performance of the transfer learning module before the data selection module selects the bth batch of samples of the source domain data, G f represents the feature extractor, represents the target domain classifier, x b represents the bth batch of samples of the source domain data; for classification tasks, L can be set to the accuracy on the target domain validation set; for regression tasks, L is the correlation coefficient between the predicted value and the true value.

[0077] Different from the traditional reinforcement learning method, the model of the present invention is updated in batches, which greatly improves the training efficiency of the model. For a single batch of samples in a training episode, the cumulative reward is:

[0078]

[0079] Where T represents the number of batches of source domain data in a training round, γ represents the discount factor, and γ b represents the discount factor to the power of b, r t+b Represents the reward signal corresponding to the t+bth batch of samples of the source domain data; t represents the index value of the current batch, and b represents a variable with a value range of [0,Tt].

[0080] Optimization: The present invention also uses a policy gradient-based approach to update the data selection module to maximize the expected return J(θ). The update formula is as follows:

[0081]

[0082] Among them, α represents the learning rate, θ r Represents the parameters of the policy network. represents the gradient, R t Represents the cumulative return of the t-th batch of samples of the source domain data.

[0083] S5: The transfer learning module 10, the domain discrimination module 20 and the data selection module 30 perform collaborative training by executing steps S1 to S4, so that the reinforcement transfer learning model can select useful data from the source domain data to help the reinforcement transfer learning model learn the target domain data.

[0084] It should be noted that the above steps S1 to S4 are not executed in sequence in the order of S1 to S4, but are collaboratively trained synchronously, so that adversarial training and reinforcement learning are carried out simultaneously and collaborate with each other, and ultimately the reinforcement transfer learning model can select useful data from the source domain data to help the reinforcement transfer learning model learn the target domain data.

[0085] Another preferred embodiment of the present invention discloses a computer-readable storage medium storing a computer program, which implements the steps of the above-mentioned reinforcement instance transfer learning method when executed by a processor.

[0086] The enhanced instance transfer learning method combined with domain adversarial training proposed in the preferred embodiment of the present invention is universal and can be used for different task feature extractors G f Different network structures can be used.

[0087] The effectiveness of the proposed method is verified on three real-world tasks: paraphrase recognition, natural language inference, and comment usefulness prediction. For natural language inference and paraphrase recognition tasks, the DAM (Decomposable Attention Mode) model is used as a feature extractor shared by the source domain and the target domain, and the two fully connected layers are used as two domain classifiers. The comparison of the results of the method of the present invention and the prior art is shown in Table 1. For the comment usefulness prediction task, TextCNN can be used as a feature extractor shared by the source domain and the target domain. f , and the two fully connected layers are used as two domain classifiers respectively. The comparison of the results of the method of the present invention and the prior art is shown in Table 2. The transfer learning module and the domain discrimination module are trained collaboratively so that the transfer learning module learns the domain invariant features, thereby providing a good state representation for the data selection module based on reinforcement learning.

[0088] Table 1

[0089]

[0090] Table 2

[0091]

[0092] The relevant definitions in Table 1 and Table 2 are as follows:

[0093] ACC: accuracy;

[0094] AUC: area of ​​the ROC curve;

[0095] Src-only: Only source domain data is used to train the model, and target domain data is used for testing;

[0096] Tgt-only: Use target domain data to train and test the model;

[0097] TL-shared: A classic transfer learning method that uses a domain-shared feature extractor and two domain-specific classifiers;

[0098] Ruder and Plank: "Learning to select data for transfer learning with bayesian optimization" proposed a method for data selection using Bayesian optimization;

[0099] MGTL: Mini-Max Adversarial Methods for Instance Transfer Learning;

[0100] RSTL: A reinforced instance transfer learning method combined with domain adversarial training according to a preferred embodiment of the present invention;

[0101] RSTL w / o adv : RSTL method removes adversarial learning;

[0102] RSTL w / o RL : The RSTL method removes the data selection module based on reinforcement learning;

[0103] STR, UGR, LIWC, INQ (INQUIRER), and ASP are baseline models that use manually defined features;

[0104] Fusion 1: A method that integrates STR, UGR, LIWC, and INQ features;

[0105] Fusion 2: A method that integrates STR, UGR, LIWC, INQ, and ASP features;

[0106] TL-dd: A cross-domain model with an additional domain discriminator;

[0107] TL-adv: A transfer learning method that uses adversarial training to alleviate the interference between shared and private features.

[0108] It can be seen from Table 1 and Table 2 that the method (RSTL) proposed in the preferred embodiment of the present invention significantly exceeds the baseline model in three real-world tasks. The model RSTL with the domain discrimination module removed is w / o adv , delete the model RSTL of the data selection module w / o RL The comparison also shows the important role of the domain discrimination module and the data selection module. The experimental results fully demonstrate the effectiveness of the reinforcement transfer learning method combined with adversarial training.

[0109] The background section of the present invention may contain background information about the problem or environment of the present invention, rather than describing the prior art by others. Therefore, the content contained in the background section is not an admission by the applicant that the prior art is available.

[0110] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, they can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, the technical personnel in this field can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope defined by the attached claims.

Claims

1. A reinforcement instance transfer learning method combined with domain adversarial training for paraphrase recognition, natural language inference and review usefulness prediction tasks, characterized in that: A reinforcement transfer learning model including a domain discrimination module, a transfer learning module and a data selection module is used to perform reinforcement instance transfer learning, including the following steps: S1: training the domain discrimination module to maximize the feature distance between the source domain data and the target domain data; S2: training the transfer learning module to minimize the feature distance between the source domain data and the target domain data; S3: The transfer learning module learns domain-invariant features through adversarial training of step S1 and step S2; S4: outputting the output of the transfer learning module as a state signal of reinforcement learning to the data selection module, wherein the data selection module selects source domain data according to the state signal, and outputs the source domain data after data selection to the transfer learning module; S5: The transfer learning module, the domain discrimination module and the data selection module perform collaborative training by executing steps S1 to S4, so that the reinforcement transfer learning model can select useful data from the source domain data to help the reinforcement transfer learning model learn the target domain data; Among them, the transfer learning module includes a feature extractor and two domain classifiers, the two domain classifiers are a source domain classifier and a target domain classifier, wherein the feature extractor is used to extract features of source domain data and target domain data and output them to the source domain classifier and the target domain classifier respectively, the source domain classifier is used to classify source domain data, and the target domain classifier is used to classify target domain data; the source domain classifier and the target domain classifier respectively adopt fully connected layers, wherein for natural language inference and interpretation recognition tasks, the feature extractor adopts the DAM model, and for the comment usefulness prediction task, the feature extractor adopts TextCNN.

2. The method for enhanced instance transfer learning according to claim 1, characterized in that: The domain discrimination module includes a domain discriminator and a gradient reversal layer, and step S1 specifically includes: training the domain discriminator through the gradient reversal layer to maximize the feature distance between the source domain data and the target domain data; further, step S4 also includes: outputting the performance of the transfer learning module after data selection to the data selection module as a reward signal, and in the process of the data selection module selecting source domain data, the reward signal is also used to guide the learning of the data selection module; further, in step S5, the transfer learning module, the domain discrimination module and the data selection module are collaboratively trained by executing steps S1 to S4, specifically: the transfer learning module, the domain discrimination module and the data selection module are collaboratively trained in batches by executing steps S1 to S4.

3. The method for enhanced instance transfer learning according to claim 1, characterized in that: The loss function in the adversarial training process of step S1 and step S2 is: In the formula, θ f represents the parameters of the feature extractor, represents the parameters of the source domain classifier, represents the parameters of the target domain classifier, θ d represents the parameters of the domain discriminator, n s Indicates the number of samples of source domain data, n t represents the number of samples in the target domain, D s Represents the source domain data, D t Represents the target domain data, represents the loss of the source domain classifier, represents the loss of the target domain classifier, represents the source domain classifier, represents the target domain classifier, G f represents the feature extractor, G d represents the domain discriminator, λ represents the hyperparameter, L d represents the loss of the domain discriminator, d i represents the domain label, where for the target domain data, d i =1, for source domain data, d i =0.

4. The method for enhanced instance transfer learning according to claim 1, characterized in that: Step S4 specifically includes: cascading the output of the feature extractor and the output of the target domain classifier as a state signal of reinforcement learning Output to the data selection module, the data selection module according to the state signal To get the actions taken on samples of source domain data According to the action The source domain data after data selection is obtained, and the source domain data after data selection is input into the transfer learning module; at the same time, the difference in performance of the transfer learning module after data selection is used as a reward signal r b Output to the data selection module, according to the reward signal r b The expected return is obtained, and maximizing the expected return is used as a goal to guide the learning of the data selection module.

5. The method for enhanced instance transfer learning according to claim 4, characterized in that: The data selection module selects the data according to the status signal To get the actions taken on samples of source domain data Specifically: the data selection module uses a strategy function To decide the action to take on a sample of source domain data The policy function for: In the formula, θ r Represents the parameters of the strategy network used by the data selection module, including W k and b k ; W k represents the weight matrix of the kth layer of the policy network, b k represents the bias term of the kth layer of the policy network, and g represents the ReLU function.

6. The method for enhanced instance transfer learning according to claim 4, characterized in that: Reward signal r b The expression is: In the formula, r b represents the reward signal corresponding to the b-th batch of samples of the source domain data, represents the performance of the transfer learning module after the data selection module selects the bth batch of samples of the source domain data, represents the performance of the transfer learning module before the data selection module selects the bth batch of samples of the source domain data, G f represents the feature extractor, represents the target domain classifier, x b Represents the bth batch of samples of the source domain data; According to the reward signal r b Get the expected return J(θ r ): In the formula, represents the policy function of the policy network adopted by the data selection module, E represents the expectation, t represents the index value of the current batch, R t represents the cumulative return of the t-th batch of samples of the source domain data, where the cumulative return R t The expression is: Where T represents the number of batches of source domain data in a training round, γ represents the discount factor, and γ b represents the discount factor to the power of b, r t+b Represents the reward signal corresponding to the t+bth batch of samples in the source domain data.

7. The method for enhanced instance transfer learning according to claim 6, characterized in that: Step S4 also includes: updating the data selection module in a policy gradient-based manner to maximize the expected return, wherein the parameter θ of the policy network used by the data selection module is r The update formula is: In the formula, α represents the learning rate, Represents the gradient.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the enhanced instance transfer learning method applied to interpretation recognition, natural language inference and review usefulness prediction tasks described in any one of claims 1 to 7 are implemented.

Images