Transform-based natural language processing system

Abstract

The invention relates to the technical field of natural language processing, and discloses a natural language processing system based on Transform, comprising: a neural architecture search assembly for automatically searching and optimizing a Transform model architecture for natural language processing; the multi-task learning component is used for simultaneously training a plurality of natural language processing tasks on the basis of the shared model layer; the meta-learning component is used for optimizing parameter initialization of the model, so that the model can quickly adapt to a new task through a small amount of data and training steps; and the dynamic data enhancement component is used for generating diversified training data and enhancing the generalization ability and robustness of the model. According to the method, knowledge sharing and resource efficiency are enhanced through the multi-task learning and meta-learning component, dependence on a large amount of data is reduced, the generalization ability and rapid adaptability of the model are improved, the adaptability to a new scene is enhanced through the dynamic data enhancement component, and the robustness of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a natural language processing system based on Transformer. Background Art

[0002] Since the introduction of the Transformer architecture, it has become an important breakthrough in the field of natural language processing (NLP). Due to its unique self-attention mechanism, the Transformer is able to capture long-range dependencies in text data, which shows excellent results when processing various language model tasks. The core advantage of the Transformer lies in its parallel processing capabilities and its ability to dynamically pay attention to different parts of a sequence, which enables it to achieve advanced performance in a variety of tasks such as machine translation, text summarization, and sentiment analysis.

[0003] Existing natural language processing systems usually use the standard Transformer architecture or its variants to perform specific tasks, such as the BERT and GPT series. These models learn rich language representations by pre-training on large amounts of text data, and can then be fine-tuned to adapt to specific downstream tasks. This pre-training plus fine-tuning method greatly improves the efficiency and effectiveness of processing specific tasks, allowing Transformer-based models to reach new performance heights in multiple NLP benchmarks.

[0004] Although existing Transformer-based NLP systems have performed well in many fields, they still have some shortcomings. First, the training and optimization process of a single task often cannot utilize the potential connections between different tasks, resulting in low resource utilization and lack of knowledge transfer between tasks. Second, when these systems encounter inconsistent data distribution or new tasks, they require a large amount of new data and repeated training, which limits the generalization ability and flexibility of the model. Finally, when dealing with multiple tasks, existing systems often need to adjust or retrain the model for each task separately, which not only increases the computational cost, but also reduces the scalability of the system. Summary of the invention

[0005] In view of the deficiencies of the prior art, the present invention provides a natural language processing system based on Transformer. By introducing multi-task learning components and meta-learning components, the present invention improves the knowledge sharing and resource utilization of the system when processing multiple tasks, while reducing the dependence on a large amount of training data, improving the generalization ability of the model and the ability to quickly adapt to new tasks. The dynamic data enhancement component further enhances the adaptability of the model to new scenarios, enhances its robustness, and solves the deficiencies of the prior art in flexibility, efficiency and generalization.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a natural language processing system based on Transformer, comprising:

[0007] A neural architecture search component for automatically searching and optimizing Transformer model architectures for natural language processing;

[0008] A multi-task learning component for simultaneously training multiple natural language processing tasks based on shared model layers;

[0009] A meta-learning component that optimizes the model’s parameter initialization so that it can quickly adapt to new tasks with a small amount of data and training steps;

[0010] Dynamic data augmentation component, used to generate diverse training data and enhance the generalization and robustness of the model.

[0011] Preferably, the neural architecture search component comprises:

[0012] Define searchable architecture parameters, including the type of self-attention mechanism, the number of Transformer layers, the number of attention heads, the width of the feedforward network, and the type of activation function;

[0013] Use the DARTS algorithm to explore the optimal Transformer architecture in the search space in a continuously differentiable manner;

[0014] The controller is optimized via a policy gradient method in reinforcement learning, which produces an optimal architecture for a natural language processing task.

[0015] Preferably, the neural architecture search component further comprises:

[0016] The controller parameters θ are optimized by the policy gradient method using the following formula:

[0017]

[0018] Among them, θ: controller parameters, representing the strategy of generating network architecture, α: learning rate, controlling the amplitude of each parameter update, affecting the convergence speed and stability, Represents the architecture set, including all candidate architectures α, R(a): the performance score of architecture a on the validation set, usually the accuracy or loss of the model on a specific task, reflecting the effectiveness of the architecture; Represents the gradient of the controller parameter θ, pointing to the direction where the loss function rises fastest;

[0019] P(a|θ): Given the parameter θ, the probability distribution of the generated architecture a is usually generated using the softmax function:

[0020]

[0021] Here, f(a;θ) is the score function associated with architecture a, which characterizes the quality of architecture generation.

[0022] Preferably, the multi-task learning component includes:

[0023] Set up a shared encoder layer in the underlying architecture of Transformer for feature representation of multiple natural language processing tasks;

[0024] Setting task-specific adaptation layers on each task enables the model to handle the special requirements of each task;

[0025] An uncertainty weighted method is applied to dynamically adjust the weight of the loss function according to the uncertainty of each task.

[0026] Preferably, the multi-task learning component further comprises:

[0027] The uncertainty-weighted loss function includes the following formula:

[0028]

[0029] Among them, L i is the loss function of the ith task, σ i is the model uncertainty parameter for the ith task.

[0030] Preferably, the meta-learning component includes:

[0031] Optimize the model's parameter initialization through a model-independent meta-learning algorithm;

[0032] In the inner loop, a small amount of gradient updates are performed for each task, so that the model generates task-related parameters;

[0033] In the outer loop, the initial parameters are globally updated based on the accumulated losses of all tasks, allowing them to quickly adapt to new tasks with a small number of gradient updates.

[0034] Preferably, the meta-learning component further comprises:

[0035] The initialization parameters of the model in the outer loop are updated by the following formula:

[0036]

[0037] Among them, φ is the initialization parameter, α is the learning rate, It's a task The loss function of .

[0038] Preferably, the dynamic data enhancement component includes:

[0039] A conditional generative adversarial network is used to generate training samples that are semantically consistent with the original training data but in a variety of forms;

[0040] Generate realistic data samples by training the generator and classify the generated data and real data through the discriminator;

[0041] The generated data is added to the model training set to enhance the generalization and robustness of the model.

[0042] Preferably, the dynamic data enhancement component further comprises:

[0043] The generator and discriminator of the conditional generative adversarial network are trained with the following objective function:

[0044]

[0045] Where x is a real data sample, G(z) is a generated data sample, D(x) is the probability that the discriminator identifies the real data, and D(G(z)) is the probability that the discriminator identifies the generated data.

[0046] Preferably, the multi-task learning component is designed by separating the shared Transformer encoder layer from the task-specific adaptation layer, so that whenever a new task is added, the corresponding task-specific adaptation layer can be directly added without modifying the shared Transformer encoder layer.

[0047] The present invention provides a natural language processing system based on Transformer, which has the following beneficial effects:

[0048] 1. The present invention adopts a design in which a shared Transformer encoder layer is separated from a task-specific adaptation layer, allowing the system to quickly add or update a task-specific adaptation layer without modifying the existing encoder architecture, thereby improving the flexibility and scalability of the system in processing a variety of natural language tasks.

[0049] 2. The dynamic data enhancement component of the present invention enriches the training data set through the training samples generated by the conditional generative adversarial network, and enhances the model's ability to handle new and unseen data. This method improves the generalization ability of the model, making it more robust, and is particularly effective in dealing with inconsistencies in data distribution.

[0050] 3. The present invention not only reduces the model's dependence on a large amount of training data through the joint action of meta-learning components and multi-task learning components, but also improves resource utilization and model training efficiency during training by sharing learning features and dynamically adjusting training resource allocation, which makes the system more efficient when facing multiple tasks and new tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Please refer to the attached Figure 1 , an embodiment of the present invention provides a natural language processing system based on Transformer, including:

[0054] Neural architecture search component, used to automatically search and optimize the Transformer model architecture for natural language processing. The NAS component first builds multiple candidate architectures and evaluates them based on their performance on the validation set. The system continuously adjusts the architecture search strategy through the policy gradient optimization method until it finds the model architecture that performs best in multiple natural language processing tasks. Through the NAS component, the system can dynamically search and optimize the optimal Transformer architecture according to task requirements, avoiding the complexity of manual parameter adjustment and improving the adaptability and performance of the model in different tasks;

[0055] Multi-task learning component, used to train multiple natural language processing tasks simultaneously on the basis of a shared model layer. The training of multiple tasks is balanced by dynamically adjusting the loss weights of the tasks. The system automatically adjusts the allocation of learning resources between tasks according to the uncertainty and complexity of the tasks to ensure that each task can fairly obtain the learning ability of the model during the training process. Through the MTL component, the system can effectively share knowledge between tasks when processing multiple tasks, improving the efficiency of training and the ability of tasks to collaborate with each other. At the same time, the dynamic weight adjustment mechanism further enhances the system's ability to adapt to task complexity, enabling the model to balance the learning progress of different tasks and improve the performance of the overall system.

[0056] Meta-learning component, used to optimize the model's parameter initialization so that it can quickly adapt to new tasks with a small amount of data and training steps. Through the meta-learning component, the system significantly improves its ability to quickly adapt to new tasks and reduces its dependence on large amounts of data and long training times. It is especially suitable for scenarios where data is scarce or tasks change frequently. The introduction of meta-learning makes the system more efficient and flexible in handling multiple and new tasks, shortens the model's training cycle, and improves the model's practicality and performance in actual applications.

[0057] Dynamic data enhancement components are used to generate diverse training data and enhance the generalization and robustness of the model. Through dynamic data enhancement components, the system improves the diversity of training data and enhances the robustness of the model in the face of data imbalance and data scarcity. The generated diverse data can better help the model learn complex language features and improve the generalization ability of the model, so that it can better adapt to different scenarios and changing data environments in practical applications.

[0058] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the neural architecture search component comprises:

[0059] Define searchable architecture parameters, including the type of self-attention mechanism, the number of Transformer layers, the number of attention heads, the width of the feedforward network, and the type of activation function. The neural architecture search component not only improves the adaptability and performance of the natural language processing system, but also improves the efficiency and effectiveness of system design.

[0060] Use the DARTS algorithm to explore the optimal Transformer architecture in the search space in a continuously differentiable way. The DARTS algorithm is an efficient neural architecture search method that allows the choice of architecture to be directly optimized through gradient descent. DARTS parameterizes the architecture selection variables as continuous differentiable variables, which means that the gradients of these variables can be directly calculated through backpropagation.

[0061] The controller is optimized through the policy gradient method in reinforcement learning, which generates the optimal architecture for natural language processing tasks. The controller generates architectural decisions based on the current state (such as the performance of the previous architecture) through the policy network (usually a recurrent neural network or a Transformer network). Each decision corresponds to a reward, that is, the performance of the architecture on the validation set. Through the policy gradient method, the system optimizes the parameters of the controller to maximize the expected reward, thereby learning to generate a better architectural configuration. Through DARTS and the policy gradient method, the system can automatically discover and optimize the optimal Transformer architecture suitable for specific natural language processing tasks, reducing human intervention and design deviations.

[0062] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the neural architecture search component further comprises:

[0063] The controller parameters θ are optimized by the policy gradient method using the following formula:

[0064]

[0065] Among them, θ: controller parameters, representing the strategy of generating network architecture, α: learning rate, controlling the amplitude of each parameter update, affecting the convergence speed and stability, Represents the architecture set, including all candidate architectures α, R(a): the performance score of architecture a on the validation set, usually the accuracy or loss of the model on a specific task, reflecting the effectiveness of the architecture; Represents the gradient of the controller parameter θ, pointing to the direction where the loss function rises fastest;

[0066] P(a|θ): Given the parameter θ, the probability distribution of the generated architecture a is usually generated using the softmax function:

[0067]

[0068] Among them, f(a;θ) is the score function associated with architecture a, which characterizes the quality of architecture generation;

[0069] Implementation process:

[0070] Initialize controller parameters:

[0071] Randomly initialize the controller parameters θ to lay the foundation for the subsequent architecture generation and determine the learning rate a, which can generally be set to 0.01 and dynamically adjusted according to the complexity of the task;

[0072] Generate network architecture:

[0073] Generate a set of candidate network architectures a using the current controller parameters θ. This generation process is usually implemented through a parameterized strategy, such as using a softmax function.

[0074] Calculate the policy gradient:

[0075] Calculate the gradient of the controller parameter θ with respect to the expected reward. The key to this step is to multiply the performance score of each architecture by its generation probability, sum it over all architectures, and then calculate the gradient of this sum;

[0076] Delivery process:

[0077] Repeat the above steps until the convergence condition is met or the predetermined number of iterations is reached.

[0078] The policy gradient method allows the controller to quickly adapt to performance feedback, thereby searching the architecture space more efficiently, reducing search time and resource consumption. At the same time, this adaptive capability enables the NAS system to cope with a variety of tasks and data sets, improving the generalization ability of the model.

[0079] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the multi-task learning component includes:

[0080] A shared encoder layer is set up in the underlying architecture of Transformer for feature representation of multiple natural language processing tasks. In a multi-task learning environment, the shared encoder layer is responsible for extracting high-level features common to each task. For example, whether it is text classification, sentiment analysis or machine translation, it is necessary to extract grammatical structure and semantic information from the text. These common features are learned and provided by the shared encoder layer. The advantage of this architecture is that it can utilize the information overlap between different tasks and improve learning efficiency and model generalization ability by learning multiple tasks at once;

[0081] Setting a task-specific adaptation layer on each task enables the model to handle the special needs of each task. By sharing the encoder layer, the MTL component enables the system to share the underlying language features between multiple tasks, improving learning efficiency and feature reuse. This not only reduces the computational cost of repeated learning, but also helps improve the performance of the model on similar tasks that have not been seen before. At the same time, whenever a new task is added or the architecture of an existing task is optimized, NAS will automatically generate the corresponding architecture, which will be input into the multi-task learning component for training;

[0082] By applying the uncertainty weighting method, the weight of the loss function is dynamically adjusted according to the uncertainty of each task. The uncertainty weighting method enables the system to dynamically adjust training resources according to the learning difficulty and uncertainty of each task, ensuring the rational allocation of resources. This method is particularly suitable for multi-task learning scenarios with uneven task difficulty or uneven data quality, and can improve the overall efficiency and effectiveness of multi-task learning.

[0083] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the multi-task learning component further comprises:

[0084] The uncertainty-weighted loss function includes the following formula:

[0085]

[0086] Among them, L i is the loss function of the ith task, σ iis the model uncertainty parameter of the i-th task. During implementation, the system calculates and applies the uncertainty weighted loss function through the following steps:

[0087] Task loss calculation: In each training step, the system first calculates the loss L of all tasks i , and store these loss values.

[0088] Uncertainty parameter estimation: Based on the model’s prediction results for each task, estimate the uncertainty parameter σ for each task i , based on the model's predictive distribution, for example, obtained by computing the standard deviation or confidence interval of the model's output.

[0089] Weighted loss calculation: Use the above uncertainty weighted loss function formula to calculate the total loss L i ,in, As a form of weights, tasks with higher uncertainty account for a larger proportion of the total loss, thereby increasing attention to these tasks.

[0090] Gradient update: Perform gradient backpropagation based on the weighted total loss L and update the model parameters to optimize the performance of all tasks.

[0091] By dynamically adjusting the loss weights, the uncertainty weighting method enables the system to utilize training resources more effectively, improving the overall efficiency of multi-task learning. By focusing on high-uncertainty tasks, the model can better learn task features and improve its generalization ability in new scenarios and unseen data.

[0092] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the meta-learning component comprises:

[0093] The model parameter initialization is optimized through a model-independent meta-learning algorithm. During the implementation process, the meta-learning component first randomly initializes the model parameters, and then iteratively trains on multiple tasks to optimize these initial parameters. This training method enables the initialization parameters to show better adaptability and flexibility on new tasks. By optimizing the model parameter initialization, the meta-learning component improves the system's ability to quickly adapt to new tasks. This design ensures that meta-learning can quickly adapt to different tasks and optimize performance by sharing features;

[0094] In the inner loop, a small amount of gradient updates are performed for each task, so that the model generates task-related parameters;

[0095] Specifically:

[0096] Get task data: extract a small number of samples from the task dataset and prepare for fine-tuning;

[0097] Calculate task loss: Calculate the loss value L using the current initialization parameters i , that is, the performance of the model on this task.

[0098] Perform gradient updates: Calculate the gradient of the loss with respect to the model parameters through backpropagation and perform a small update. The update formula is:

[0099]

[0100] Among them, β is the learning rate in the inner loop. In this way, the model can generate specific parameters on each task, thereby improving the performance on that task.

[0101] In the outer loop, the initial parameters are globally updated based on the cumulative losses of all tasks, so that they can quickly adapt to new tasks through a small amount of gradient updates. The specific implementation process is:

[0102] Calculate the cumulative loss: After the inner loop is completed, calculate the cumulative loss of all tasks:

[0103]

[0104] Calculate the global gradient: Calculate the gradient of the initialization parameters based on the accumulated loss:

[0105]

[0106] Update initialization parameters: Update initialization parameters using the following formula:

[0107]

[0108] This method enables the initialization parameters to fully utilize the learning information of all tasks and achieve overall performance improvement.

[0109] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the meta-learning component further comprises:

[0110] The initialization parameters of the model in the outer loop are updated by the following formula:

[0111]

[0112] Among them, φ is the initialization parameter, α is the learning rate, It is task T i The loss function of , the implementation process includes the following steps:

[0113] Initialization parameters: At the beginning of meta-learning, the system randomly initializes the parameters φ.

[0114] Inner loop task training:

[0115] For each task T i , using the current initialization parameters φ to generate task-specific parameters φ′ i .

[0116] In Task T i Perform a small amount of gradient updates on the task to calculate the loss

[0117] Compute the gradient:

[0118] In the outer loop, the system aggregates the losses of all tasks and calculates the gradient of the total loss

[0119]

[0120] This gradient reflects the impact of the initialization parameter φ on the performance of all tasks.

[0121] Parameter update:

[0122] Use the above formula to update the initialization parameter φ to make it more suitable for the requirements of each task. This step ensures that the overall performance of the parameters on multiple tasks is optimized.

[0123] Iterative Optimization:

[0124] The above process is repeated until the preset performance standard or convergence condition is reached.

[0125] By optimizing the initialization parameters, the system improves its ability to quickly adapt to new tasks, enabling it to quickly adjust its behavior when faced with different tasks and achieve efficient learning. Through shared initialization parameters, the model not only performs well on known tasks, but can also better handle new unseen tasks, thereby improving the generalization ability of the model.

[0126] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the dynamic data enhancement component includes:

[0127] The conditional generative adversarial network is used to generate training samples that are semantically consistent with the original training data but diverse in form. CGAN consists of two parts: a generator and a discriminator. The goal of the generator is to generate realistic data samples that are semantically consistent with the real training data but diverse in form; while the discriminator tries to distinguish the generated samples from the real samples. The generator receives a noise signal and a conditional label as input and outputs the generated text data, while the discriminator evaluates the authenticity of the data. During the training process, the generator and the discriminator are continuously optimized in an adversarial framework. The generator tries to deceive the discriminator, while the discriminator strives to improve its ability to identify true and false data.

[0128] The generator is trained to generate data samples, and the discriminator is used to classify the generated data and real data. In CGAN, the training purpose of the generator is to generate samples that look like real data, which is achieved by minimizing the adversarial loss between the generator and the discriminator. The generator tries to generate samples that the discriminator can judge as real, while the task of the discriminator is to correctly distinguish between real data and generated data. In this way, the generator learns how to improve the samples it generates to make them less recognizable by the discriminator. At the same time, the training purpose of the discriminator is to improve its ability to accurately judge the authenticity of samples. The discriminator evaluates the authenticity of each sample it receives and continuously optimizes its parameters to better identify the pseudo samples generated by the generator.

[0129] The generated data is added to the model training set to enhance the generalization and robustness of the model. The purpose of this step is to increase the diversity and coverage of the training set, especially when the original data is insufficient or has large deviations. By introducing these diverse samples, the system can be trained on a wider data distribution, thereby improving the generalization and robustness of the model. By introducing the diverse data samples generated by CGAN, the dynamic data augmentation component improves the performance of the model in unseen data or new scenarios. The generated diverse data samples enable the model to learn a wider range of data features and patterns, enhance the model's adaptability and fault tolerance to changes in input data, and can quickly generate missing CGAN data types to help balance the data set, thereby improving the model's learning effect on minority data.

[0130] Please refer to the attached Figure 1 In a preferred embodiment of the present invention, the dynamic data enhancement component further comprises:

[0131] The generator and discriminator of the conditional generative adversarial network are trained with the following objective function:

[0132]

[0133] Where x is a real data sample, G(z) is a generated data sample, D(x) is the probability that the discriminator identifies the real data, and D(G(z)) is the probability that the discriminator identifies the generated data. In the implementation process, the training of CGAN includes the following steps:

[0134] Initialize the network: Initialize the parameters of the generator G and the discriminator D.

[0135] Train the discriminator:

[0136] Randomly extract a batch of real samples x from the real dataset.

[0137] From the noise z (z) Random noise z is extracted from the cloth.

[0138] Use generator G to generate a batch of samples G(z).

[0139] Compute the discriminator loss on real and generated samples:

[0140]

[0141] Update the parameters of the discriminator to make it more accurate in distinguishing real from generated samples.

[0142] Training the generator:

[0143] By fixing the parameters of the discriminator D, the generator G is trained to minimize the loss of generated samples:

[0144]

[0145] Update the parameters of the generator so that the data samples G(z) it generates are easier for the discriminator to identify them as real samples.

[0146] Iterative training: The discriminator and generator are trained alternately until the model converges and the quality of the generated data samples reaches the expected level. Through adversarial training, CGAN can generate diverse samples that are highly semantically consistent with real samples, enrich the training data set, and effectively enhance the learning ability of the model. The generated diverse training samples provide the model with more learning features, enhance the adaptability and robustness of the model when facing new data and complex scenarios, thereby improving the performance and stability of the model. At the same time, the samples generated by dynamic data enhancement will be directly used in the multi-task learning component as additional training data to enhance the model's learning ability on different tasks. The generated diverse data will help the meta-learning component better evaluate and optimize the initialization parameter θ, and improve adaptability through rich training samples.

[0147] Please refer to the attached Figure 1In a preferred embodiment of the present invention, the multi-task learning component is designed by separating the shared Transformer encoder layer and the task-specific adaptation layer, so that whenever a new task is added, the corresponding task-specific adaptation layer can be directly added without modifying the shared Transformer encoder layer. The shared encoder layer first receives the input text, performs standard self-attention and feedforward operations, and extracts high-level feature representations of the text. These feature representations capture the core semantics and structural information of the input data, providing a basis for processing a variety of downstream tasks. At the same time, in order to meet the specific needs of different tasks, the system configures a task-specific adaptation layer on top of the shared encoder layer. Each adaptation layer operates independently on the features output by the shared layer to perform further specific task processing. This design allows the task-specific adaptation layer to be quickly added or modified without affecting the structure and training of the shared encoder layer, thereby improving the overall training and reasoning efficiency and saving computing resources and training time.

[0148] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Transformer-based natural language processing system, characterized by: include: A neural architecture search component for automatically searching and optimizing Transformer model architectures for natural language processing; A multi-task learning component for simultaneously training multiple natural language processing tasks based on shared model layers; A meta-learning component that optimizes the model’s parameter initialization so that it can quickly adapt to new tasks with a small amount of data and training steps; Dynamic data augmentation component, used to generate diverse training data and enhance the generalization and robustness of the model.

2. The Transformer-based natural language processing system according to claim 1, characterized in that: The neural architecture search component includes: Define searchable architecture parameters, including the type of self-attention mechanism, the number of Transformer layers, the number of attention heads, the width of the feedforward network, and the type of activation function; Use the DARTS algorithm to explore the optimal Transformer architecture in the search space in a continuously differentiable manner; The controller is optimized via a policy gradient method in reinforcement learning, which produces an optimal architecture for a natural language processing task.

3. The Transformer-based natural language processing system according to claim 2, characterized in that: The neural architecture search component further comprises: The controller parameters θ are optimized by the policy gradient method using the following formula: Among them, θ: controller parameters, representing the strategy of generating network architecture, α: learning rate, controlling the amplitude of each parameter update, affecting the convergence speed and stability, Represents the architecture set, including all candidate architectures a, R(a): the performance score of architecture a on the validation set, usually the accuracy or loss of the model on a specific task, reflecting the effectiveness of the architecture; Represents the gradient of the controller parameter θ, pointing to the direction where the loss function rises fastest; P(a|θ): Given the parameter θ, the probability distribution of the generated architecture a is usually generated using the softmax function: Here, f(a;θ) is the score function associated with architecture a, which characterizes the quality of architecture generation.

4. The Transformer-based natural language processing system according to claim 1, characterized in that: The multi-task learning component includes: Set up a shared encoder layer in the underlying architecture of Transformer for feature representation of multiple natural language processing tasks; Setting task-specific adaptation layers on each task enables the model to handle the special requirements of each task; An uncertainty weighted method is applied to dynamically adjust the weight of the loss function according to the uncertainty of each task.

5. The Transformer-based natural language processing system according to claim 4, characterized in that: The multi-task learning component further comprises: The uncertainty-weighted loss function includes the following formula: Among them, L i is the loss function of the ith task, σ i is the model uncertainty parameter for the ith task.

6. The Transformer-based natural language processing system according to claim 1, characterized in that: The meta-learning components include: Optimize the model's parameter initialization through a model-independent meta-learning algorithm; In the inner loop, a small amount of gradient updates are performed for each task, so that the model generates task-related parameters; In the outer loop, the initial parameters are globally updated based on the accumulated losses of all tasks, allowing them to quickly adapt to new tasks with a small number of gradient updates.

7. The Transformer-based natural language processing system according to claim 6, characterized in that: The meta-learning component further comprises: The initialization parameters of the model in the outer loop are updated by the following formula: Among them, φ is the initialization parameter, α is the learning rate, It's a task The loss function of .

8. The Transformer-based natural language processing system according to claim 1, characterized in that: The dynamic data enhancement component includes: A conditional generative adversarial network is used to generate training samples that are semantically consistent with the original training data but in a variety of forms; Generate realistic data samples by training the generator and classify the generated data and real data through the discriminator; The generated data is added to the model training set to enhance the generalization and robustness of the model.

9. The Transformer-based natural language processing system according to claim 8, characterized in that: The dynamic data enhancement component further includes: The generator and discriminator of the conditional generative adversarial network are trained with the following objective function: Where x is a real data sample, G(z) is a generated data sample, D(x) is the probability that the discriminator identifies the real data, and D(G(z)) is the probability that the discriminator identifies the generated data.

10. The Transformer-based natural language processing system according to claim 1, characterized in that: The multi-task learning component is designed by separating the shared Transformer encoder layer from the task-specific adaptation layer, so that whenever a new task is added, the corresponding task-specific adaptation layer can be directly added without modifying the shared Transformer encoder layer.

Images