Transform model training method and device, electronic equipment and readable storage medium

Abstract

The invention provides a Transform model training method and device, electronic equipment and a readable storage medium, and relates to the technical field of machine learning, and the method comprises the steps: dividing an attention head in each layer of an original Transform model into a self-attention head and a cross-attention head, carrying out the feature extraction of a target domain text through a vector model consistent with the original Transform model in structure, and carrying out the feature extraction of a target domain text; the method comprises the steps of obtaining a target domain text feature vector, introducing the target domain text feature vector into a cross-attention head for feature fusion to obtain an intermediate Transform model, and training the intermediate Transform model by using a target domain text to obtain a target Transform model; according to the method, a text migration task of a large-scale pre-training language model from a source domain to a target domain can be realized in a low-computing-resource scene in a high-quality manner, so that the demand on text training data of the target domain during domain migration is remarkably reduced, and the computing resource cost is reduced; and the adaptability of a large-scale pre-training language model to identify texts in different fields is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a training method, device, electronic device and readable storage medium for a Transformer model. Background Art

[0002] In recent years, large-scale pre-trained language models based on the Transformer architecture (such as BERT, GPT, etc.) have made significant progress in the field of natural language processing (NLP). These models can capture rich language features and knowledge by pre-training on large-scale general corpora. However, when these models need to be adapted to specific fields, such as medicine, law, finance, etc., traditional methods usually rely on a large amount of parallel corpora in the field for fine-tuning, which brings the following problems: Difficulty in data acquisition: corpus data in many professional fields is scarce and difficult to obtain. For example, texts in the medical field may involve patient privacy, texts in the legal field may be protected by copyright, and texts in the financial field may contain sensitive business information; therefore, obtaining sufficient in-field data for fine-tuning is a major challenge; High computational cost: Fine-tuning large-scale pre-trained models requires a lot of computing resources, especially when dealing with large-scale domain data; Domain adaptability problem: Even with sufficient domain data, the model may overfit to the data in a specific domain during fine-tuning, resulting in decreased performance on general tasks. This domain adaptability problem limits the model's ability to generalize between different tasks. Summary of the invention

[0003] The present invention provides a training method, device, electronic device and readable storage medium of a Transformer model, which are used to solve the problems of difficulty in data acquisition, high computational cost and poor domain adaptability of existing pre-trained language models for domain migration, and efficiently realize the domain migration task of large-scale pre-trained language models in low computing resource scenarios.

[0004] The present invention provides a training method for a Transformer model, comprising: dividing the attention heads in each layer of an original Transformer model into self-attention heads and cross-attention heads; the original Transformer model is obtained based on source domain text training; feature extraction is performed on a target domain text to obtain a target domain text feature vector; the target domain text feature vector is introduced into the cross-attention head for feature fusion to obtain an intermediate Transformer model; and the target domain text is used to train the intermediate Transformer model to obtain a target Transformer model.

[0005] Optionally, using the target domain text to train the intermediate Transformer model to obtain the target Transformer model includes: freezing all parameters except the LoRA parameters in the cross-attention heads of the intermediate Transformer model, and training the LoRA parameters using the target domain text; the LoRA parameters are adjustment parameters introduced by low-rank decomposition; unfreezing the self-attention parameters of the specified key layer of the intermediate Transformer model for fine-tuning to obtain the target Transformer model.

[0006] Optionally, the unfreezing and fine-tuning the self-attention parameters of the specified key layers of the intermediate Transformer model to obtain the target Transformer model includes: unfreezing and fine-tuning the self-attention parameters of the last three Transformer layers of the intermediate Transformer model to obtain the target Transformer model.

[0007] Optionally, the feature extraction of the target domain text to obtain the target domain text feature vector includes: using a vector model consistent with the original Transformer model structure to vectorize the target domain text separately to obtain multiple text vectors; calculating the arithmetic mean of the multiple text vectors to obtain a target domain average feature vector; and normalizing the target domain average feature vector to obtain a target domain text feature vector.

[0008] Optionally, the calculation formula for the cross-attention head is:

[0009] in, is the cross-attention head, Q is the query matrix of the input sequence, K is the key matrix of the input sequence, V is the value matrix of the input sequence, is the average eigenvector of the input sequence, is the dimension of the self-attention head, is the key matrix K The transposed matrix of is the average eigenvector after normalization calculation, for The transposed matrix of .

[0010] Optionally, the original Transformer model uses LLaMA-7B as a base model.

[0011] Optionally, dividing the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads includes: dividing 32 attention heads in each layer of the original Transformer model into 24 self-attention heads and 8 cross-attention heads.

[0012] Optionally, dividing the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads includes: adjusting the ratio of the number of the self-attention heads to the cross-attention heads based on actual task requirements.

[0013] The present invention also provides a training device for a Transformer model, comprising the following modules: A processing module is configured to divide the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads; the original Transformer model is trained based on the source domain text; A feature extraction module is configured to extract features from the target domain text to obtain a target domain text feature vector; A feature fusion module is configured to introduce the target domain text feature vector into the cross-attention head to perform feature fusion to obtain an intermediate Transformer model; The training module is configured to train the intermediate Transformer model using the target domain text to obtain a target Transformer model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a training method for a Transformer model as described in any one of the above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a Transformer model as described in any one of the above.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the training method of the Transformer model as described in any one of the above.

[0017] The training method, device, electronic device and readable storage medium of the Transformer model provided by the present invention divide the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads, introduce the target domain text feature vectors into the cross-attention heads for feature fusion, and use the target domain text to train the original Transformer model to obtain the target Transformer model. In low computing resource scenarios, it is possible to achieve high-quality text migration tasks from the source domain to the target domain for large-scale pre-trained language models, significantly reduce the demand for target domain text training data, reduce computing resource costs, and improve the adaptability of large-scale pre-trained language models to recognize texts in different domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a training method for a Transformer model provided by the present invention.

[0020] Figure 2 A schematic flow chart of the sub-steps of step 102 provided by the present invention.

[0021] Figure 3 A schematic diagram of the structure of a training device for a Transformer model provided by the present invention.

[0022] Figure 4 An example physical structure diagram of an electronic device is provided.

[0023] Reference numerals: Transformer model training device 30; processing module 301; feature extraction module 302; feature fusion module 303; training module 304; processor 410; communication interface 420; memory 430; communication bus 440. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The adaptation of large-scale pre-trained language models from the source domain to the target domain refers to adjusting or optimizing a language model pre-trained on a large amount of general data to improve its performance in a specific target domain, such as medicine, law, finance, etc. The adaptation methods of large-scale pre-trained language models from the source domain to the target domain mainly include the following categories: The full fine-tuning method directly uses the target domain text data to fully fine-tune the pre-trained model. This method requires a large amount of target domain text data and computing resources, and is prone to catastrophic forgetting. The prompt learning method guides the pre-trained model to adapt to the target domain by designing specific prompt templates. Although this method is parameter-efficient, its effect often depends on the quality of the manually designed prompts. The adapter method inserts a small adapter module into the original pre-trained model. Although it reduces the number of parameters, it fails to fully utilize the feature extraction capabilities of the pre-trained model.

[0026] The above adaptation method has the following main problems: It has strong data dependence and requires a large amount of data in the target text domain to achieve good adaptation effects. It consumes a lot of computing resources and the full fine-tuning method needs to update all parameters of the pre-trained language model, which has high computational costs. The feature fusion is insufficient and existing methods often treat domain adaptation as an independent module and fail to deeply integrate it with the core attention mechanism of the pre-trained language model. The migration effect is unstable and in low computing resource scenarios, the domain migration effect of existing methods is often not ideal.

[0027] These problems seriously restrict the widespread application of pre-trained language models in professional fields. Therefore, the present invention proposes a training method for a Transformer model that can efficiently achieve domain migration, requires a small amount of data, has a stable migration effect, and consumes low computing resources.

[0028] Among them, the Transformer model is a deep learning architecture based on the self-attention mechanism, which is widely used in natural language processing (NLP) tasks such as machine translation, text generation and classification; the Transformer model completely relies on the self-attention mechanism to capture the relationship between different positions in the input sequence, including input embedding, positional encoding, encoder, decoder and output layer, and also includes multi-head attention. By calculating multiple attention heads in parallel, the model can extract information from different subspaces and enhance its expression ability.

[0029] Figure 1 A flowchart of a training method for a Transformer model provided by the present invention, such as Figure 1 As shown, the training method of the Transformer model is used in devices such as servers, desktops, and laptops, and includes the following steps.

[0030] In step 101, the attention heads in each layer of the original Transformer model are divided into self-attention heads and cross-attention heads.

[0031] Among them, the original Transformer model is trained based on the source domain text, which can be a large-scale general corpus text; the corresponding is the target domain text, which is a text different from the initial domain. For example, if the source domain text is a general corpus text, then the target domain text can be text in the fields of medicine, law, finance, etc.

[0032] One of the inventive points of the present invention is to improve the structural design of the attention mechanism of the original Transformer model, that is, in each layer of the original Transformer model, the total H attention heads are divided into H1 self-attention heads and H2 cross-attention heads, where H = H1 + H2. The self-attention head maintains the original intra-sequence attention calculation mechanism and adopts the standard attention calculation formula. The calculation formula of the self-attention head is:

[0033] in, is the self-attention head, Q is the query matrix of the input sequence, K is the key matrix of the input sequence, V is the value matrix of the input sequence, is the dimension of the self-attention head, is the key matrix K The transposed matrix of is the activation function; the self-attention head is responsible for maintaining the basic understanding of the input sequence of the Transformer model.

[0034] The calculation formula across attention heads is:

[0035] in, For cross-attention heads, Q is the query matrix of the input sequence, K is the key matrix of the input sequence, V is the value matrix of the input sequence, is the average eigenvector of the input sequence, is the dimension of the self-attention head, is the key matrix K The transposed matrix of is the average eigenvector after normalization calculation, for The transposed matrix of .

[0036] In one implementation, the original Transformer model uses LLaMA-7B as a base model; LLaMA-7B uses 32 Transformer layers, each layer contains 32 attention heads, the hidden dimension of LLaMA-7B is 4096, and Rotary Position Embedding (RoPE) is used as the position encoding scheme of the text, and the Root Mean Square Layer Normalization (RMSNorm) method is used for normalization.

[0037] Among them, LLaMA-7B is an open source Large Language Model (LLM), which belongs to the LLaMA (Large Language Model Meta AI) series of models. The goal of the LLaMA series of models is to achieve or exceed the performance of existing large models through more efficient training methods and smaller model sizes. LLaMA-7B is a version with 7 billion parameters. Compared with larger models (such as LLaMA-13B, LLaMA-30B and LLaMA-65B), LLaMA-7B is lighter in computing resource requirements while still maintaining strong language understanding and generation capabilities.

[0038] In the present invention, the original Transformer model using LLaMA-7B as the basic model is taken as an example for explanation.

[0039] For example, the 32 attention heads of each layer of LLaMA-7B can be divided into 24 self-attention heads and 8 cross-attention heads.

[0040] In one implementation, the self-attention heads and cross-attention heads may be divided based on either of the following two methods: Task-driven allocation: Depending on the specific task requirements, some heads are designed as self-attention heads and some as cross-attention heads; for example, in machine translation, cross-attention heads specifically process the relationship between source domain text and target domain text.

[0041] Dynamic allocation: In more complex language models, the functionality of attention heads can be adjusted dynamically, automatically deciding which heads to use for self-attention and which to use for cross-attention based on the input data.

[0042] For example, the ratio of the number of self-attention heads to the number of cross-attention heads can be dynamically adjusted based on actual task requirements, and the present invention is not limited to this.

[0043] By dividing the H attention heads into H1 self-attention heads and H2 cross-attention heads, the Transformer model can simultaneously capture the semantic relations within the sequence and the interaction information between sequences; this design is particularly effective in domain transfer tasks because the model can better understand the relationship between the source domain and the target domain, thereby achieving a more natural domain transfer.

[0044] It should be noted that RoPE is a position encoding method that encodes the position information of tokens in the form of rotation matrices, so that LLaMA-7B can better understand the relative position relationship of tokens in the input sequence. A token refers to a word or part of a word in the input sequence text. The calculation formula of RoPE is:

[0045] Where θ is the rotation angle related to the position, q is the query matrix of the input sequence text, k is the key matrix of the input sequence text, is the query matrix after the rotation matrix is ​​calculated, is the bond matrix after the rotation matrix is ​​calculated.

[0046] RoPE can better handle sequence lengths that exceed those processed during training, has better computational efficiency, and provides a flexible and efficient way to encode position information. It is especially suitable for scenarios that need to process long sequences or have strict constraints on computing resources.

[0047] In step 102, feature extraction is performed on the target domain text to obtain a target domain text feature vector.

[0048] The target domain text is the text corresponding to the source domain text. The target domain text is a text different from the initial domain. For example, if the source domain text is a general corpus text, then the target domain text can be a text in the fields of medicine, law, finance, etc.

[0049] It should be noted that step 102 also includes sub-steps 1021, 1022 and 1023. The specific method of determining the target domain text feature vector will be described in detail in the sub-steps of step 102. Figure 2 , Figure 2 A schematic flow chart of the sub-steps of step 102 provided by the present invention.

[0050] Sub-step 1021, using a vector model consistent with the original Transformer model structure to vectorize the target domain texts respectively, to obtain multiple text vectors.

[0051] Exemplarily, the vector model consistent with the original Transformer model structure may be a vector model built into LLaMA-7B, and the vector model built into LLaMA-7B is used as a vectorization tool to vectorize the target domain text to ensure consistency during training.

[0052] For each target domain text in the target domain corpus , calculate its vector representation:

[0053] in, E(x) is the vectorization function of the vector model, is the vector representation of the target domain text; to ensure the quality of the vector representation of the target domain text, the following preprocessing is performed before vectorization: Segment the target domain text: segment the long text into natural paragraphs. For example, you can identify the paragraph boundaries based on punctuation marks; or you can use blank lines as the basis for paragraph separation; Remove special formats from target domain text: clean up HyperText Markup Language (HTML) tags, special symbols, etc. For example, you can use regular expressions or a specialized HTML parsing library to remove HTML tags from the text.

[0054] After the target domain texts are vectorized separately using the built-in vector model of LLaMA-7B, multiple text vectors are obtained.

[0055] Sub-step 1022, calculating the arithmetic mean of multiple text vectors to obtain the target domain average feature vector.

[0056] In this step, the formula for calculating the arithmetic mean of multiple text vectors is:

[0057] in, is the average feature vector of the target domain, is the vector representation of the target domain text, N for The number of

[0058] Calculating the arithmetic mean of multiple target domain text vectors can effectively capture the global characteristics of the target domain; the averaging operation can smooth out the noise or specificity in a single text and retain the common characteristics of the target domain; the vector after arithmetic averaging can more stably represent the overall semantic distribution of the target domain and reduce the impact of deviations or outliers in a single text on domain characteristics.

[0059] Sub-step 1023, normalizing the target domain average feature vector to obtain the target domain text feature vector.

[0060] In order to maintain the accuracy of the numerical values ​​during the calculation process and avoid deviation or failure of the calculation results caused by extreme values, L2 normalization is used to normalize the average feature vector of the target domain obtained. The calculation formula is as follows:

[0061] in, is the average feature vector of the target domain after normalization, that is, the text feature vector of the target domain; is the average feature vector of the target domain.

[0062] The normalized target domain text feature vector can provide the model with a clear representation of the target domain features; this representation can be used as a reference for domain migration tasks to help the model better adapt to the semantics and distribution of the target domain. In domain migration tasks, the distribution difference between the source domain and the target domain is a key challenge. By introducing the target domain text feature vector, the model can more effectively align the feature space of the source domain and the target domain, thereby reducing the negative impact of domain differences.

[0063] In step 103, the target domain text feature vector is introduced into the cross-attention head for feature fusion to obtain an intermediate Transformer model.

[0064] For example, the 32 attention heads of each layer of LLaMA-7B can be divided into 24 self-attention heads and 8 cross-attention heads, which is explained below using this example.

[0065] The self-attention head maintains the original rotation position encoding and attention calculation mechanism of LLaMA-7B. The original rotation position encoding of LLaMA-7B is RoPE. The calculation formula of the self-attention head is as follows:

[0066]

[0067] in, RoPE () is the position encoding function, is the query matrix encoded by the RoPE method, is the key matrix encoded by RoPE method, V is the value matrix of the target domain text sequence; is the self-attention head, is the activation function, is the dimension of the self-attention head ， for The transposed matrix of .

[0068] For cross-attention heads, on the basis of maintaining the RoPE position encoding, the target domain text feature vector is introduced for interaction. The calculation formula for cross-attention heads is:

[0069] in, For cross-attention heads, is the query matrix encoded by the RoPE method, is the key matrix encoded by RoPE method, V is the value matrix of the target domain text sequence, is the average feature vector of the target domain text sequence, is the activation function, is the average eigenvector after normalization calculation, for The transposed matrix of .

[0070] By introducing the target domain text feature vector in the cross-attention head, the model can explicitly use the semantic information of the target domain to guide feature fusion. This guidance can help the model better adapt to the distribution of the target domain, thereby improving the effect of domain transfer; feature fusion in the cross-attention head can effectively align the feature space of the source domain and the target domain, reducing the negative impact of domain differences on model performance; by fusing the target domain feature vector in the cross-attention head, the model can effectively transfer the knowledge of the source domain to the target domain, thereby improving the generalization ability of the model in the target domain.

[0071] In step 104, the target domain text is used to train the intermediate Transformer model to obtain the target Transformer model.

[0072] In this step, the target domain text is used to train the intermediate Transformer model to obtain the target Transformer model.

[0073] In terms of parameter optimization, the QLoRA (Quantized Low-Rank Adaptation) method can be used for efficient fine-tuning. First, the intermediate Transformer model weights obtained in the previous steps are quantized to 4-bit precision, and then a low-rank adapter is added based on the quantized weights:

[0074] in, is the weight matrix after adding the low-rank adapter on the basis of the quantized weights, is the original weight after 4-bit quantization, B and A are low-rank matrices with 16-bit precision, and the rank is set to 64. Fine-tuning by the QLoRA method significantly reduces the computational cost and memory usage while maintaining high performance.

[0075] It should be noted that B and A are low-rank adapters with complementary dimensions. A is a dimensionality reduction matrix that compresses the original weight dimension into a low-rank space; B is a dimensionality increase matrix that maps the features of the low-rank space back to the original dimension; through the synergy of the two low-rank matrices A and B, efficient parameter updates can be achieved while maintaining model performance.

[0076] In terms of training strategy, for example, a phased training program can be adopted: Phase 1: Freeze all parameters except the Low-Rank Adaptation (LoRA) parameters in the cross-attention head, and train the LoRA parameters using the target domain text; the LoRA parameters are adjustment parameters introduced by low-rank decomposition; The second stage: unfreeze the self-attention parameters of the specified key layers of the intermediate Transformer model and fine-tune them to obtain the target Transformer model.

[0077] The AdaFactor optimizer can be used, the base learning rate is 1e-4, and the cosine learning rate scheduling strategy is adopted.

[0078] In one implementation, the self-attention parameters of the last three layers of the intermediate Transformer model can be unfrozen for fine-tuning to obtain the target Transformer model. The last three layers of the Transformer layer are close to the output end and are responsible for integrating high-level semantic features. After unfreezing, they can more flexibly adapt to downstream recognition tasks.

[0079] The target Transformer model can be used to identify text data in the target domain. It has high recognition accuracy and good migration stability. The target Transformer model does not require a large amount of data in the target text domain to achieve good adaptation effects, consumes less computing resources, and has low computing costs. It can fully integrate the features of the target domain text and deeply integrate the features of the target domain text with the core attention mechanism of the pre-trained language model. The migration effect is stable, and efficient domain migration effects can be achieved in low computing resource scenarios.

[0080] The training method, device, electronic device and readable storage medium of the Transformer model provided by the present invention divide the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads, introduce the target domain text feature vectors into the cross-attention heads for feature fusion, and use the target domain text to train the original Transformer model to obtain the target Transformer model. In the scenario of low computing resources, it is possible to achieve the text migration task of a large-scale pre-trained language model from a source domain to a target domain with high quality, significantly reduce the demand for target domain text training data during domain migration, reduce computing resource costs, and improve the adaptability of large-scale pre-trained language models to recognize texts in different domains.

[0081] The training device of the Transformer model provided by the present invention is described below. The training device of the Transformer model described below and the training method of the Transformer model described above can refer to each other.

[0082] Figure 3 The structure diagram of a training device for a Transformer model provided by the present invention is shown in FIG. The training device for the Transformer model is applied to devices such as servers, desktop computers, and notebook computers. Figure 3 The training device 30 of the Transformer model includes a processing module 301, a feature extraction module 302, a feature fusion module 303 and a training module 304.

[0083] The processing module 301 is configured to divide the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads; the original Transformer model is trained based on the source domain text; The feature extraction module 302 is configured to extract features from the target domain text to obtain a target domain text feature vector; A feature fusion module 303 is configured to introduce the target domain text feature vector into the cross-attention head to perform feature fusion to obtain an intermediate Transformer model; The training module 304 is configured to train the intermediate Transformer model using the target domain text to obtain a target Transformer model.

[0084] Optionally, the training module 304 is further configured to freeze all parameters except the LoRA parameters in the cross-attention head of the intermediate Transformer model, and train the LoRA parameters using the target domain text; the LoRA parameters are adjustment parameters introduced by low-rank decomposition; The self-attention parameters of the specified key layers of the intermediate Transformer model are unfrozen and fine-tuned to obtain the target Transformer model.

[0085] Optionally, the training module 304 is also configured to unfreeze the self-attention parameters of the last three Transformer layers of the intermediate Transformer model and perform fine-tuning to obtain the target Transformer model.

[0086] Optionally, the feature extraction module 302 is further configured to use a vector model consistent with the structure of the original Transformer model to vectorize the target domain texts respectively to obtain multiple text vectors; Calculating the arithmetic mean of the multiple text vectors to obtain an average feature vector of the target domain; The target domain average feature vector is normalized to obtain the target domain text feature vector.

[0087] Optionally, the calculation formula for the cross-attention head is:

[0088] in, is the cross-attention head, Q is the query matrix of the input sequence, K is the key matrix of the input sequence, V is the value matrix of the input sequence, is the average eigenvector of the input sequence, is the dimension of the self-attention head, is the key matrix K The transposed matrix of is the average eigenvector after normalization calculation, for The transposed matrix of .

[0089] Optionally, the original Transformer model uses LLaMA-7B as a base model.

[0090] Optionally, the processing module 301 is also configured to divide the 32 attention heads of each layer of the original Transformer model into 24 self-attention heads and 8 cross-attention heads.

[0091] Optionally, the processing module 301 is further configured to adjust the ratio of the number of the self-attention heads to the number of the cross-attention heads based on actual task requirements.

[0092] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the training method of the Transformer model, which includes: dividing the attention head in each layer of the original Transformer model into a self-attention head and a cross-attention head; the original Transformer model is obtained based on the source domain text training; feature extraction is performed on the target domain text to obtain the target domain text feature vector; the target domain text feature vector is introduced into the cross-attention head for feature fusion to obtain an intermediate Transformer model; the target domain text is used to train the intermediate Transformer model to obtain the target Transformer model.

[0093] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the Transformer model provided by the above methods, which method includes: dividing the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads; the original Transformer model is obtained based on source domain text training; feature extraction is performed on the target domain text to obtain a target domain text feature vector; the target domain text feature vector is introduced into the cross-attention head for feature fusion to obtain an intermediate Transformer model; the target domain text is used to train the intermediate Transformer model to obtain a target Transformer model.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training method of the Transformer model provided by the above-mentioned methods, the method comprising: dividing the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads; the original Transformer model is obtained based on source domain text training; feature extraction is performed on the target domain text to obtain a target domain text feature vector; the target domain text feature vector is introduced into the cross-attention head for feature fusion to obtain an intermediate Transformer model; the target domain text is used to train the intermediate Transformer model to obtain a target Transformer model.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a Transformer model, characterized in that: include: The attention heads in each layer of the original Transformer model are divided into self-attention heads and cross-attention heads; The original Transformer model is trained based on source domain text; Extract features from the target domain text to obtain the target domain text feature vector; Introducing the target domain text feature vector into the cross-attention head to perform feature fusion to obtain an intermediate Transformer model; The target domain text is used to train the intermediate Transformer model to obtain a target Transformer model.

2. The method according to claim 1, characterized in that The using the target domain text to train the intermediate Transformer model to obtain a target Transformer model includes: Freeze all parameters except the LoRA parameters in the cross-attention head of the intermediate Transformer model, and train the LoRA parameters using the target domain text; the LoRA parameters are adjustment parameters introduced by low-rank decomposition; The self-attention parameters of the specified key layers of the intermediate Transformer model are unfrozen and fine-tuned to obtain the target Transformer model.

3. The method according to claim 2, characterized in that The unfreezing of the self-attention parameters of the designated key layer of the intermediate Transformer model is fine-tuned to obtain the target Transformer model, including: The self-attention parameters of the last three Transformer layers of the intermediate Transformer model are unfrozen and fine-tuned to obtain the target Transformer model.

4. The method according to claim 1, characterized in that: The feature extraction of the target domain text to obtain the target domain text feature vector includes: Using a vector model consistent with the structure of the original Transformer model to vectorize the target domain texts respectively, to obtain multiple text vectors; Calculating the arithmetic mean of the multiple text vectors to obtain an average feature vector of the target domain; The target domain average feature vector is normalized to obtain the target domain text feature vector.

5. The method according to claim 1, characterized in that The calculation formula for the cross-attention head is: ； in, is the cross-attention head, Q is the query matrix of the input sequence, K is the key matrix of the input sequence, V is the value matrix of the input sequence, is the average eigenvector of the input sequence, is the dimension of the self-attention head, is the key matrix K The transposed matrix of is the average eigenvector after normalization calculation, for The transposed matrix of .

6. The method according to claim 1, characterized in that The original Transformer model uses LLaMA-7B as the base model.

7. The method according to claim 6, characterized in that The attention heads in each layer of the original Transformer model are divided into self-attention heads and cross-attention heads, including: The 32 attention heads of each layer of the original Transformer model are divided into 24 self-attention heads and 8 cross-attention heads.

8. The method according to claim 1, characterized in that The attention heads in each layer of the original Transformer model are divided into self-attention heads and cross-attention heads, including: The ratio of the number of the self-attention heads to the cross-attention heads is adjusted based on actual task requirements.

9. A training device for a Transformer model, characterized in that: include: A processing module configured to separate the attention heads in each layer of the original Transformer model into self-attention heads and cross-attention heads; The original Transformer model is trained based on source domain text; A feature extraction module is configured to extract features from the target domain text to obtain a target domain text feature vector; A feature fusion module is configured to introduce the target domain text feature vector into the cross-attention head to perform feature fusion to obtain an intermediate Transformer model; The training module is configured to train the intermediate Transformer model using the target domain text to obtain a target Transformer model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the training method of the Transformer model as described in any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method of the Transformer model as described in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the training method of the Transformer model as described in any one of claims 1 to 8 is implemented.

Images