A data processing method and related device

Abstract

A data processing method applied to a pre-training model, the method comprising: processing text data by a first pre-training language model (PLM) to obtain a target feature representation; wherein the first PLM comprises one or more first network layers, and the target feature representation is obtained according to a first feature representation output by the one or more first network layers; processing the text data by a second PLM; the second PLM comprises a first attention layer and a second network layer connected with the first attention layer; an input of the first attention layer comprises the target feature representation and a second feature representation output by the second network layer. The application constructs a general memory knowledge representation from a general pre-training language model, and then fuses it into a pre-training language model (second PLM) of a specific field through a memory enhancement layer (first attention layer), so that the pre-training language model of the specific field can obtain forgotten general field knowledge, and the problem of catastrophic forgetting in the pre-training process is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a data processing method and related equipment. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Domain transfer in pre-trained language models (PLMs) has attracted increasing research attention because domain differences between pre-training corpora and downstream tasks can lead to significant performance degradation. Catastrophic forgetting of general domain knowledge can occur after adaptive pre-training, resulting in poor performance on downstream tasks. Catastrophic forgetting is a common phenomenon in continuous learning, where a trained model forgets previously learned knowledge and overfits to a new task.

[0004] Existing implementations mitigate catastrophic forgetting through memory-based methods. Specifically, they store important samples from past tasks in external memory and refine them using gradient transformation strategies to reduce forgetting. However, these methods require external storage, consuming additional storage space to hold samples from past tasks. Summary of the Invention

[0005] This application provides a data processing method that reduces the catastrophic forgetting problem that occurs during pre-training.

[0006] Firstly, this application provides a data processing method, the method comprising: processing text data through a first pre-trained language model (PLM) to obtain a target feature representation; wherein the first PLM includes one or more first network layers, and the target feature representation is obtained based on a first feature representation output by the one or more first network layers; processing the text data through a second PLM; wherein the second PLM includes a first attention layer and a second network layer connected to the first attention layer; the input of the first attention layer includes the target feature representation and a second feature representation output by the second network layer. Through the above method, a general memory knowledge representation is effectively constructed from a general pre-trained language model (first PLM), and then fused into a domain-specific pre-trained language model (second PLM) through a memory enhancement layer (first attention layer), enabling the domain-specific pre-trained language model to acquire forgotten general domain knowledge, reducing the catastrophic forgetting problem that occurs during pre-training without requiring additional storage space.

[0007] In one possible implementation, the first PLM is trained based on multi-domain text data.

[0008] In one possible implementation, the first PLM is a model with fixed parameters.

[0009] In other words, the first PLM can be a general pre-trained language model with fixed parameters, meaning it is a model free from catastrophic forgetting. During model updates, the domain-specific pre-trained language model updates the gradient based on the loss function value. To maintain the invariance of general domain knowledge, the parameters of the general domain model are fixed, so the gradient of the general domain model does not change during training. Therefore, a pre-trained language model with fixed parameters is not affected by forgetting.

[0010] In one possible implementation, the step of processing text data through a first pre-trained language model (PLM) to obtain a target feature representation includes: processing text data through the first pre-trained language model (PLM) to obtain multiple first feature representations output by the first network layers; and fusing the multiple first feature representations output by the first network layers to obtain the target feature representation.

[0011] Specifically, the situations can be divided into the following categories:

[0012] Case 1: The feature representation output from one network layer in the first PLM can be input into an attention layer in the second PLM.

[0013] Scenario 2: The feature representations output by multiple network layers in the first PLM can be input into multiple attention layers in the second PLM, and the feature representation output by each network layer can be input into one attention layer in the second PLM. Here, the multiple network layers can be some of the network layers in the first PLM.

[0014] Scenario 3: The feature representations output by multiple network layers in the first PLM can be input into multiple attention layers in the second PLM, and the feature representation output by each network layer can be input into one attention layer in the second PLM. Here, the multiple network layers can be all network layers of the first PLM (all network layers can be understood as all network layers used to obtain the feature representation, excluding output layers).

[0015] Scenario 4: The feature representations output by multiple network layers in the first PLM can be fused, and the fused result can be input into an attention layer in the second PLM. The multiple network layers can be some of the network layers in the first PLM.

[0016] Scenario 5: The feature representations output by multiple network layers in the first PLM can be fused, and the fused result can be input into an attention layer in the second PLM. The feature representation output by each network layer can be input into an attention layer in the second PLM. Here, multiple network layers can be all network layers of the first PLM (the so-called all network layers can be understood as all network layers used to obtain feature representations, for example, excluding output layers).

[0017] Scenario 6: Each group of network layers in the first PLM can be fused (each group of network layers may include multiple network layers) to obtain multiple fusion results, and each of the multiple fusion results is input into an attention layer in the second PLM. The multiple groups of network layers may include some of the network layers in the first PLM.

[0018] Scenario 7: Each group of network layers in the first PLM can be fused (each group of network layers may include multiple network layers) to obtain multiple fusion results, and each of the multiple fusion results is input into an attention layer in the second PLM. The multiple groups of network layers can include all network layers of the first PLM (the so-called all network layers can be understood as all network layers used to obtain feature representations, for example, excluding the output layer).

[0019] Case 8: Each group of network layers in the first PLM can be fused (each group of network layers may include one or more network layers; if it includes only one network layer, the group of network layers does not need to be fused) to obtain multiple fusion results, and each of the multiple fusion results is input into an attention layer in the second PLM.

[0020] In one possible implementation, fusing the first feature representations output by the first network layers to obtain the target feature representation includes: obtaining the target feature representation by weighted summation based on the first feature representations output by multiple first network layers and the weights corresponding to each first network layer. The weights corresponding to each first network layer can be updated during training, and the output of the first network layer with larger weights can be considered more important.

[0021] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0022] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM. Alternatively, the layer closest to the top of the domain-specific PLM model can be chosen as the memory enhancement layer; this method has shown the best performance in experiments and does not require additional parameters.

[0023] In one possible implementation, processing the text data via a second PLM includes:

[0024] The first attention layer processes the target feature representation and the second feature representation; wherein, the first attention layer of the text is used to perform interaction between different embedding vectors in the target feature representation and the second feature representation to obtain attention information, and each embedding vector corresponds to a text unit in the text data.

[0025] In one possible implementation, the first attention layer is specifically used to: obtain a first Q matrix and a first V matrix based on the target feature representation, and obtain a first K matrix based on the second feature representation; and perform interaction between the first Q matrix, the first V matrix, and the first K matrix.

[0026] Since the input includes data from both the general PLM (i.e., the first PLM) and its own network layer (i.e., the second PLM), the difference between the memory-enhanced layer and the traditional self-attention layer lies only in the design of the multi-head self-attention module. This application proposes a novel memory-enhanced attention module that integrates general domain memory representations into a domain-specific pre-trained language model, represented as memory-attention. Specifically, the memory representations are linearly transformed into new value-key pairs and concatenated to the pairs generated by the domain-specific pre-trained language model. Multi-head self-attention is then performed to adaptively integrate this new concatenated representation. The entire process reuses the parameters of the transformation layer of the domain-specific pre-trained language model without introducing any new parameters.

[0027] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0028] In one possible implementation, after processing the text data through the second PLM, a processing result of the text data is obtained; the method further includes: updating the second PLM according to the processing result and the corresponding truth value, to obtain an updated second PLM.

[0029] Secondly, this application provides a data processing method, the method comprising:

[0030] Get text data;

[0031] The second PLM processes the text data to obtain the processing result corresponding to the text data; wherein, the second PLM includes a first attention layer and a second network layer connected to the first attention layer; when training the second PLM, the input of the first attention layer includes a second feature representation output by the second network layer and a target feature representation, wherein the target feature representation is obtained by the first PLM when processing the text data by one or more first network layers outputting a first feature representation.

[0032] In one possible implementation, the first PLM is trained based on multi-domain text data.

[0033] In one possible implementation, the target feature representation is specifically obtained by fusing the first feature representations output by multiple first network layers.

[0034] In one possible implementation, the target feature representation is specifically obtained by weighted summation of the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

[0035] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; when training the second PLM, the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0036] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM.

[0037] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0038] Thirdly, this application provides a data processing apparatus, the apparatus comprising:

[0039] The processing module is used to process text data through a first pre-trained language model (PLM) to obtain a target feature representation; wherein the first PLM includes one or more first network layers, and the target feature representation is obtained based on the first feature representation output by the one or more first network layers;

[0040] The text data is processed by a second PLM; wherein the second PLM includes a first attention layer and a second network layer connected to the first attention layer; the input of the first attention layer includes the target feature representation and the second feature representation output by the second network layer.

[0041] In one possible implementation, the first PLM is trained based on multi-domain text data.

[0042] In one possible implementation, the processing module is specifically used for:

[0043] The text data is processed by the first pre-trained language model PLM to obtain multiple first feature representations output by the first network layer.

[0044] The target feature representation is obtained by fusing the first feature representations output by multiple first network layers.

[0045] In one possible implementation, the processing module is specifically used for:

[0046] The target feature representation is obtained by weighted summation based on the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

[0047] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0048] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM.

[0049] In one possible implementation, the processing module is specifically used for:

[0050] The first attention layer processes the target feature representation and the second feature representation; wherein, the first attention layer of the text is used to perform interaction between different embedding vectors in the target feature representation and the second feature representation to obtain attention information, and each embedding vector corresponds to a text unit in the text data.

[0051] In one possible implementation, the first attention layer is specifically used for:

[0052] Based on the target feature representation, a first Q matrix and a first V matrix are obtained, and based on the second feature representation, a first K matrix is ​​obtained;

[0053] The interaction between the first Q matrix, the first V matrix, and the first K matrix is ​​performed.

[0054] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0055] In one possible implementation, after processing the text data via the second PLM, a processing result of the text data is obtained; the apparatus further includes:

[0056] The update module is used to update the second PLM based on the processing result and the corresponding truth value, so as to obtain the updated second PLM.

[0057] Fourthly, this application provides a data processing apparatus, the apparatus comprising:

[0058] The processing module is used to acquire text data;

[0059] The second PLM processes the text data to obtain the processing result corresponding to the text data; wherein, the second PLM includes a first attention layer and a second network layer connected to the first attention layer; when training the second PLM, the input of the first attention layer includes a second feature representation output by the second network layer and a target feature representation, wherein the target feature representation is obtained by the first PLM processing the text data by outputting feature representations from one or more first network layers.

[0060] In one possible implementation, the first PLM is trained based on multi-domain text data.

[0061] In one possible implementation, the target feature representation is specifically obtained by fusing the first feature representations output by multiple first network layers.

[0062] In one possible implementation, the target feature representation is specifically obtained by weighted summation of the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

[0063] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; when training the second PLM, the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0064] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM.

[0065] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0066] Fifthly, embodiments of this application provide an execution device that may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform steps related to model inference in any of the optional methods in the first and second aspects described above.

[0067] In a sixth aspect, embodiments of this application provide a training device that may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform steps related to model training as described in any of the optional methods in the first and second aspects above.

[0068] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional methods thereof, or the methods described in the second aspect and any optional methods thereof.

[0069] Eighthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional methods thereof, or the methods described in the second aspect and any optional methods thereof.

[0070] Ninthly, this application provides a chip system including a processor for supporting a training device or execution device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0071] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0072] Figure 2 It is a natural language processing system;

[0073] Figure 3a For another natural language processing system;

[0074] Figure 3b This is a schematic diagram of the system structure;

[0075] Figure 4 A schematic diagram of the natural language processing related devices provided in the embodiments of this application;

[0076] Figure 5 This is a schematic diagram of a transformer layer architecture;

[0077] Figure 6A This is an illustration of an embodiment of a data processing method provided in this application.

[0078] Figure 6B This is a schematic diagram of an embodiment of a model training method;

[0079] Figure 6C This is a schematic diagram of a transformer layer structure;

[0080] Figure 6D A schematic diagram of the operation of an attention head;

[0081] Figure 7 This is an illustration of an embodiment of a data processing method provided in this application.

[0082] Figure 8 This is an illustration of an embodiment of a data processing method provided in this application.

[0083] Figure 9 This is an illustration of an embodiment of a data processing method provided in this application.

[0084] Figure 10This is an illustration of an embodiment of a data processing method provided in this application.

[0085] Figure 11 This is an illustration of an embodiment of a data processing method provided in this application.

[0086] Figure 12 This is an illustration of an embodiment of a data processing method provided in this application.

[0087] Figure 13 A schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0088] Figure 14 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0089] Figure 15 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0090] Figure 16 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0091] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0092] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0093] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0094] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0095] (1) Infrastructure

[0096] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0097] (2) Data

[0098] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0099] (3) Data processing

[0100] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0101] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0102] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0103] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0104] (4) General ability

[0105] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0106] (5) Smart Products and Industry Applications

[0107] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0108] This application can be applied to the field of natural language processing in the field of artificial intelligence. The following will introduce several application scenarios that have been implemented in products, taking natural language processing as an example.

[0109] To better understand the solutions of the embodiments of this application, the following will first combine... Figures 2 to 3a A brief introduction to the possible application scenarios of the embodiments of this application is provided.

[0110] Figure 2 A natural language processing (NLP) system is illustrated, comprising user devices and data processing devices. The user devices include smart terminals such as mobile phones, personal computers, or information processing centers. The user devices are the initiators of natural language data processing, acting as the initiators of requests such as language question answering or queries; typically, users initiate requests through their user devices.

[0111] The aforementioned data processing equipment can be cloud servers, network servers, application servers, management servers, or other devices or servers with data processing capabilities. The data processing equipment receives queries / voice / text from smart terminals via an interactive interface, then performs language data processing through a storage device and a data processing processor, employing methods such as machine learning, deep learning, search, reasoning, and decision-making. The processing results are then fed back to the user device. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0112] exist Figure 2In the natural language processing system shown, the user device can receive instructions from the user. For example, the user device can receive a piece of text input by the user and then send a request to the data processing device, so that the data processing device can perform natural language processing applications (such as natural language generation, text classification, text reasoning, named entity recognition, translation, etc.) on the piece of text obtained by the user device, thereby obtaining the processing results of the corresponding natural language processing applications on the piece of text (such as word prediction results, classification results, reasoning results, named entity recognition results, translation results, etc.).

[0113] Taking natural language generation (NLP) as an example, NLP, also known as text prediction or natural language synthesis, refers to the task of generating missing or subsequent text given a given text. NLP is widely used in search engines, input methods, and other scenarios. It can predict a user's next input based on partial input, greatly improving the efficiency of using the product. Furthermore, it can recover missing text from existing text.

[0114] The generalizability of pre-trained models allows them to learn general language knowledge from large-scale corpora, making downstream tasks and applications related to the training corpora of pre-trained language models a focus of attention. Related tasks include General Language Understanding Evaluation (GLUE), Question Answering, Sentiment Analysis, Named Entity Recognition, Machine Translation, Text Summarization, and Named Entity Recognition.

[0115] General Language Understanding Evaluation (GLUE): The benchmark consists of nine natural language understanding task datasets, including single-sentence classification, text pair classification, text similarity, and ranking tasks. The latest improved version, SuperBLUE, includes a wider variety of task types compared to GLUE.

[0116] Question Answering (QA) is a sub-module of reading comprehension. From simple to difficult, QA can be categorized into three types: single-turn extractive question answering, multi-turn generative question answering, and multi-hop question answering. For extractive question answering tasks, researchers have proposed a retrospective reading model and used a pre-trained model to initialize the encoder. For multi-turn generative question answering tasks, researchers have proposed a model combining a pre-trained model, adversarial training, logical annotation, and knowledge distillation. For multi-hop question answering, researchers have proposed an interpretable system of "selection, answer, and explanation" using a pre-trained language model as the encoder.

[0117] Sentiment Analysis: Researchers achieved top results on the SST-2 sentiment classification dataset by fine-tuning pre-trained language models; however, direct fine-tuning on the more granular sentiment classification task ABSA generally yielded limited results. Some researchers transformed the ABSA task from single-sentence classification to sentence-pair classification; others achieved good results using incremental training of pre-trained language models in the ABSA domain.

[0118] Machine Translation: Machine translation is a crucial task in Natural Language Processing (NLP). Neural network-based translation models typically employ an encoder-decoder framework, where the input text is encoded into a hidden representation on the encoding side, and this hidden representation is decoded into the target language text on the decoding side. Researchers have attempted to initialize the encoder and decoder in NMT using the multilingual pre-trained language model BERT, achieving significant improvements in both unsupervised translation and English-Roman translation tasks.

[0119] Text summarization: The text summarization task involves generating a short text that represents the central meaning of a long text. The introduction of pre-trained models has significantly improved the performance of summarization tasks. Researchers have attempted to train pre-trained language models at the article level by directly predicting sentences and then directly apply them to text summarization tasks.

[0120] Named Entity Recognition (NER): The main purpose of this application is to identify entity words in a sentence and their corresponding entity types. The basic solution for NER is to treat it as a sequence classification task, typically using annotation prediction methods such as BIO and BIOES. The BIO annotation method assigns a label to each word in the sentence. This label consists of two parts: one part indicates the position of the word within the entity, where B indicates that the word is the first word of the entity, I indicates that the word is a middle word of the entity, and O indicates that it is not an entity. A common model is a pre-trained language model combined with a conditional random field algorithm.

[0121] The data processing methods described in this application can be applied to, but are not limited to, the application areas mentioned above.

[0122] For example, in this embodiment of the application, the user equipment can receive a piece of text data input by the user, wherein the text data includes known words and words to be predicted. The words to be predicted are not visible, only the position of the words to be predicted in the text data is known. Then the user equipment can send a request to the data processing device (the request carries text data), so that the data processing device can predict the words to be predicted in the text data, thereby obtaining the words to be predicted, and feeding back the words to be predicted to the user equipment.

[0123] For example, a user equipment can receive a piece of text data input by a user, and then send a request to a data processing device to enable the data processing device to perform entity classification on the piece of text data, thereby obtaining the entity classification result for the piece of text data, and feeding the entity classification result back to the user equipment.

[0124] For example, a user equipment can receive a piece of text data (the text data is Chinese text) input by a user, and then send a request to a data processing device to translate the text data into English, thereby obtaining an English translation of the text data, and then sending the English translation back to the user equipment.

[0125] Figure 3a This demonstrates another natural language processing system, in Figure 3a In this context, the user equipment (UE) directly functions as a data processing device. This UE can directly receive input from the user and process it directly through its own hardware. The specific process is similar to... Figure 2 Similar to the description above, it will not be repeated here.

[0126] Figure 4 This is a schematic diagram of the natural language processing related device 300 provided in the embodiments of this application.

[0127] The above Figure 2 and Figure 3a The user equipment in the context can specifically be Figure 4 Local device 301 or local device 302 in the system. Figure 2 The data processing equipment in the middle can specifically be Figure 4 The execution device 310 in the process includes a data storage system 350 that can store the data to be processed by the execution device 310. The data storage system 350 can be integrated into the execution device 310 or set up in the cloud or on other network servers.

[0128] Figure 2 and Figure 3a The processor in the system can be trained on data using neural network models or other models for machine learning / deep learning, and then use the models trained or learned from the data to perform natural language processing applications on text data (such as natural language generation, text classification, sequence labeling, reading comprehension, text generation, text reasoning, translation, etc.) to obtain the corresponding processing results.

[0129] The high-precision model, after fine-tuning the pre-trained language model in this embodiment, can be deployed in a data processing device. The data processing device can provide the high-precision model to process text data to obtain the processing results of the above-mentioned natural language processing application.

[0130] The following is combined Figure 3b The system architecture provided in the embodiments of this application will be described in detail. Figure 3b This is a schematic diagram of a system architecture provided for an embodiment of this application. For example... Figure 3b As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0131] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0132] The data acquisition device 560 is used to collect training data.

[0133] In the natural language synthesis task, the training data can be text data with missing text and the complete text data corresponding to the missing text data.

[0134] In translation tasks, training data can include, but is not limited to, parallel corpora and monolingual corpora.

[0135] Parallel corpora refer to bilingual or multilingual corpora (i.e., labeled text data) consisting of a source text and its corresponding translation text. The source and translation texts share the same semantics, and there is a correspondence between the text units. For example, if the source text is "This trip needs careful planning," and its corresponding English text is "The trip needs careful planning," then "This trip needs careful planning" and "The trip needs careful planning" can be considered a set of parallel corpora. This set of parallel corpora is a Chinese-English parallel language pair. The source text "This trip needs careful planning" can be considered the source corpus of this set of parallel corpora, and the translation text "The trip needs careful planning" can be considered the target corpus. Here, "trip" can be used as the equivalent of "travel."

[0136] In addition, "This trip needs careful planning" can be regarded as a monolingual corpus, and "The trip needs careful planning" can also be regarded as a monolingual corpus.

[0137] After collecting the training data, the data acquisition device 560 stores the training data in the database 530, and the training device 520 trains the target model / rule 501 based on the training data maintained in the database 530.

[0138] The training device 520 trains the pretrained language model (PLM) in this embodiment of the application based on the training data maintained in the database 530 to obtain the target model / rule 501.

[0139] In order to adapt to downstream tasks, the training device 520 can fine-tune the pre-trained language model based on the training data maintained in the database 530 to obtain the target model / rule 501.

[0140] It should be understood that the training device 520 for training the pre-trained language model can be a different device from the training device 520 for fine-tuning the pre-trained language model.

[0141] It should be noted that in practical applications, the training data maintained in database 530 may not all come from the data acquisition device 560; it may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training data maintained in database 530; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0142] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 3b The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud, etc. Figure 3b In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data to the I / O interface 512 through the client device 540.

[0143] Preprocessing modules 513 and 514 are used to preprocess the input data received from I / O interface 512 (e.g., obtaining the positions of known data units and data units to be predicted in the target data, or generating attention information, etc.). It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can directly process the input data.

[0144] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0145] Finally, the I / O interface 512 presents the processing results to the client device 540, thereby providing them to the user.

[0146] exist Figure 3bIn the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0147] It is worth noting that, Figure 3b This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3b In this context, the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 may also be placed within the execution device 510.

[0148] It should be understood that the aforementioned execution device 510 can also be deployed in customer device 540.

[0149] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0150] (1) Neural Network

[0151] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0152]

[0153] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0154] (2) Transformer layer

[0155] Reference Figure 5 , Figure 5 This is a schematic diagram of a transformer layer architecture, such as Figure 5 As shown, the neural network includes an embedding layer and at least one transformer layer. The at least one transformer layer can be N transformer layers (N being an integer greater than 0). Each transformer layer includes sequentially adjacent attention layers, add and normalize layers, feed-forward layers, and add and normalize layers. In the embedding layer, the current input is embedded to obtain multiple embedding vectors. In the attention layer, P input vectors are obtained from the layer above the first transformer layer. Using any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained. This process determines P intermediate vectors corresponding to the P input vectors. In the pooling layer, the P intermediate vectors are merged into Q output vectors, where the multiple output vectors obtained from the last transformer layer are used as feature representations of the current input.

[0156] (3) Attention mechanism

[0157] Attention mechanisms mimic the internal processes of biological observation—aligning internal experience with external senses to increase the precision of observation in specific areas. They enable the rapid sifting of high-value information from a large volume of data using limited attentional resources. Attention mechanisms can quickly extract important features from sparse data and are therefore widely used in natural language processing tasks, particularly machine translation. Self-attention mechanisms, an improvement on attention mechanisms, reduce reliance on external information and are better at capturing the internal correlations of data or features. The core idea of ​​attention mechanisms can be rewritten as follows:

[0158] In this formula, Lx = ||Source|| represents the length of the Source. The meaning is that the elements in the Source are imagined as a series of data pairs. Given a Query element in the Target, the similarity or relevance between the Query and each Key is calculated to obtain the weight coefficient of the Value corresponding to each Key. Then, the Values ​​are weighted and summed to obtain the final Attention value. Therefore, the Attention mechanism essentially performs a weighted sum of the Values ​​of the elements in the Source, while the Query and Key are used to calculate the weight coefficients of their corresponding Values. Conceptually, Attention can be understood as selectively filtering a small amount of important information from a large amount of information and focusing on this important information, ignoring most of the unimportant information. The focusing process is reflected in the calculation of the weight coefficients; the larger the weight, the more focused it is on its corresponding Value. That is, the weight represents the importance of the information, and the Value is the corresponding information. Self-attention can be understood as intra attention. The attention mechanism occurs between the elements of the Target (Query) and all elements of the Source. Self-attention refers to the attention mechanism that occurs between elements within the Source or between elements within the Target. It can also be understood as the attention calculation mechanism in the special case where Target = Source. The specific calculation process is the same, only the calculation object changes.

[0159] (4) Natural Language Processing (NLP)

[0160] Natural language is human language, and Natural Language Processing (NLP) is the processing of human language. NLP is a systematic process of analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. By using NLP and its components, we can manage very large amounts of text data, perform numerous automated tasks, and solve a wide variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering systems, and topic segmentation, among others.

[0161] (5) Pre-trained language model (PLM)

[0162] A pre-trained language model is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. Its training consists of two phases. In the pre-training phase, the model is trained on a large-scale unsupervised text environment to learn word representations. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase and then trained on downstream tasks such as text classification and sequence labeling with fewer steps, successfully transferring the semantic information obtained in pre-training to downstream tasks.

[0163] (6) Sequence-to-sequence natural language generation

[0164] Sequence-to-sequence natural language generation is a crucial area in natural language processing, often employing an encoder-decoder framework. Given a training instance (X, Y), where X is the source sequence and Y is the target sequence, during training, X is input to the encoder to generate a vector representation z. This representation z is then fed into the decoder via a cross-attention module, where it is decoded to generate the target sequence Y. Based on the method of generating the target sequence, sequence generation tasks can be categorized into autoregressive generation and non-autoregressive (parallel) generation. Autoregressive generation predicts the first character of the target sequence and then predicts the entire sequence step-by-step based on the already generated subsequences. Non-autoregressive generation, on the other hand, generates the complete target sequence in parallel during decoding, eliminating the need for iterative steps and significantly reducing the time required to generate the target sequence. For tasks with high real-time requirements, such as translation and dialogue, non-autoregressive generation becomes increasingly important.

[0165] (7) Pre-trained word vectors

[0166] For example, Word2Vec, CBOW, and GloVe, which train word vectors from unlabeled corpora, can be considered the earliest language representation models. Word vector models have a simple structure, but they can capture the syntactic and semantic information between words in the text.

[0167] (8) Pre-trained recurrent neural network encoding model

[0168] For example, LSTM and bidirectional LSTM, as encoders, can obtain context-sensitive embeddings. Using a sequence-to-sequence architecture, more efficient word vector representations can be obtained, leading to significant progress in downstream tasks such as machine translation. Based on bidirectional LSTM, researchers proposed the ELMO model, whose generated contextual semantic representations further improve downstream tasks. However, early pre-trained models based on recurrent neural networks often had fixed parameters, requiring the main model parameters to be retrained from scratch. Therefore, the ULMFiT model proposes an improvement: first, pre-training on a general corpus, and then fine-tuning on a target task-related dataset related to the pre-training corpus.

[0169] (9) Pre-trained language model based on self-attention architecture

[0170] With the introduction of self-attention architecture models (Transformer), pre-trained language models with more complex structures and more layers have received more research, such as GPT and BERT. These models typically use language model objective functions, language model objective functions with masking mechanisms, fully ranked language model objective functions, and contrastive learning-based objective functions to pre-train on massive amounts of corpus data.

[0171] It should be understood that the above architecture can also be applied to other natural language processing tasks, such as natural language synthesis, semantic understanding, and summarization.

[0172] (10) Domain Adaptation

[0173] Pre-trained language models, such as BERT and RoBERTa, have achieved excellent performance on NLP tasks. Typically, these general-purpose models are first pre-trained on large unlabeled corpora and then fine-tuned directly on downstream tasks. However, there is an inherent text distribution gap between the unlabeled pre-training corpus and the labeled task corpus, leading to a distribution shift problem that causes PLMs to perform poorly on certain domain tasks. To address this distribution shift problem, related research has proposed domain-adaptive pre-training, which further pre-trains a general PLM on large-scale domain corpora, achieving better performance than a standard PLM.

[0174] Recently, the domain transfer problem in PLM has attracted increasing research attention because the domain difference between the pre-training corpus and the downstream task can lead to a significant performance drop. Catastrophic forgetting of general domain knowledge can occur after adaptive pre-training, resulting in poor performance on the downstream task. Catastrophic forgetting is a common phenomenon in continuous learning, where a trained model forgets previously learned knowledge and overfits to the new task.

[0175] Existing implementations mitigate catastrophic forgetting through memory-based methods. Specifically, they store important samples from past tasks in external memory and refine them using gradient transformation strategies to reduce forgetting. However, these methods require external storage, consuming additional storage space to hold samples from past tasks.

[0176] To address the aforementioned problems, this application provides a data processing method.

[0177] Reference Figure 6A , Figure 6AThis is an illustrative embodiment of a data processing method provided in this application. The data processing method provided in this application can be applied to the training or execution devices described above. Specifically, the data processing method can be applied to terminal devices such as mobile phones, tablets, laptops, and smart wearable devices, or to cloud-based servers, such as... Figure 6A As shown, an embodiment of this application provides a data processing method, including:

[0178] 601. Process text data through a first pre-trained language model PLM to obtain a target feature representation; wherein the first PLM includes one or more first network layers, and the target feature representation is obtained based on the first feature representation output by the one or more first network layers.

[0179] In one possible implementation, training samples (i.e., text data) for PLM can be obtained. For example, the training samples may include a first data sequence and a second data sequence. The first data sequence may be obtained based on the source corpus, and the second data sequence may be obtained based on the target corpus. PLM needs to predict and generate the target corpus based on the source corpus.

[0180] In one possible implementation, PLM can be used to perform sequence conversion tasks between different language types, such as text translation tasks, summary generation tasks between different languages, etc. In this case, the first data sequence and the second data sequence can be texts including different language types (it is not limited that every data unit in the first data sequence is a different language type from the data units in the second data sequence; for example, some data units in the first data sequence and some or all data units in the second data sequence may be of the same language type). The language type can also be referred to as the language.

[0181] For example, in a Chinese-English translation task, the original text is "This trip needs careful planning," and its parallel English text is "The trip needs careful planning." "This trip needs careful planning" and "The trip needs careful planning" can be considered as a set of parallel corpora. This set of parallel corpora is a Chinese-English parallel language pair. The original text "This trip needs careful planning" can be considered as the source corpus of this set of parallel corpora, and the translated text "The trip needs careful planning" can be considered as the target corpus of this set of parallel corpora.

[0182] For example, in an English-German translation task, the source text is "We danse on the grass," and its parallel German text is "Wir tanzen auf dem gras." "We danse on the grass" and "Wir tanzenauf dem gras" can be regarded as a set of parallel corpora. This set of parallel corpora is an English-German parallel language pair. The source text "We danse on the grass" can be regarded as the source corpus of this set of parallel corpora, and the translated text "Wir tanzen auf demgras" can be regarded as the target corpus of this set of parallel corpora.

[0183] In one possible implementation, the first data sequence before the masking operation and the second data sequence before the masking operation are different data sequences that have been labeled with samples.

[0184] In one possible implementation, PLM can be used to implement text summarization tasks, where the source corpus can be the source corpus from which summaries need to be extracted, and the target corpus can be the summary text to be generated.

[0185] In one possible implementation, PLM can be used to implement a text response task, where the source corpus can be the source corpus that needs to be responded to, and the target corpus can be the response content based on the source corpus.

[0186] In one possible implementation, the first data sequence before the masking operation and the second data sequence before the masking operation are the same data sequence, that is, the first data sequence before the masking operation and the second data sequence before the masking operation are unlabeled data.

[0187] In one possible implementation, the first data sequence can be obtained by masking the original source corpus, and the second data sequence can be obtained by masking the original target corpus. Where the PLM can be used to implement sequence transformation tasks (e.g., translation tasks) between texts of different language types, the original source corpus and the original target corpus can be texts expressed in different language types.

[0188] Optionally, the original source corpus and the original target corpus can be obtained from external databases.

[0189] In one possible implementation, the PLM may include an embedding layer, which can be used to embed unmasked data units in the data sequence. This embedding layer can be called the input embedding layer. The current input can be unmasked data units. After acquiring the current input, the embedding layer can perform embedding processing on each unmasked data unit in the current input, obtaining the embedding vector corresponding to each unmasked data unit.

[0190] In some embodiments, a position vector of each data unit in the unmasked data units can also be obtained. The position vector is used to indicate the position of each data unit in the data sequence. Specifically, the position vector can be used to indicate the relative positional relationship between each data unit in the unmasked data units and other masked data units as well as the masked data units.

[0191] In one possible implementation, the embedding layer may include an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each data unit among the unmasked data units in the current input to obtain a word vector (e.g., representing semantic information) for each data unit. In the positional encoding layer, the position of each data unit among the unmasked data units in the current input can be obtained, and a position vector can be generated for the position of each data unit among the unmasked data units.

[0192] In some examples, the position information of each data unit in the unmasked data units can be the absolute position of each data unit in the unmasked data units in the data sequence. Taking the current input "When should Huabei be repaid?" as an example, the position of "When" can be represented as the first position, the position of "should" can be represented as the second position, and so on. In some examples, the position of each data unit in the unmasked data units in the data sequence can be the relative position of each data unit in the unmasked data units in the data sequence. Still taking the current input "When should Huabei be repaid?" as an example, the position of "When" can be represented as before "should", the position of "should" can be represented as after "When" and before "be", and so on. When obtaining the word vectors and position vectors of each data unit in the unmasked data units of the current input, the position vectors and the corresponding word vectors of each data unit in the unmasked data units can be fused to obtain the embedding vectors of each data unit in the unmasked data units. It should be understood that the fusion method can be an addition operation on the position vector and the corresponding word vector, or other operations, and the specific fusion method is not limited here. The embedding vector can be represented as an embedding matrix with a preset dimension. It can be set that the number of the embedding vectors is M and the preset dimension is H dimensions, then the embedding vector can be represented as an M×H embedding matrix.

[0193] In a possible implementation, the first PLM is trained based on text data in multiple domains.

[0194] In a possible implementation, the first PLM is a model with fixed parameters.

[0195] That is to say, the first PLM can be a general pre-trained language model with fixed parameters, that is, a model without catastrophic forgetting. During the model update process, the pre-trained language model in a specific domain will update the magnitude of the gradient according to the loss function value. In order to maintain the invariance of general domain knowledge, the parameters of the general domain model will be fixed, so the gradients of the general domain model will not change during training. Therefore, the pre-trained language model with fixed parameters is not affected by forgetting.

[0196] In the embodiments of the present application, the first PLM can include multiple network layers (for example, including one or more first network layers), and the output of one or more first network layers in the first PLM can be transmitted to the neural network model to be trained (that is, the second PLM in the embodiments of the present application), so as to fuse the general domain knowledge (from the first PLM) and the specific domain knowledge, and further reduce the degree of catastrophic forgetting of the second PLM.

[0197] In one possible implementation, the first PLM and the second PLM can be the Pangu model, the BERT series model, the GPT series model, etc.

[0198] In one possible implementation, the first network layer of the first PLM can be a transformer layer.

[0199] In one possible implementation, the PLM may include multiple transformer layers in sequence. Each transformer layer can process the data output from the previous transformer layer adjacent to it to obtain an intermediate vector, and then output the intermediate vector to the next transformer layer adjacent to it. Wherein, if the transformer layer is the one closest to the input side among the multiple transformer layers, then the input data of the transformer layer is an embedding vector; if the transformer layer is the one closest to the output side among the multiple transformer layers, then the data output by the transformer layer is a hidden state.

[0200] The core feature of the transformer layer lies in its unique attention mechanism. When processing natural language, such as a sentence, the transformer model uses this attention mechanism to assign different attention coefficients to the embedding vectors of each word in the sentence, thus more comprehensively considering the influence of the context on each word. A specific transformer layer may include sequentially adjacent multi-head attention layers, add and normalize layers, feed-forward layers, and add and normalize layers. The attention layer is connected to the embedding layer, obtaining embedding vectors as input vectors. Based on the correlation between the various embedding vectors, it synthesizes the embedding vectors to obtain an output vector, which is then fed to subsequent transformer layers. The transformer layer takes the output of the previous layer as its input vector and performs similar operations as the previous transformer layer.

[0201] Reference Figure 6C , Figure 6C This is a schematic diagram of a transformer layer structure. All transformer layers in the embodiments of this application can be referenced from this diagram. Figure 6C The structure shown in the figure, such as Figure 6CAs shown, the transformer layer consists of a multi-head attention layer, an add & normalization layer, a feed forward layer, and another add & normalization layer, which are sequentially adjacent to each other.

[0202] The multi-head attention layer obtains M input vectors X from the layer above it. l This can also be represented as matrix X. Employing a self-attention mechanism, it transforms each vector based on the correlation between them, yielding M output vectors, which can also be represented as matrix Y. It can be understood that when this multi-head attention layer is directly connected to the embedding layer, its input vector is the embedding vector output by the embedding layer; when this multi-head attention layer is included in a subsequent transformer layer, its input vector is the output vector of the previous transformer layer. In a multi-head attention layer, an MHA layer based on multi-head attention (MHA) includes multiple attention heads (such as...). Figure 6C The following are Head 1, Head 2, ..., Head N shown in the figure.

[0203] Figure 6D This is a schematic diagram illustrating the operation of an attention head, showing how the attention head transforms an input matrix X into an output matrix Y. For example... Figure 6D As shown, the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V are respectively applied to the M input vectors.<X1,X2,…,XN> The input vectors Xi are transformed to obtain the first intermediate vector (q vector), second intermediate vector (k vector), and third intermediate vector (v vector) corresponding to each input vector. Operationally, the input matrix X composed of N input vectors can be linearly transformed using the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V, respectively, to obtain the Q matrix, K matrix, and V matrix of the input matrix. Then, the matrices are split to obtain the q vector, k vector, and v vector corresponding to each input vector. For any i-th input vector Xi among the M input vectors, the correlation degree between the i-th input vector Xi and each input vector Xj is determined based on the dot product operation of the first intermediate vector (q vector, qi) corresponding to the i-th input vector and the second intermediate vector (k vector, kj) corresponding to each input vector Xj. Although the dot product result of qi and kj can be directly determined as the correlation degree, a more classic approach is to first divide the dot product result by a constant, then perform a softmax operation, and use the result as the correlation degree between the input vector Xi and Xj, that is:

[0204]

[0205] Therefore, the correlation degrees αi,j between the i-th input vector Xi and each input vector Xj can be used as weighting factors to perform a weighted combination of the third intermediate vectors (v vector, vj) corresponding to each input vector Xj, resulting in the i-th combined vector Ci corresponding to the i-th input vector Xi:

[0206]

[0207] Therefore, we can obtain a vector sequence of M combined vectors corresponding to the M input vectors.<C1,C2,…,CN> Or matrix C. Based on this combined vector sequence, M output vectors can be obtained. Specifically, in one embodiment, the vector sequence of N combined vectors can be directly used as the M output vectors, i.e., Yi = Ci. In this case, the output matrix Y is the combined vector matrix C, which can also be written as:

[0208]

[0209] The above describes the processing flow of an attention head. In the MHA architecture, the MHA layer maintains m sets of transformation matrices. Each set of transformation matrices includes the aforementioned first transformation matrix Q, second transformation matrix K, and third transformation matrix V, allowing the above operations to be performed in parallel to obtain m combined vector sequences (i.e., m matrices C). Each vector sequence includes N combined vectors obtained based on a set of transformation matrices. In this case, the MHA layer concatenates the m combined vector sequences to obtain a concatenated matrix; then, it transforms this concatenated matrix using the fourth transformation matrix W to obtain the final output matrix Y. This output matrix Y is then split into M output vectors.<Y1,Y2,…,YN> Through the above operations, the MHA layer performs transformation operations based on the correlation between N input vectors to obtain M output vectors.

[0210] like Figure 8 As shown, a transformer layer may include a feedforward layer, which comprises an input layer, an intermediate layer, and an output layer. As previously mentioned, a neural network model may contain multiple transformer layers. In one embodiment, these multiple transformer layers may be stacked and connected in a residual network manner.

[0211] Next, we will introduce how to fuse the outputs of one or more first network layers in the first PLM.

[0212] In one possible implementation, the output (target feature representation) of a first network layer in the first PLM can be transmitted to the second PLM.

[0213] In one possible implementation, the outputs of multiple first network layers in a first PLM can be fused to obtain a target feature representation, and this target feature representation can be transmitted to a second PLM. Each of the multiple network layers can obtain a first feature representation, and these multiple first feature representations can then be fused to obtain the target feature representation.

[0214] In one possible implementation, the target feature representation can be obtained by weighted summation based on the first feature representations output by multiple first network layers and the weights corresponding to each first network layer. The weights corresponding to each first network layer can be updated during training, and the output of the first network layer with larger weights can be considered more important.

[0215] The weighted summation fusion method described above can be called a gated memory transfer strategy. It utilizes a fine-grained token-level gating mechanism to adaptively calculate weights (learnable or adjustable parameters) for the same token representations at different layers, sums these weighted representations into a single memory representation, and then fuses them into a domain-specific PLM layer. The gated memory transfer strategy is as follows: Figure 6B As shown, Figure 6B The left side is the hidden layer memory representation cache. Figure 6B The right side represents a domain-specific pre-trained language model.

[0216] After obtaining the target feature representation, it can be input into a network layer of the second PLM. Specifically, it can be input into the attention layer of a network layer of the second PLM.

[0217] The following section will introduce the different layer correspondences. Essentially, this is a layer allocation problem between a general pre-trained language model and a domain-specific pre-trained language model, involving one-to-many, many-to-many, or many-to-one layer assignments.

[0218] Scenario 1:

[0219] Reference Figure 7 The feature representation output from one network layer in the first PLM can be input into an attention layer in the second PLM.

[0220] Scenario 2:

[0221] Reference Figure 8 The feature representations output by multiple network layers in the first PLM can be input into multiple attention layers in the second PLM, and the feature representation output by each network layer can be input into one attention layer in the second PLM. The multiple network layers can be some of the network layers in the first PLM.

[0222] Scenario 3:

[0223] The feature representations output by multiple network layers in the first PLM can be input into multiple attention layers in the second PLM, and the feature representation output by each network layer can be input into one attention layer in the second PLM. The multiple network layers can be all network layers of the first PLM (all network layers can be understood as all network layers used to obtain the feature representation, excluding output layers).

[0224] Scenario 4:

[0225] Reference Figure 9 The feature representations output by multiple network layers in the first PLM can be fused, and the fused result can be input into an attention layer in the second PLM. These multiple network layers can be some of the network layers in the first PLM.

[0226] Scenario 5:

[0227] The feature representations output by multiple network layers in the first PLM can be fused, and the fused result can be input into an attention layer in the second PLM. The feature representation output by each network layer is input into an attention layer in the second PLM. The multiple network layers can be all network layers of the first PLM (the so-called all network layers can be understood as all network layers used to obtain feature representations, for example, excluding the output layer).

[0228] Situation 6:

[0229] Reference Figure 10 The system can fuse multiple network layers from multiple network layers in the first PLM (each network layer group can include multiple network layers) to obtain multiple fusion results, and input each of the multiple fusion results into an attention layer in the second PLM. The multiple network layers can include some of the network layers in the first PLM.

[0230] Situation 7:

[0231] Each group of network layers in the first PLM can be fused (each group of network layers may include multiple network layers) to obtain multiple fusion results, and each of the multiple fusion results is input into an attention layer in the second PLM. The multiple groups of network layers can include all network layers of the first PLM (the so-called all network layers can be understood as all network layers used to obtain feature representation, for example, excluding the output layer).

[0232] Situation 8:

[0233] Reference Figure 11The system can fuse each of the multiple network layers in the first PLM (each network layer may include one or more network layers; if it includes only one network layer, the network layers in the group do not need to be fused) to obtain multiple fusion results, and input each of the multiple fusion results into an attention layer in the second PLM.

[0234] Situations 1, 4, and 5 above can be called single-layer memory transfer strategies. That is, M can be extracted based on the last hidden state (not limited to one or more) in the hidden layer memory representation cache. f (The fusion result or the output of a certain layer) is then fused into one layer of a domain-specific pre-trained language model. Situations 2 and 3 above can be called multi-layer memory transfer strategies: single-layer memory transfer strategies may ignore shallow knowledge learned from a general pre-trained language model. To enable layered interaction between a general pre-trained language model and a domain-specific pre-trained language model, a multi-layer transfer strategy is proposed. This strategy utilizes all hidden states in the hidden layer memory representation cache as memory representations, and then fuses them into the corresponding layers of the domain-specific pre-trained and language models without introducing any new parameters.

[0235] Because the upper and lower layers of a pre-trained language model have significantly different representations, a block-based variant (i.e., cases 6, 7, and 8 above) is proposed. This variant separates the layers of a typical PLM into higher-level and lower-level blocks, and then applies a gating fusion strategy to obtain the memory representations of the upper and lower layers separately. and Finally, we integrate them into two memory enhancement layers in a domain-specific pre-trained language model.

[0236] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; the input to the second attention layer includes the target feature representation and the third feature representation output by the third network layer. Furthermore, to reduce the complexity of network processing, the multiple network layers of the second PLM can also share the target feature representation, that is, the target feature representation can be input to multiple attention layers of the second PLM (e.g., including a first attention layer and a second attention layer).

[0237] 602. The text data is processed by a second PLM; wherein the second PLM includes a first attention layer and a second network layer connected to the first attention layer; the input of the first attention layer includes the target feature representation and the second feature representation output by the second network layer.

[0238] In this embodiment, the hidden layer memory representation cache (i.e., the target feature representation) and domain-related text content are input into a domain-specific pre-trained model, which performs forward inference on the domain-related text content. The attention layer, which processes both the target feature representation and data from its own model, can be called the memory enhancement layer. In the memory enhancement layer, the knowledge representation obtained by the memory enhancement strategy is adaptively fused with the domain-specific knowledge representation. The enhanced table, after general knowledge fusion, is then forward-propagated to the output layer, improving the model's generalization ability for downstream tasks.

[0239] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM. Alternatively, the layer closest to the top of the domain-specific PLM model can be chosen as the memory enhancement layer; this method has shown the best performance in experiments and does not require additional parameters.

[0240] Next, we will introduce the first attention layer, also known as the memory enhancement layer, in the embodiments of this application.

[0241] In one possible implementation, the target feature representation and the second feature representation can be processed through the first attention layer; wherein, the first attention layer of the text is used to perform interaction between different embedding vectors in the target feature representation and the second feature representation to obtain attention information, and each embedding vector corresponds to a text unit in the text data.

[0242] In one possible implementation, the first attention layer is specifically used to: obtain a first Q matrix and a first V matrix based on the target feature representation, and obtain a first K matrix based on the second feature representation; and perform interaction between the first Q matrix, the first V matrix, and the first K matrix.

[0243] Since the input includes data from both the general PLM (i.e., the first PLM) network layer and the network layer of its own network (i.e., the second PLM), the difference between the memory enhancement layer and the traditional self-attention layer lies only in the design of the multi-head self-attention module. This application proposes a novel memory-enhanced attention module that integrates general domain memory representations into a domain-specific pre-trained language model, represented as memory-attention. Specifically, the memory representations are linearly transformed into new value-key pairs and concatenated to the pairs generated by the domain-specific pre-trained language model. Then, multi-head self-attention is performed to adaptively integrate this new concatenated representation. The entire process reuses the parameters of the transformation layer of the domain-specific pre-trained language model without introducing any new parameters. Therefore, the formula for the memory enhancement layer can be concisely expressed as follows:

[0244]

[0245] Here, the subscript i indicates that the current layer is i, and the subscript j indicates a specific attention head j. and Q represents the new value-bond pair obtained after concatenation. i,j Query representations for pre-trained language models for specific domains.

[0246] In one possible implementation, after the text data is processed by the second PLM, the processing result of the text data can be obtained; then, the second PLM can be updated according to the processing result and the corresponding truth value to obtain the updated second PLM.

[0247] This application provides a data processing method, comprising: processing text data through a first pre-trained language model (PLM) to obtain a target feature representation; wherein the first PLM includes one or more first network layers, and the target feature representation is obtained based on the first feature representation output by the one or more first network layers; processing the text data through a second PLM; wherein the second PLM includes a first attention layer and a second network layer connected to the first attention layer; the input of the first attention layer includes the target feature representation and the second feature representation output by the second network layer. Through this method, a general memory knowledge representation is effectively constructed from a general pre-trained language model (first PLM), and then fused into a domain-specific pre-trained language model (second PLM) through a memory enhancement layer (first attention layer). This enables the domain-specific pre-trained language model to acquire forgotten general domain knowledge, reducing the catastrophic forgetting problem that occurs during pre-training without requiring additional storage space.

[0248] The following is a specific illustration of the data processing method in the embodiments of this application:

[0249] In this embodiment, the method uses a general pre-trained language model (General RoBERTa) and domain-specific pre-trained language models (including biomedical, computer science, news, and online commentary domains) to form the main architecture of the model. Based on this main architecture, we innovate memory enhancement layers and memory enhancement strategies. For example... Figure 1 As shown, in this embodiment, the method takes text content in the computer field as input, and the task is mainly to classify the annotation citations of articles in the computer field.

[0250] Specifically, such as Figure 12As shown, for each input domain-specific text statement, it is first segmented into words, and the words are converted into dictionary indices based on the dictionary provided by the pre-trained language model. Simultaneously, the positional encoding of each word is obtained, vectorized together with the word's dictionary index, and then fused with the word's positional encoding and word vector. After obtaining the initial representation of the text statement, we first pass the representation through a general pre-trained language model with fixed parameters. In this model, after inputting the representation, the model performs forward inference, obtaining representations of the general domain knowledge at each layer, which are stored in the hidden layer memory cache in layer order. Next, we input the hidden layer memory cache and the initial representation into the domain-specific pre-trained language model. At this point, the memory enhancement strategy module retrieves the general memory representation M from the hidden layer memory cache according to the set strategy. f In a memory enhancement layer of a domain-specific pre-trained model, M f The key K and value V from the domain-specific knowledge representation are concatenated sequentially to obtain key-value pairs that simultaneously contain both general and domain-specific knowledge. Simultaneously, the query Q, representing the domain-specific knowledge, undergoes the same attention weight calculation as in Equation 1 with the fused key-value pairs. This attention mechanism adaptively and automatically selects general and domain-specific knowledge through the query Q, ensuring the model doesn't forget previously learned general-domain knowledge. After general-domain knowledge enhancement, the fused representation output by the pre-trained domain-specific model is forward-propagated to the output layer. A softmax operation is performed at the output layer to obtain the probability of each category, and a cross-entropy operation is performed with the label index value to obtain the forward propagation loss.

[0251] During backpropagation, the domain-specific pre-trained language model updates the gradient magnitude based on the loss function value. To maintain the invariance of general domain knowledge, the parameters of the general domain model are fixed, so the gradient of the general domain model does not change during training.

[0252] This application also provides a data processing method, applied to data processing based on... Figure 6A The inference process of the model trained in the corresponding embodiment, specifically, the method includes:

[0253] Get text data;

[0254] The second PLM processes the text data to obtain the processing result corresponding to the text data; wherein, the second PLM includes a first attention layer and a second network layer connected to the first attention layer; when training the second PLM, the input of the first attention layer includes a second feature representation output by the second network layer and a target feature representation, wherein the target feature representation is obtained by the first PLM when processing the text data by one or more first network layers outputting a first feature representation.

[0255] In one possible implementation, the first PLM is trained based on multi-domain text data.

[0256] In one possible implementation, the target feature representation is specifically obtained by fusing the first feature representations output by multiple first network layers.

[0257] In one possible implementation, the target feature representation is specifically obtained by weighted summation of the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

[0258] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; when training the second PLM, the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0259] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM.

[0260] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0261] exist Figures 1 to 12 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 13 , Figure 13 This is a schematic diagram of a data processing device 1300 provided in an embodiment of this application. The data processing device 1300 may be a terminal device or a server, and may include:

[0262] Processing module 1301 is used to process text data through a first pre-trained language model PLM to obtain a target feature representation; wherein the first PLM includes one or more first network layers, and the target feature representation is obtained based on the first feature representation output by the one or more first network layers;

[0263] The text data is processed by a second PLM; wherein the second PLM includes a first attention layer and a second network layer connected to the first attention layer; the input of the first attention layer includes the target feature representation and the second feature representation output by the second network layer.

[0264] The specific description of the processing module 1301 can be found in the descriptions of steps 601 and 602 in the above embodiments, and will not be repeated here.

[0265] In one possible implementation, the first PLM is trained based on multi-domain text data.

[0266] In one possible implementation, the processing module is specifically used for:

[0267] The text data is processed by the first pre-trained language model PLM to obtain multiple first feature representations output by the first network layer.

[0268] The target feature representation is obtained by fusing the first feature representations output by multiple first network layers.

[0269] In one possible implementation, the processing module is specifically used for:

[0270] The target feature representation is obtained by weighted summation based on the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

[0271] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0272] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM.

[0273] In one possible implementation, the processing module is specifically used for:

[0274] The first attention layer processes the target feature representation and the second feature representation; wherein, the first attention layer of the text is used to perform interaction between different embedding vectors in the target feature representation and the second feature representation to obtain attention information, and each embedding vector corresponds to a text unit in the text data.

[0275] In one possible implementation, the first attention layer is specifically used for:

[0276] Based on the target feature representation, a first Q matrix and a first V matrix are obtained, and based on the second feature representation, a first K matrix is ​​obtained;

[0277] The interaction between the first Q matrix, the first V matrix, and the first K matrix is ​​performed.

[0278] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0279] In one possible implementation, after processing the text data via the second PLM, a processing result of the text data is obtained; the apparatus further includes:

[0280] The update module is used to update the second PLM based on the processing result and the corresponding truth value, so as to obtain the updated second PLM.

[0281] In one possible implementation, the target feature representation is specifically obtained by fusing the first feature representations output by multiple first network layers.

[0282] In one possible implementation, the target feature representation is specifically obtained by weighted summation of the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

[0283] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; when training the second PLM, the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0284] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM.

[0285] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0286] This application embodiment also provides a data processing apparatus, the apparatus comprising:

[0287] The processing module is used to acquire text data;

[0288] The second PLM processes the text data to obtain the processing result corresponding to the text data; wherein, the second PLM includes a first attention layer and a second network layer connected to the first attention layer; when training the second PLM, the input of the first attention layer includes a second feature representation output by the second network layer and a target feature representation, wherein the target feature representation is obtained by the first PLM processing the text data by outputting feature representations from one or more first network layers.

[0289] In one possible implementation, the first PLM is trained based on multi-domain text data.

[0290] In one possible implementation, the target feature representation is specifically obtained by fusing the first feature representations output by multiple first network layers.

[0291] In one possible implementation, the target feature representation is specifically obtained by weighted summation of the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

[0292] In one possible implementation, the second PLM further includes a second attention layer and a third network layer connected to the second attention layer; when training the second PLM, the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

[0293] In one possible implementation, the first attention layer is the attention layer closest to the output layer in the second PLM.

[0294] In one possible implementation, the first network layer and the second network layer are transformer layers.

[0295] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 14 , Figure 14 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1400 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device, or a server, etc., and is not limited thereto. Specifically, the execution device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403, and a memory 1404 (wherein the execution device 1400 may have one or more processors 1403). Figure 14 (Taking a processor as an example), processor 1403 may include application processor 14031 and communication processor 14032. In some embodiments of this application, receiver 1401, transmitter 1402, processor 1403 and memory 1404 may be connected via a bus or other means.

[0296] Memory 1404 may include read-only memory and random access memory, and provides instructions and data to processor 1403. A portion of memory 1404 may also include non-volatile random access memory (NVRAM). Memory 1404 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0297] Processor 1403 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0298] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1403. The processor 1403 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1403 or by instructions in software form. The processor 1403 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1403 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1404. Processor 1403 reads the information in memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0299] Receiver 1401 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1402 can be used to output digital or character information through the first interface; transmitter 1402 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1402 may also include a display device such as a display screen.

[0300] This application also provides a training device; please refer to [link / reference]. Figure 15 , Figure 15 This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 1500 is implemented by one or more servers. The training device 1500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1515 (e.g., one or more processors) and memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1515 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the training device 1500.

[0301] The training device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558; or, one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0302] In this embodiment, the central processing unit 1515 is used to execute... Figure 6A The data processing method described in the corresponding embodiment.

[0303] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0304] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0305] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the model training method described in the above embodiments, or to cause the chip within the training device to execute the model training method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0306] For details, please refer to Figure 16 , Figure 16 This is a schematic diagram of a chip structure provided in an embodiment of this application. Figure 6A The data processing method described in the corresponding embodiment can be used in... Figure 16 This is implemented in the chip shown. Specifically, the chip can be represented as a neural network processor (NPU) 1600. The NPU 1600 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 1603, and the controller 1604 controls the arithmetic circuit 1603 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0307] Optionally, the above Figure 6A The model training method described in the corresponding embodiments can be provided by Figure 16 The main CPU and NPU in the chip shown work together to complete this task.

[0308] In some implementations, the arithmetic circuit 1603 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1603 is a two-dimensional pulsating array. The arithmetic circuit 1603 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1603 is a general-purpose matrix processor.

[0309] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1602 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1601 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1608.

[0310] Unified memory 1606 is used to store input and output data. Weight data is directly transferred to weight memory 1602 via Direct Memory Access Controller (DMAC) 1605. Input data is also transferred to unified memory 1606 via DMAC.

[0311] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1609.

[0312] The Bus Interface Unit (BIU) 1610 is used by the instruction fetch memory 1609 to fetch instructions from external memory, and also by the memory access controller 1605 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0313] The DMAC is mainly used to move input data from external memory DDR to unified memory 1606, or to weight data to weight memory 1602, or to input data to input memory 1601.

[0314] The vector computation unit 1607 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0315] In some implementations, the vector computation unit 1607 can store the processed output vector in the unified memory 1606. For example, the vector computation unit 1607 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1603, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1607 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1603, for example, for use in subsequent layers of the neural network.

[0316] The instruction fetch buffer 1609 connected to the controller 1604 is used to store the instructions used by the controller 1604.

[0317] The unified memory 1606, input memory 1601, weight memory 1602, and instruction fetch memory 1609 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0318] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0319] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0320] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0321] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0322] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Text data is processed through a first pre-trained language model (PLM) to obtain a target feature representation; wherein the first PLM includes one or more first network layers, and the target feature representation is obtained based on the first feature representation output by the one or more first network layers; The text data is processed by a second PLM; wherein the second PLM includes a first attention layer and a second network layer connected to the first attention layer; the input of the first attention layer includes the target feature representation and the second feature representation output by the second network layer.

2. The method according to claim 1, characterized in that, The first PLM is obtained by training on multi-domain text data; or, The first PLM is a model with fixed parameters.

3. The method according to claim 1 or 2, characterized in that, The process of processing text data through a first pre-trained language model (PLM) to obtain target feature representations includes: The text data is processed by the first pre-trained language model PLM to obtain multiple first feature representations output by the first network layer. The target feature representation is obtained by fusing the first feature representations output by multiple first network layers.

4. The method according to claim 3, characterized in that, The step of fusing the first feature representations output by multiple first network layers to obtain the target feature representation includes: The target feature representation is obtained by weighted summation based on the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

5. The method according to claim 1 or 2, characterized in that, The second PLM further includes a second attention layer and a third network layer connected to the second attention layer; the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

6. The method according to claim 1 or 2, characterized in that, The first attention layer is the attention layer closest to the output layer in the second PLM.

7. The method according to claim 1 or 2, characterized in that, The process of processing the text data through the second PLM includes: The first attention layer processes the target feature representation and the second feature representation; wherein, the first attention layer of the text is used to perform interaction between different embedding vectors in the target feature representation and the second feature representation to obtain attention information, and each embedding vector corresponds to a text unit in the text data.

8. The method according to claim 7, characterized in that, The first attention layer is specifically used for: Based on the target feature representation, a first Q matrix and a first V matrix are obtained, and based on the second feature representation, a first K matrix is ​​obtained; The interaction between the first Q matrix, the first V matrix, and the first K matrix is ​​performed.

9. The method according to claim 1 or 2, characterized in that, The first network layer and the second network layer are transformer layers.

10. The method according to claim 1 or 2, characterized in that, After processing the text data through the second PLM, the processing result of the text data is obtained; the method further includes: Based on the processing result and the corresponding truth value, the second PLM is updated to obtain the updated second PLM.

11. A data processing method, characterized in that, The method includes: Get text data; The second PLM processes the text data to obtain the processing result corresponding to the text data; wherein, the second PLM includes a first attention layer and a second network layer connected to the first attention layer; when training the second PLM, the input of the first attention layer includes a second feature representation output by the second network layer and a target feature representation, wherein the target feature representation is obtained by the first PLM processing the text data by outputting feature representations from one or more first network layers.

12. The method according to claim 11, characterized in that, The first PLM is obtained by training on multi-domain text data; or, The first PLM is a model with fixed parameters.

13. A data processing apparatus, characterized in that, The device includes: The processing module is used to process text data through a first pre-trained language model (PLM) to obtain a target feature representation; wherein the first PLM includes one or more first network layers, and the target feature representation is obtained based on the first feature representation output by the one or more first network layers; The text data is processed by a second PLM; wherein the second PLM includes a first attention layer and a second network layer connected to the first attention layer; the input of the first attention layer includes the target feature representation and the second feature representation output by the second network layer.

14. The apparatus according to claim 13, characterized in that, The first PLM is obtained by training on multi-domain text data; or, The first PLM is a model with fixed parameters.

15. The apparatus according to claim 13 or 14, characterized in that, The processing module is specifically used for: The text data is processed by the first pre-trained language model PLM to obtain multiple first feature representations output by the first network layer. The target feature representation is obtained by fusing the first feature representations output by multiple first network layers.

16. The apparatus according to claim 13 or 14, characterized in that, The processing module is specifically used for: The target feature representation is obtained by weighted summation based on the first feature representations output by multiple first network layers and the weights corresponding to each first network layer.

17. The apparatus according to claim 13 or 14, characterized in that, The second PLM further includes a second attention layer and a third network layer connected to the second attention layer; the input of the second attention layer includes the target feature representation and the third feature representation output by the third network layer.

18. The apparatus according to claim 13 or 14, characterized in that, The first attention layer is the attention layer closest to the output layer in the second PLM.

19. The apparatus according to claim 13 or 14, characterized in that, The processing module is specifically used for: The first attention layer processes the target feature representation and the second feature representation; wherein, the first attention layer of the text is used to perform interaction between different embedding vectors in the target feature representation and the second feature representation to obtain attention information, and each embedding vector corresponds to a text unit in the text data.

20. The apparatus according to claim 19, characterized in that, The first attention layer is specifically used for: Based on the target feature representation, a first Q matrix and a first V matrix are obtained, and based on the second feature representation, a first K matrix is ​​obtained; The interaction between the first Q matrix, the first V matrix, and the first K matrix is ​​performed.

21. The apparatus according to claim 13 or 14, characterized in that, The first network layer and the second network layer are transformer layers.

22. The apparatus according to claim 13 or 14, characterized in that, After processing the text data through the second PLM, the processing result of the text data is obtained; the device further includes: The update module is used to update the second PLM based on the processing result and the corresponding truth value, so as to obtain the updated second PLM.

23. A data processing apparatus, characterized in that, The device includes: The processing module is used to acquire text data; The second PLM processes the text data to obtain the processing result corresponding to the text data; wherein, the second PLM includes a first attention layer and a second network layer connected to the first attention layer; when training the second PLM, the input of the first attention layer includes a second feature representation output by the second network layer and a target feature representation, wherein the target feature representation is obtained by the first PLM processing the text data by outputting feature representations from one or more first network layers.

24. The apparatus according to claim 23, characterized in that, The first PLM is obtained by training on multi-domain text data; or, The first PLM is a model with fixed parameters.

25. A data processing apparatus, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method as described in any one of claims 1 to 12.

26. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 12.

27. A computer program product, characterized in that, The computer program product includes code that, when executed, performs the steps of the method according to any one of claims 1 to 12.

Images