A data processing method and related equipment

Abstract

A data processing method relates to the field of artificial intelligence, comprising: obtaining a text to be processed and a pretrained language model, the pretrained language model comprising a feature extraction network and a prediction network; extracting features from the text to be processed using the feature extraction network of the pretrained language model to obtain a feature representation of the data to be processed, wherein the feature representation is a complex number; performing an orthogonal transformation on the length-unitized feature representation using the prediction network of the pretrained language model to obtain an orthogonal transformed result, and determining a text prediction result based on the orthogonal transformed result. This application can perform operations on the pretrained language model on a quantum circuit, and by constructing a pretrained language model represented by a complex value, the representation capability of the model is improved, thereby enhancing the performance of the network.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method and related equipment. Background Art

[0002] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0003] With the continuous development of artificial intelligence technology, natural language human-computer interaction systems, which enable human-computer interaction through natural language, are becoming increasingly important. To enable human-computer interaction through natural language, systems must be able to recognize the specific meaning of human natural language. Typically, these systems extract key information from natural language sentences to identify the specific meaning of sentences.

[0004] The transformer architecture possesses powerful semantic expression capabilities and can capture long-term dependencies in text. Since its introduction, it has significantly surpassed previous models in a range of natural language processing tasks, particularly translation. Pre-trained language models based on the transformer architecture have also achieved excellent results in areas such as question-answering systems and voice assistants.

[0005] Pretrained language models have significantly advanced the field of artificial intelligence. Deep neural networks based on a multi-layer transformer architecture, trained on large-scale corpora, have achieved significant performance leaps in fields such as natural language processing, image processing, and speech recognition. Research on neural networks based on complex-valued representations has been ongoing since the 1990s and has recently garnered increasing attention due to the success of neural networks. Over a long period of research, complex-valued representations have been successfully applied to neural network architectures such as transformers, demonstrating a range of advantages, including superior performance, rapid convergence, high model robustness, and strong interpretability.

[0006] Quantum computing has been applied to various fields in recent years, but its application in natural language processing is still in its infancy. Researchers have successfully built quantum circuits capable of text classification and question-answer matching. The forward pass uses quantum simulation, and the resulting errors are used to update the circuit parameters via classical backpropagation. However, the representation capabilities of these models are significantly limited, resulting in generally poor performance. Summary of the Invention

[0007] The present application provides a data processing method that uses orthogonal transformation to adapt to quantum computing of quantum circuits, and then can perform prediction network operations on quantum circuits. By constructing a pre-trained language model represented by complex values, the representation ability of the model is improved, and the performance of the network is improved.

[0008] In a first aspect, the present application provides a data processing method, comprising: obtaining a text to be processed and a pre-trained language model, the pre-trained language model comprising a feature extraction network and a prediction network; performing feature extraction on the text to be processed through the feature extraction network to obtain a feature representation of the data to be processed, wherein the feature representation is a complex number; and performing an orthogonal transformation on the feature representation after length unitization through the prediction network to obtain a result after the orthogonal transformation, and determining a text prediction result based on the result after the orthogonal transformation.

[0009] In the embodiment of the present application, compared with the W*a+b operation adopted by the existing prediction network, the orthogonal transformation can be adapted to the quantum computing of the quantum circuit. The quantum state obtained by the orthogonal transformation can pass through the quantum measurement layer, that is, measure the probability of collapse to each quantum ground state, and then perform the prediction network operation on the quantum circuit. Moreover, by constructing a pre-trained language model represented by complex values, the representation ability of the model is improved, and the performance of the network is improved.

[0010] In one possible implementation, the pre-trained language model is used to perform a target task, and the text processing result is the processing result of the target task; the target task is one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language reasoning, text automatic question answering, text intent recognition, text classification, text simplification, and text story generation.

[0011] In one possible implementation, the fully connected layers in the prediction network can be connected through activation layers, and the module length of the output close to the fully connected layer is normalized by softmax to obtain the probability distribution of each category.

[0012] For example, two fully connected layers can be added to the output of the last complex transformer layer for the prediction network of the mask language model (MLM) and next sentence prediction (NSP) tasks, connected by a Tanh nonlinear activation function for the real and imaginary parts. The final output of both networks is the probability distribution of their respective categories, obtained by normalizing the modulus of the output of the last complex fully connected layer through softmax.

[0013] In one possible implementation, in order to adapt the prediction network to the quantum adaptation structure of the quantum circuit, the input quantum state (unit complex vector) can first be orthogonally transformed (the parameters of the orthogonal transformation can be trained). The parameterization method of the orthogonal transformation layer is as follows:

[0014]

[0015] Here, A is a complex positive definite matrix, which is transformed into an orthogonal matrix U through matrix exponential operations. The network's errors can be backpropagated into the weights W, allowing the entire orthogonal transformation to be trained through backpropagation. The quantum state that has undergone the orthogonal transformation passes through a quantum measurement layer, which measures the probability of collapse to each quantum ground state. The square of the modulus of the complex vector corresponding to the quantum state is the measured probability vector. This probability vector is then passed through a linear projection layer to obtain the sentence's category label.

[0016] Specifically, in a possible implementation, performing an orthogonal transformation on the feature representation after the unitization process includes: performing an orthogonal transformation on the feature representation after the unitization process through an orthogonal matrix: the orthogonal matrix is:

[0017] U=e iA ;

[0018] in, W is the trainable weight and H is the conjugate transpose of the complex matrix.

[0019] It should be understood that the above-mentioned orthogonalization calculation process can be used during model pre-training or model fine-tuning.

[0020] In one possible implementation, a quantum-adapted pre-trained language model can be built upon the aforementioned model and adapted for quantum computation. To ensure the mathematical validity of this quantum computation mapping, the intermediate representation of the complex network's [CLS] character can be constrained to be unitary, allowing it to be considered a quantum state throughout the network, facilitating adaptation to quantum circuits.

[0021] In one possible implementation, the normalization layer may change the operation on CLS characters to a length unitization operation.

[0022] Specifically, in one possible implementation, the feature extraction network includes a transformer layer, which includes a normalization layer. The normalization layer is used to perform length unitization processing on a target character, where the target character is a CLS character inserted at the beginning of the text to be processed. The multi-layer normalization operation can refer to the following formula:

[0023]

[0024] in and σ z are the mean and standard deviation of the complex sequence, respectively.

[0025] In one possible implementation, the feature extraction network includes a transformer layer, the transformer layer includes a feed-forward layer FFN, the FFN includes an activation layer, and the activation layer is used to perform nonlinear activation on part of the data input into the activation layer, the part of the data does not include data corresponding to the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

[0026] In a possible implementation, the portion of data input into the activation layer is a complex number, and the activation layer is specifically configured to perform nonlinear activation on the real part and the imaginary part of the portion of data, respectively.

[0027] In one possible implementation, the prediction network includes a target fully connected layer, which is a fully connected layer close to the output layer in the prediction network, and the target fully connected layer includes a trainable first unit vector and a second unit vector; the result after the orthogonal transformation can be respectively operated with the first unit vector and the second unit vector to obtain the first probability and the second probability, the first unit vector corresponds to the first probability, and the second unit vector corresponds to the second probability, the first probability represents the probability that the text prediction result belongs to the target label, and the second probability represents the probability that the text prediction result does not belong to the target label; the text prediction result is determined based on the first probability and the second probability.

[0028] Through the above method, the last fully connected layer of the prediction network is also changed to a form similar to quantum measurement, that is, two unit vectors are trained as measurement states, the inner product of the input representation and these two vectors is calculated respectively, and the square of the inner product is linearly normalized to a probability value. Optionally, the probability value can be used to calculate the cross entropy with the binary classification label as the loss function.

[0029] It should be understood that since the Q matrix, K matrix and V matrix in the embodiments of the present application are complex matrices, that is, each element therein is a complex number including a real part and an imaginary part, it is necessary to adopt an attention mechanism suitable for complex numbers.

[0030] In one possible implementation, since the softmax operation needs to be performed in the real number domain, and the result of the operation between the Q matrix and the K matrix (that is, the object of the softmax operation) is a complex number, the result (complex number) of the operation between the Q matrix and the K matrix can be mapped to the real number domain in an embodiment of the present application.

[0031] In one possible implementation, taking the result of the operation between the Q matrix and the K matrix as the first data as an example, the real part and the imaginary part can be mapped to the second data (real number) according to the values ​​of the real part and the imaginary part of the first data. For example, a preset operation can be performed, that is, a numerical operation can be performed on the values ​​of the real part and the imaginary part to obtain a real number value as the second data.

[0032] In one possible implementation, the modulus of the first data can be determined as the second data based on the values ​​of the real part and the imaginary part of the first data. Since in the transformer model based on complex numbers, the modulus of the complex number and the probability of the final output are related, that is, the modulus of the complex number itself has a definite physical meaning. In the process of mapping the real part and the imaginary part to the second data, the mapping method of the complex modulus can increase the interpretability of the network and improve the accuracy of the network.

[0033] Specifically, in a possible implementation, a head (an attention head in a multi-head attention) can be used to obtain the K matrix and the Q matrix of the text to be processed; the K matrix and the Q matrix are calculated to obtain first data, which is a complex number; the numerical values ​​of the real part and the imaginary part of the first data are mapped to second data, which is a real number; and a softmax operation is performed on the second data.

[0034] Specifically, in a possible implementation, mapping the values ​​of the real part and the imaginary part of the first data to the second data includes: determining the modulus of the first data as the second data according to the values ​​of the real part and the imaginary part of the first data.

[0035] In one possible implementation, the method further includes: determining a target loss based on the text prediction result; and performing backpropagation of the pre-trained language model based on the target loss, wherein the gradient and momentum used in the backpropagation are complex numbers.

[0036] In a second aspect, the present application provides a data processing device, comprising:

[0037] An acquisition module is used to acquire the text to be processed and a pre-trained language model, wherein the pre-trained language model includes a feature extraction network and a prediction network;

[0038] a feature extraction module, configured to extract features from the text to be processed through the feature extraction network to obtain a feature representation of the data to be processed, wherein the feature representation is plural; and

[0039] The prediction module is used to perform an orthogonal transformation on the feature representation after the length unitization processing through the prediction network to obtain an orthogonal transformed result, and determine a text prediction result based on the orthogonal transformed result.

[0040] In one possible implementation, the pre-trained language model is used to perform a target task, and the text processing result is a processing result of the target task; the target task is one of the following:

[0041] Reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language reasoning, text automatic question answering, text intent recognition, text classification, text simplification, and text story generation.

[0042] In a possible implementation, the prediction module is specifically configured to:

[0043] The orthogonal transformation is performed on the normalized feature representation through an orthogonal matrix: the orthogonal matrix is:

[0044] U=e iA ;

[0045] in, W is the trainable weight and H is the conjugate transpose of the complex matrix.

[0046] In one possible implementation, the feature extraction network includes a transformer layer, the transformer layer includes a normalization layer, and the normalization layer is used to perform length unit processing on the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

[0047] In one possible implementation, the feature extraction network includes a transformer layer, the transformer layer includes a feed-forward layer FFN, the FFN includes an activation layer, and the activation layer is used to perform nonlinear activation on part of the data input into the activation layer, the part of the data does not include data corresponding to the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

[0048] In a possible implementation, the portion of data input into the activation layer is a complex number, and the activation layer is specifically configured to perform nonlinear activation on the real part and the imaginary part of the portion of data, respectively.

[0049] In a possible implementation, the prediction network includes a target fully connected layer, where the target fully connected layer is a fully connected layer close to an output layer in the prediction network, and the target fully connected layer includes a trainable first unit vector and a second unit vector;

[0050] The prediction module is specifically used to:

[0051] The result after the orthogonal transformation is respectively operated with the first unit vector and the second unit vector to obtain the first probability and the second probability, where the first unit vector corresponds to the first probability and the second unit vector corresponds to the second probability, the first probability represents the probability that the text prediction result belongs to the target label, and the second probability represents the probability that the text prediction result does not belong to the target label;

[0052] A text prediction result is determined according to the first probability and the second probability.

[0053] In one possible implementation, the feature extraction network includes a transformer layer, and the transformer layer includes an attention head;

[0054] The head is used to obtain the K matrix and Q matrix of the text to be processed;

[0055] Calculating the K matrix and the Q matrix to obtain first data, where the first data is a complex number;

[0056] Mapping the values ​​of the real part and the imaginary part of the first data into second data, where the second data is a real number;

[0057] Perform a softmax operation on the second data.

[0058] In a possible implementation, mapping the values ​​of the real part and the imaginary part of the first data into second data includes:

[0059] The modulus of the first data is determined as the second data according to values ​​of the real part and the imaginary part of the first data.

[0060] In a possible implementation, the apparatus further includes:

[0061] A model updating module, configured to determine a target loss based on the text prediction result;

[0062] Backpropagation of the pre-trained language model is performed according to the target loss, wherein the gradient and momentum used in the backpropagation are complex numbers.

[0063] In a third aspect, an embodiment of the present application provides an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the first aspect and any optional method thereof.

[0064] In a fourth aspect, an embodiment of the present application provides a training device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the first aspect and any optional method thereof.

[0065] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0066] In a sixth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0067] In a seventh aspect, the present application provides a chip system comprising a processor for supporting an execution device or a training device in implementing the functions described in the aforementioned aspects, such as transmitting or processing data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory for storing program instructions and data necessary for the execution device or the training device. The chip system may consist of a single chip or may include a chip and other discrete components.

[0068] The embodiment of the present application provides a data processing method, the method comprising: obtaining a text to be processed and a pre-trained language model, the pre-trained language model comprising a feature extraction network and a prediction network; extracting features from the text to be processed through the feature extraction network to obtain a feature representation of the data to be processed, the feature representation being a complex number; and performing an orthogonal transformation on the feature representation after length unitization through the prediction network to obtain a result after the orthogonal transformation, and determining a text prediction result based on the result after the orthogonal transformation. In the embodiment of the present application, compared with the W*a+b operation adopted by the existing prediction network, the orthogonal transformation can be adapted to quantum computing of quantum circuits, and the quantum state obtained by the orthogonal transformation can pass through the quantum measurement layer, that is, measure the probability of collapse to each quantum ground state, and then perform prediction network operations on the quantum circuit, and the construction of the pre-trained language model represented by the complex value improves the representation capability of the model and the performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0070] Figure 2 It is a natural language processing system;

[0071] Figure 3a is another natural language processing system;

[0072] Figure 3b A schematic diagram of a system architecture;

[0073] Figure 4 A schematic diagram of a device related to natural language processing provided in an embodiment of the present application;

[0074] Figure 5 This is a schematic diagram of the architecture of a transformer layer;

[0075] Figure 6 An example of a data processing method provided in an embodiment of the present application is shown;

[0076] Figure 7 This is a schematic diagram of the structure of a neural network model in an embodiment of the present application;

[0077] Figure 8 This is a schematic diagram of the structure of a transformer layer;

[0078] Figure 9 This is a schematic diagram of the operation of an attention head;

[0079] Figure 10 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0080] Figure 11 An example of a data processing method provided in an embodiment of the present application is shown;

[0081] Figure 12 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0082] Figure 13 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0083] Figure 14 This is a structural diagram of a training device provided in an embodiment of the present application;

[0084] Figure 15 A schematic diagram of the structure of the chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0086] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0087] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0088] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1The following diagram illustrates a structural diagram of the AI ​​framework. This framework is explained below from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed progression from "data-information-knowledge-wisdom." The "IT value chain," encompassing the entire process from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0089] (1) Infrastructure

[0090] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0091] (2) Data

[0092] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0093] (3) Data processing

[0094] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0095] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0096] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0097] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0098] (4) General ability

[0099] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0100] (5) Smart products and industry applications

[0101] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0102] This application can be applied to the field of natural language processing in the field of artificial intelligence. The following will introduce multiple application scenarios of multiple products.

[0103] In order to better understand the solution of the embodiment of the present application, Figure 1 FIG3 briefly introduces possible application scenarios of the embodiment of the present application.

[0104] Figure 2 A natural language processing system is shown, comprising a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, personal computer, or information processing center. The user device is the initiator of natural language data processing, initiating requests such as language questions and answers or inquiries. Typically, users initiate requests through the user device.

[0105] The aforementioned data processing devices can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing devices receive query statements, voice, text, and other information from smart terminals via interactive interfaces. They then use their memory and processors to perform language data processing, including machine learning, deep learning, search, reasoning, and decision-making, and then feed the results back to the user device. The memory in a data processing device is a general term encompassing both local storage and databases storing historical data. The databases can be located on the data processing device or on other network servers.

[0106] exist Figure 2In the natural language processing system shown, the user device can receive user instructions. For example, the user device can receive a piece of text input by the user, and then initiate a request to the data processing device, so that the data processing device executes a natural language processing application (such as natural language generation, text classification, text reasoning, named entity recognition, translation, etc.) for the piece of text obtained by the user device, thereby obtaining the processing results of the corresponding natural language processing application for the piece of text (such as predicted word results, classification results, reasoning results, named entity recognition results, translation results, etc.).

[0107] Taking natural language generation (NLG) as an example, also known as text prediction or natural language synthesis, it involves generating missing or subsequent text within a given text segment. NLG is widely used in search engines, input methods, and other scenarios. It can predict a user's next input based on partial text, significantly improving user efficiency. It can also recover text with missing text.

[0108] Illustratively, in an embodiment of the present application, a user device may receive a piece of text data input by a user, wherein the text data includes known words and words to be predicted, the words to be predicted are invisible, and only the position of the words to be predicted in the text data is known. The user device may then initiate a request to a data processing device (the request carries text data), so that the data processing device predicts the words to be predicted in the text data, thereby obtaining the words to be predicted, and feeding back the words to be predicted to the user device.

[0109] For example, the user device may receive a piece of text data input by the user, and then initiate a request to the data processing device, so that the data processing device performs entity classification on the piece of text data, thereby obtaining an entity classification result for the piece of text data, and feeding the entity classification result back to the user device;

[0110] For example, the user device can receive a piece of text data input by the user (the text data is Chinese text), and then initiate a request to the data processing device, so that the data processing device translates the piece of text data into English, thereby obtaining an English translation for the piece of text data, and feeding the English translation back to the user device.

[0111] Figure 3a Another natural language processing system is shown in Figure 3a In the process, the user device directly acts as a data processing device. The user device can directly receive input from the user and process it directly by the hardware of the user device itself. The specific process is the same as Figure 2 Similarly, please refer to the above description and will not be repeated here.

[0112] Figure 4 It is a schematic diagram of a natural language processing related device 300 provided in an embodiment of the present application.

[0113] above Figure 2 and Figure 3a The user equipment in the example may be Figure 4 The local device 301 or the local device 302 in Figure 2 The data processing device in the embodiment can be Figure 4 The execution device 310 in the embodiment, wherein the data storage system 350 can store the data to be processed by the execution device 310, and the data storage system 350 can be integrated on the execution device 310, or can be set on the cloud or other network servers.

[0114] Figure 2 and Figure 3a The processor in the system can perform data training / machine learning / deep learning through neural network models or other models, and use the model finally trained or learned from the data to perform natural language processing applications (such as natural language generation, text classification, sequence labeling, reading comprehension, text generation, text reasoning, translation, etc.) on text data to obtain corresponding processing results.

[0115] Among them, the high-precision model after fine-tuning the pre-trained language model in the embodiment of the present application can be deployed in a data processing device, and the data processing device can provide a high-precision model to process text data to obtain the processing results of the above-mentioned natural language processing application.

[0116] The following combination Figure 3b The system architecture provided in the embodiments of the present application is introduced in detail. Figure 3b This is a schematic diagram of the system architecture provided by an embodiment of the present application. Figure 3b As shown, the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data collection system 560 .

[0117] The execution device 510 includes a calculation module 511, an I / O interface 512, a pre-processing module 513, and a post-processing module 514. The calculation module 511 may include the target model / rule 501, and the pre-processing module 513 and the post-processing module 514 are optional.

[0118] The data acquisition device 560 is used to collect training data.

[0119] In the task of natural language synthesis, the training data may be text data with missing text and complete text data corresponding to the text data with missing text.

[0120] In translation tasks, training data may include but is not limited to parallel corpora, monolingual corpora, etc.

[0121] Parallel corpus refers to bilingual or multilingual corpora (i.e., annotated text data) consisting of a source text and its corresponding target text. The source and target texts share the same semantics, and there is a correspondence between the text units. For example, if the source text is "This trip needs careful planning," and its corresponding English text is "The trip needs careful planning," then "This trip needs careful planning" and "The trip needs careful planning" can be considered a parallel corpus. This parallel corpus is a Chinese-English parallel language pair. The source text "This trip needs careful planning" can be considered the source text of this parallel corpus, and the target text "The trip needs careful planning" can be considered the target text of this parallel corpus. Here, "travel" can correspond to "trip."

[0122] In addition, “This trip needs careful planning” can be regarded as a monolingual corpus, and “The trip needs careful planning” can also be regarded as a monolingual corpus.

[0123] After collecting the training data, the data collection device 560 stores the training data in the database 530 , and the training device 520 obtains the target model / rule 501 through training based on the training data maintained in the database 530 .

[0124] Among them, the training device 520 trains the pretrained language model (PLM) in the embodiment of the present application based on the training data maintained in the database 530 to obtain the target model / rule 501.

[0125] In order to adapt to downstream tasks, the training device 520 can fine-tune the trained pre-trained language model based on the training data maintained in the database 530 to obtain the target model / rule 501.

[0126] It should be understood that the training device 520 for training the pre-trained language model and the training device 520 for fine-tuning the trained pre-trained language model may be different devices.

[0127] It should be noted that, in actual applications, the training data maintained in the database 530 may not all be collected by the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model / rule 501 entirely based on the training data maintained by the database 530, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0128] The target model / rule 501 obtained by training the training device 520 can be applied to different systems or devices, such as Figure 3b The execution device 510 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle terminal, etc. It can also be a server or a cloud. Figure 3b In the embodiment, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with an external device. A user can input data to the I / O interface 512 through a client device 540 .

[0129] Preprocessing modules 513 and 514 are used to perform preprocessing based on the input data received by I / O interface 512 (e.g., obtaining the positions of known data units and data units to be predicted in the target data, or generating attention information, etc.). It should be understood that preprocessing modules 513 and 514 may be absent, or there may be only one preprocessing module. If preprocessing modules 513 and 514 are not present, computing module 511 may be used directly to process the input data.

[0130] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.

[0131] Finally, the I / O interface 512 presents the processed results to the client device 540 so as to be provided to the user.

[0132] exist Figure 3bIn the illustrated case, the user can manually input data, and this "manual input data" can be operated through the interface provided by I / O interface 512. In another case, client device 540 can automatically send input data to I / O interface 512. If the automatic transmission of input data by client device 540 requires user authorization, the user can set the corresponding permissions in client device 540. The user can view the results output by execution device 510 on client device 540, and the specific presentation form can be a display, sound, action, etc. Client device 540 can also serve as a data acquisition terminal, collecting input data input into I / O interface 512 and output results from I / O interface 512 as new sample data and storing them in database 530. Of course, collection can also be performed without client device 540, and instead the I / O interface 512 directly stores the input data input into I / O interface 512 and output results from I / O interface 512 as new sample data in database 530.

[0133] It is worth noting that Figure 3b This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 3b In the embodiment, the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510.

[0134] It should be understood that the execution device 510 may also be deployed in the client device 540 .

[0135] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0136] (1) Neural Network

[0137] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0138]

[0139] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0140] (2) Transformer layer

[0141] Reference Figure 5 , Figure 5 This is a schematic diagram of the architecture of a transformer layer, such as Figure 5 As shown, the neural network includes an embedding layer and at least one transformer layer, and the at least one transformer layer can be N transformer layers (N is an integer greater than 0), wherein each transformer layer includes an attention layer, an add & norm layer, a feed forward layer, and an add & normalization layer that are adjacent in sequence. In the embedding layer, the current input is embedded to obtain multiple embedding vectors; in the attention layer, P input vectors are obtained from the previous layer of the first transformer layer, and with any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, the intermediate vector corresponding to the first input vector is obtained, thereby determining the P intermediate vectors corresponding to the P input vectors; in the pooling layer, the P intermediate vectors are merged into Q output vectors, wherein the multiple output vectors obtained by the last transformer layer in the transformer layer are used as feature representations of the current input.

[0142] (3) Attention mechanism

[0143] The attention mechanism mimics the internal process of biological observation behavior, namely, a mechanism that aligns internal experience and external sensations to increase the observation precision of certain areas. It can quickly filter out high-value information from a large amount of information using limited attention resources. The attention mechanism can quickly extract important features from sparse data and is therefore widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement on the attention mechanism, which reduces dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of ​​the attention mechanism can be rewritten as the following formula:

[0144] Here, Lx = ||Source|| represents the length of the Source. This formula implies that the elements in the Source are imagined to consist of a series of data pairs. Given a Query element in the target, the similarity or correlation between the Query and each Key is calculated to obtain the weight coefficient for each Key's corresponding Value. The weighted sum of the Values ​​is then taken to obtain the final Attention value. Essentially, the Attention mechanism performs a weighted sum of the Values ​​of the Source elements, with the Query and Key used to calculate the weight coefficient for the corresponding Value. Conceptually, Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on this important information, while ignoring the majority of less important information. This focusing process is reflected in the calculation of the weight coefficients: the larger the weight, the more focus is placed on the corresponding Value. In other words, the weight represents the importance of the information, while the Value represents the corresponding information. The self-attention mechanism can be understood as internal attention. The attention mechanism occurs between the Query element of the Target and all elements of the Source. The self-attention mechanism refers to the attention mechanism that occurs between the internal elements of the Source or the internal elements of the Target. It can also be understood as the attention calculation mechanism in the special case of Target = Source. The specific calculation process is the same, only the calculation object has changed.

[0145] (4) Natural language processing (NLP)

[0146] Natural language refers to human language, and natural language processing (NLP) is the processing of human language. Natural language processing is the process of systematically analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. By using NLP and its components, we can manage very large amounts of text data, perform a large number of automated tasks, and solve a wide variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering, and topic segmentation.

[0147] (5) Pre-trained language model

[0148] A pretrained language model is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction. Its training consists of two phases. In the pre-training phase, the model is trained on a large amount of unsupervised text for language modeling tasks, thereby learning a word representation. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase and trained in a relatively short number of steps on downstream tasks such as text classification and sequence labeling. This allows the semantic information gained from pre-training to be successfully transferred to the downstream tasks.

[0149] It should be understood that the above architecture can also be applied to other natural language processing tasks, such as natural language synthesis, semantic understanding, summary generation, etc.

[0150] First, the data processing method provided in the embodiment of the present application is described by taking the model training stage as an example.

[0151] Reference Figure 6 , Figure 6 The embodiment of the data processing method provided in the embodiment of the present application is shown in FIG. The data processing method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablets, laptops, smart wearable devices, etc., and can also be applied to servers, such as Figure 6 As shown, a data processing method provided by an embodiment of the present application includes:

[0152] 601. Obtain a text to be processed and a pre-trained language model, where the pre-trained language model includes a feature extraction network and a prediction network.

[0153] In an embodiment of the present application, the training device can obtain the text to be processed and the pre-trained language model, wherein the pre-trained language model is a transformer model that can perform multi-tasking processing.

[0154] First, the text to be processed in the embodiment of the present application is introduced.

[0155] In one possible implementation, the text to be processed can be a training sample for a pre-trained language model, wherein the training sample can include a first data sequence and a second data sequence, the first data sequence can be obtained based on the source corpus, and the second data sequence can be obtained based on the target corpus, and the pre-trained language model needs to predict and generate the target corpus based on the source corpus.

[0156] In one possible implementation, a pre-trained language model can be used to implement sequence conversion tasks between different language types, such as text translation tasks, summary generation tasks between different languages, etc., in which case the first data sequence and the second data sequence can be texts including different language types (it is not limited to each data unit in the first data sequence being of a different language type from the data units in the second data sequence; for example, some data units in the first data sequence and the data units (partial or all data units) in the second data sequence are of the same language type). The language type can also be referred to as a language.

[0157] For example, in a Chinese-English translation task, the original text is "This trip requires careful planning", and its parallel English text is "The trip needs careful planning". Then "This trip requires careful planning" and "The trip needs careful planning" can be regarded as a set of parallel corpora, which is a Chinese-English parallel language pair. The original text "This trip requires careful planning" can be regarded as the source corpus of this set of parallel corpora, and the translated text "The trip needs careful planning" can be regarded as the target corpus of this set of parallel corpora.

[0158] For example, in an English-German translation task, the original text is "We danse on the grass" and its parallel German text is "Wir tanzen auf dem gras". Then "We danse on the grass" and "Wir tanzenauf dem gras" can be regarded as a set of parallel corpora, which is an English-German parallel language pair. The original text "We danse on the grass" can be regarded as the source corpus of this set of parallel corpora, and the translated text "Wir tanzen auf demgras" can be regarded as the target corpus of this set of parallel corpora.

[0159] In a possible implementation, the first data sequence before the mask operation and the second data sequence before the mask operation are different data sequences that have been sample labeled.

[0160] In one possible implementation, the pre-trained language model can be used to implement a text summary generation task, where the source corpus can be the source corpus from which a summary needs to be extracted, and the target corpus can be the summary text to be generated.

[0161] In one possible implementation, a pre-trained language model can be used to implement a text reply task, where the source corpus can be the source corpus that needs to be replied to, and the target corpus can be the reply content for the source corpus.

[0162] In a possible implementation, the first data sequence before the mask operation and the second data sequence before the mask operation are the same data sequence, that is, the first data sequence before the mask operation and the second data sequence before the mask operation are unmarked data.

[0163] In one possible implementation, the first data sequence can be obtained by masking the original source corpus, and the second data sequence can be obtained by masking the original target corpus. If the pre-trained language model can be used to implement sequence conversion tasks (e.g., translation tasks) between texts in different language types, the original source corpus and the original target corpus can be texts expressed in different language types.

[0164] Alternatively, the original source corpus and the original target corpus can be obtained from an external database.

[0165] In one possible implementation, a masking operation may be performed on the original source corpus and the original target corpus to obtain training data for the pre-trained language model.

[0166] In a possible implementation, the text to be processed may be a first text sequence and a second text sequence, and the pre-trained language model may be used to identify whether the second text sequence is the continuation of the first text sequence.

[0167] In one possible implementation, after obtaining the text to be processed, the text to be processed can be embedded through the embedding layer in the pre-trained language model (this embodiment of the present application may also be referred to as a plural word embedding module) to obtain an embedding vector.

[0168] Optionally, the plural word embedding module can map the token embedding, position embedding, and segment embedding (text embedding is optional) of each data unit of the text to be processed into complex-valued vectors of fixed dimensions, and fuse the above-obtained vectors as the semantic vector of the data unit.

[0169] In one possible implementation, the embedding layer may include an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing may be performed on each data unit in the unmasked data units in the current input, thereby obtaining a word vector for each data unit in the unmasked data units (e.g., which may represent semantic information). In the positional encoding layer, the position of each data unit in the unmasked data units in the current input may be obtained, thereby generating a position vector for the position of each data unit in the unmasked data units.

[0170] In some examples, the position information of each data unit in the unmasked data units in the data sequence can be the absolute position of each data unit in the unmasked data units in the data sequence. Taking the current input of "Which number should I repay Huabei" as an example, the position of "how many" can be represented as the first digit, the position of "number" can be represented as the second digit, and so on. In some examples, the position of each data unit in the unmasked data units in the data sequence can be the relative position of each data unit in the unmasked data units in the data sequence. Still taking the current input of "Which number should I repay Huabei" as an example, the position of "how many" can be represented as before "number", the position of "number" can be represented as after "how many" and before "should", and so on. When the word vector and position vector of each data unit in the unmasked data units in the current input are obtained, the position vector and the corresponding word vector of each data unit in the unmasked data units can be fused to obtain the embedding vector of each data unit in the unmasked data units. It should be understood that the fusion method can be to add the position vector and the corresponding word vector, or through other operations, and the specific fusion method is not limited here. The embedding vector can be represented as an embedding matrix with a preset dimension. The number of embedding vectors can be set to M and the preset dimension is H, so the embedding vector can be represented as an M×H embedding matrix.

[0171] In an embodiment of the present application, the feature extraction network may include multiple stacked transformer layers. In other words, the pre-trained language model may be a neural network model based on the transformer layer. Optionally, the pre-trained language model may be an NLP model based on the transformer layer.

[0172] 602. Perform feature extraction on the text to be processed through the feature extraction network to obtain a feature representation of the data to be processed, where the feature representation is plural.

[0173] Next, we describe an example structure of a pre-trained language model:

[0174] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of a neural network model in an embodiment of the present application. Figure 7 The neural network model shown can be the pre-trained language model in the embodiment of this application. Figure 7 As shown in , the pre-trained language model may include an embedding layer and multiple transformer layers (also referred to as a feature extraction network in the embodiment of the present application) connected in sequence. As those skilled in the art will appreciate, transformer models are often used to perform natural language processing (NLP) tasks. It is important to understand that Figure 7The structure is only an example. The number of transformer layers can be set as needed. For example, only one transformer layer can be set, or more transformer layers can be set. The neural network model determines the feature vector corresponding to the current node based on the N output vectors obtained by each transformer layer.

[0175] The specific working process of each layer is described below.

[0176] About the Embedding layer:

[0177] In the embedding layer (or complex embedding layer), the current input is embedded to obtain multiple feature vectors (the vector is a complex vector (or can be called a complex-valued vector), optionally, a complex-valued vector of fixed dimension). The core feature of the transformer model is its unique attention mechanism. When processing natural language, such as a sentence, the transformer model uses this attention mechanism to assign different attention coefficients to each word vector in the sentence, thereby more comprehensively considering the impact of the context in the sentence on each word. The embedding layer is based on the node features and position encoding of each node in the current sequence to obtain N embedding vectors X l The attention layer is connected to the embedding layer, taking N embedding vectors from the embedding layer as input vectors. Based on the correlation between each of the N input vectors, it synthesizes each input vector to obtain N output vectors, which are then output to the subsequent transformer layer. The transformer layer takes the output of the previous layer as its input vector and performs similar operations as the previous transformer layer.

[0178] About feature extraction network:

[0179] In one possible implementation, the feature extraction network may include multiple transformer layers.

[0180] Refer to 7 and Figure 8 , Figure 7 and Figure 8 The structure of the transformer layer is shown in FIG. 1 . The transformer layers of the various neural networks in the embodiments of the present application can refer to FIG. Figure 8 The structure shown in Figure 8 As shown in , the transformer layer includes a complex multi-head attention layer, a complex addition and normalization (add&norm) layer, a complex feed forward layer, and a complex addition and normalization layer.

[0181] Among them, the complex multi-head attention layer obtains N input vectors X from its upper layer l , which can also be expressed as matrix X. Using the self-attention mechanism, each vector is transformed based on the correlation between vectors to obtain N output vectors, which can also be expressed as matrix Y. It can be understood that when the multi-head attention layer is a layer directly connected to the embedding layer, for example Figure 7 The input vector obtained by the transformer layer directly connected to the embedding layer is the embedding vector output by the embedding layer; when the multi-head attention layer is a multi-head attention layer included in the subsequent transformer layer, for example Figure 7 The transformer layer directly connected to the previous transformer layer includes a multi-head attention layer, whose input vector is the output vector of the previous transformer layer. In the multi-head attention layer, the MHA layer based on multi-head attention (MHA) includes multiple attention heads (such as Figure 8 Head 1, Head 2, ..., Head N) shown in .

[0182] Figure 9 This is a schematic diagram of the operation of an attention head, which shows how the attention head transforms the input matrix X into the output matrix Y. Figure 9 As shown, the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V are used to transform N input vectors respectively.<X1,X2,…,XN> Each input vector Xi is transformed to obtain the first intermediate vector (q vector), the second intermediate vector (k vector) and the third intermediate vector (v vector) corresponding to each input vector. Operationally, the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V can be used to perform linear transformation on the input matrix X composed of N input vectors, respectively, to obtain the Q matrix, the K matrix and the V matrix of the input matrix, and then the matrix is ​​split respectively to obtain the q vector, the k vector and the v vector corresponding to each input vector. For any i-th input vector Xi among the N input vectors, based on the dot product operation of the first intermediate vector (q vector, qi) corresponding to the i-th input vector and the second intermediate vector (k vector, kj) corresponding to each input vector Xj, the respective correlation degrees of the i-th input vector Xi and each input vector Xj are determined. Although the dot product of qi and kj can be directly used as the correlation, the more classic approach is to first divide the dot product by a constant, then perform a softmax operation, and use the result as the correlation between the input vectors Xi and Xj, that is (the following formula is the attention mechanism operation process performed by the real number head. Complex numbers will have corresponding differences. The attention mechanism related to complex numbers will be described later):

[0183]

[0184] Therefore, the correlation degrees αi,j between the i-th input vector Xi and each input vector Xj can be used as weight factors to perform a weighted combination on the third intermediate vectors (v vectors, vj) corresponding to each input vector Xj to obtain the i-th combination vector Ci corresponding to the i-th input vector Xi:

[0185]

[0186] Thus, we can get the vector sequence of N combination vectors corresponding to N input vectors<C1,C2,…,CN> , or matrix C. Based on this combination vector sequence, N output vectors can be obtained. Specifically, in one embodiment, the vector sequence of N combination vectors can be directly used as N output vectors, that is, Yi=Ci. At this time, the output matrix Y is the combination vector matrix C, which can also be written as:

[0187]

[0188] It should be understood that since the Q matrix, K matrix and V matrix in the embodiments of the present application are complex matrices, that is, each element therein is a complex number including a real part and an imaginary part, it is necessary to adopt an attention mechanism suitable for complex numbers.

[0189] In one possible implementation, since the softmax operation needs to be performed in the real number domain, and the result of the operation between the Q matrix and the K matrix (that is, the object of the softmax operation) is a complex number, the result (complex number) of the operation between the Q matrix and the K matrix can be mapped to the real number domain in an embodiment of the present application.

[0190] In one possible implementation, taking the result of the operation between the Q matrix and the K matrix as the first data as an example, the real part and the imaginary part can be mapped to the second data (real number) according to the values ​​of the real part and the imaginary part of the first data. For example, a preset operation can be performed, that is, a numerical operation can be performed on the values ​​of the real part and the imaginary part to obtain a real number value as the second data.

[0191] In one possible implementation, the modulus of the first data can be determined as the second data based on the values ​​of the real part and the imaginary part of the first data. Since in the transformer model based on complex numbers, the modulus of the complex number and the probability of the final output are related, that is, the modulus of the complex number itself has a definite physical meaning. In the process of mapping the real part and the imaginary part to the second data, the mapping method of the complex modulus can increase the interpretability of the network and improve the accuracy of the network.

[0192] Specifically, in a possible implementation, a head (an attention head in a multi-head attention) can be used to obtain the K matrix and the Q matrix of the text to be processed; the K matrix and the Q matrix are calculated to obtain first data, which is a complex number; the numerical values ​​of the real part and the imaginary part of the first data are mapped to second data, which is a real number; and a softmax operation is performed on the second data.

[0193] In one possible implementation, the complex attention mechanism operates as follows:

[0194]

[0195] in, It can be understood as calculating the K matrix and the Q matrix to obtain the first data, which is a complex number; wherein f() can be understood as mapping the numerical values ​​of the real part and the imaginary part of the first data to the second data, which is a real number; and performing a softmax operation on the second data.

[0196] For example, f() can be understood as a softmax function with a complex modulus, and H is the conjugate transpose of a complex matrix. Complex multi-head attention can be achieved by applying the above formula to each element to obtain many low-dimensional semantic vectors, which are then concatenated to make the output vector have the same dimension as the input.

[0197] The above is a description of the processing of an attention head. In the MHA architecture, the MHA layer maintains m sets of transformation matrices. Each set of transformation matrices includes the aforementioned first transformation matrix Q, second transformation matrix K, and third transformation matrix V, so that the above operations can be performed in parallel to obtain m combination vector sequences (i.e., m matrices C). Each vector sequence includes N combination vectors obtained based on a set of transformation matrices. In this case, the MHA layer splices the obtained m combination vector sequences to obtain a spliced ​​matrix; then transforms the spliced ​​matrix through the fourth transformation matrix W to obtain the final output matrix Y. Splitting the output matrix Y corresponds to N output vectors.<Y1,Y2,…,YN> Through the above operation process, the MHA layer performs transformation operations based on the correlation between N input vectors to obtain N output vectors.

[0198] In one possible implementation, the feature extraction network includes a plurality of transformer layers, wherein the transformer layer includes a feed-forward network (FFN), and the FFN may include an activation layer, and the activation layer is used to perform nonlinear activation on part of the data input into the activation layer, and the part of the data does not include the data corresponding to the target character, and the target character is the CLS character (or the [CLS] flag) inserted at the starting position of the text to be processed. The [CLS] flag is placed at the beginning of the first sentence, which means classification, and can be used for downstream classification tasks. In order to be close to the linear rotation operation in the quantum circuit, the nonlinear activation function for the [CLS] character can be removed, for example, it can include the activation function in the complex feedforward network and the activation function in the prediction network (such as the Tanh function). For example, referring to Figure 10 , Figure 10 Schematic diagram of a quantum-adapted pre-trained language network.

[0199] In a possible implementation, the portion of data input into the activation layer is a complex number, and the activation layer is specifically used to perform nonlinear activation on the real part and the imaginary part of the portion of data respectively.

[0200] Reference Figure 7 and Figure 8 The complex transformer may include a complex feedforward network module, which may include two fully connected layers with a nonlinear activation function in between. The complex nonlinear activation function here may activate the real and imaginary parts of the input data using an activation function (e.g., GELU) respectively.

[0201] In one possible implementation, the complex attention head and the complex feedforward network can be separated by a residual module ( Figure 7 Not shown, Figure 8 As shown in the figure, they can be connected.

[0202] In one possible implementation, a quantum-adapted pre-trained language model can be built upon the aforementioned model and adapted for quantum computation. To ensure the mathematical validity of this quantum computation mapping, the intermediate representation of the complex network's [CLS] character can be constrained to be unitary, allowing it to be considered a quantum state throughout the network, facilitating adaptation to quantum circuits.

[0203] In one possible implementation, the normalization layer may change the operation on CLS characters to a length unitization operation.

[0204] Specifically, in one possible implementation, the feature extraction network includes a transformer layer, which includes a normalization layer. The normalization layer is used to perform length unitization processing on a target character, where the target character is a CLS character inserted at the beginning of the text to be processed. The multi-layer normalization operation can refer to the following formula:

[0205]

[0206] in and σ z are the mean and standard deviation of the complex sequence, respectively.

[0207] In a possible implementation, during the feedforward process of model training, the feature extraction network can be used to perform feature extraction on the text to be processed to obtain a feature representation of the data to be processed, where the feature representation is plural.

[0208] 603. Perform an orthogonal transformation on the feature representation after the length unitization processing through the prediction network to obtain an orthogonal transformation result, and determine a text prediction result based on the orthogonal transformation result.

[0209] In one possible implementation, in the feedforward process of the model, feature representation is obtained through the feature extraction network, and the feature representation can be input into the prediction network, and then the prediction network can determine the text prediction result based on the feature representation.

[0210] In one possible implementation, the pre-trained language model is used to perform a target task, and the text processing result is the processing result of the target task; the target task is one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language reasoning, text automatic question answering, text intent recognition, text classification, text simplification, and text story generation.

[0211] In one possible implementation, the prediction network may include a fully connected layer and an activation layer.

[0212] In one possible implementation, the fully connected layers in the prediction network can be connected through activation layers, and the module length of the output close to the fully connected layer is normalized by softmax to obtain the probability distribution of each category.

[0213] For example, two fully connected layers can be added to the output of the last complex transformer layer for the prediction network of the mask language model (MLM) and next sentence prediction (NSP) tasks, connected by a Tanh nonlinear activation function for the real and imaginary parts. The final output of both networks is the probability distribution of their respective categories, obtained by normalizing the modulus of the output of the last complex fully connected layer through softmax.

[0214] In one possible implementation, in order to adapt the prediction network to the quantum adaptation structure of the quantum circuit, the input quantum state (unit complex vector) can first be orthogonally transformed (the parameters of the orthogonal transformation can be trained). The parameterization method of the orthogonal transformation layer is as follows:

[0215]

[0216] Here, A is a complex positive definite matrix, which is transformed into an orthogonal matrix U through matrix exponential operations. The network's errors can be backpropagated into the weights W, allowing the entire orthogonal transformation to be trained through backpropagation. The quantum state that has undergone the orthogonal transformation passes through a quantum measurement layer, which measures the probability of collapse to each quantum ground state. The square of the modulus of the complex vector corresponding to the quantum state is the measured probability vector. This probability vector is then passed through a linear projection layer to obtain the sentence's category label.

[0217] In the embodiment of the present application, compared with the W*a+b operation used in the existing prediction network, the use of orthogonal transformation can be adapted to the quantum computing of the quantum circuit. The quantum state obtained by the orthogonal transformation can pass through the quantum measurement layer, that is, measure the probability of collapse to each quantum ground state, and then the prediction network operation can be performed on the quantum circuit.

[0218] Specifically, in a possible implementation, performing an orthogonal transformation on the feature representation after the unitization process includes: performing an orthogonal transformation on the feature representation after the unitization process through an orthogonal matrix: the orthogonal matrix is:

[0219] U=e iA ;

[0220] in, W is the trainable weight and H is the conjugate transpose of the complex matrix.

[0221] It should be understood that the above-mentioned orthogonalization calculation process can be used during model pre-training or model fine-tuning.

[0222] In one possible implementation, the prediction network includes a target fully connected layer, which is a fully connected layer close to the output layer in the prediction network, and the target fully connected layer includes a trainable first unit vector and a second unit vector; the result after the orthogonal transformation can be respectively operated with the first unit vector and the second unit vector to obtain the first probability and the second probability, the first unit vector corresponds to the first probability, and the second unit vector corresponds to the second probability, the first probability represents the probability that the text prediction result belongs to the target label, and the second probability represents the probability that the text prediction result does not belong to the target label; the text prediction result is determined based on the first probability and the second probability.

[0223] Through the above method, the last fully connected layer of the prediction network is also changed to a form similar to quantum measurement, that is, two unit vectors are trained as measurement states, the inner product of the input representation and these two vectors is calculated respectively, and the square of the inner product is linearly normalized to a probability value. Optionally, the probability value can be used to calculate the cross entropy with the binary classification label as the loss function.

[0224] 604. Determine a target loss according to the text prediction result;

[0225] 605. Perform back propagation of the pre-trained language model according to the target loss, wherein the gradient and momentum used in the back propagation are complex numbers.

[0226] The training process of the pre-trained language model in the embodiment of the present application can be divided into two processes: pre-training and fine-tuning (steps 604 and 605 can be performed during the pre-training or fine-tuning of the pre-trained language model). The pre-training and fine-tuning processes of the two networks can both use the classic backpropagation algorithm to train the network weights, specifically using an improved complex optimizer. The difference between this optimizer and the real optimizer can be shown in the following pseudo code:

[0227]

[0228] The parameters in line 2 are adjusted to complex numbers, and line 9 is modified to complex conjugate multiplication. In addition, the pre-training and fine-tuning processes of the two pre-trained language models use the default regular distribution for weight initialization, without any orthogonal regularization constraints.

[0229] Exemplarily, both models are pre-trained on a large-scale English corpus. The pre-trained models are then fine-tuned for downstream text classification and semantic matching tasks. During fine-tuning, the task-specific network structure is connected after the multi-layer transformer output and trained on a specific dataset alongside the remaining pre-trained network structure. Specifically, for the quantum-adapted and trained language model, the fine-tuned structure is the neural network described in the above embodiments, which can be implemented using quantum circuits.

[0230] The embodiment of the present application provides a data processing method, the method comprising: obtaining a text to be processed and a pre-trained language model, the pre-trained language model comprising a feature extraction network and a prediction network; extracting features from the text to be processed through the feature extraction network to obtain a feature representation of the data to be processed, the feature representation being a complex number; and performing an orthogonal transformation on the feature representation after length unitization through the prediction network to obtain a result after the orthogonal transformation, and determining a text prediction result based on the result after the orthogonal transformation. In the embodiment of the present application, compared with the W*a+b operation adopted by the existing prediction network, the orthogonal transformation can be adapted to quantum computing of quantum circuits, and the quantum state obtained by the orthogonal transformation can pass through the quantum measurement layer, that is, measure the probability of collapse to each quantum ground state, and then perform prediction network operations on the quantum circuit, and the construction of the pre-trained language model represented by the complex value improves the representation capability of the model and the performance of the network.

[0231] Next, the data processing method in the embodiment of the present application is introduced with reference to a specific example.

[0232] Taking the application in the GLUE evaluation benchmark set as an example, the complex-valued pre-trained language model CVBERT-base and the quantum-adapted pre-trained language model QBERT-base were trained on a mixed corpus consisting of Wikipedia and BookCorpus. Both models have 12 transformer layers, each with 12 self-attention heads. The model dimensions d of the two models are model =768, middle layer dimension d hidden =1536, and remove W from each self-attention mechanism Q and W O The projection matrix is ​​used to make the two models similar in size to BERT-base. For each GLUE dataset, the two models fine-tune the newly added network structure and the previously pre-trained network structure on the training set and output the performance on the validation set. The average performance across all GLUE datasets is used as a fairness metric to evaluate the pre-trained language models.

[0233]

[0234] The table above compares the performance of the two invented models with BERT-base. An end-to-end quantum-adapted NLP model was also constructed, trained on the same dataset, and tested for performance. It can be seen that the complex-valued pre-trained language model performs slightly better than the real-valued network. Due to the added constraints, the quantum-adapted model exhibits some performance degradation compared to these two models, but achieves significant performance improvements over the end-to-end quantum model across all tasks, achieving a 50%-60% average performance improvement.

[0235] Next, the data processing method in the embodiment of the present application is introduced from the perspective of model reasoning:

[0236] Reference Figure 11 , Figure 11 A data processing method according to an embodiment of the present invention is shown in FIG. Figure 11 As shown, the method includes:

[0237] 1101. Obtain a text to be processed and a pre-trained language model, where the pre-trained language model includes a feature extraction network and a prediction network;

[0238] Among them, step 1101 performed in the model inference process can refer to the steps performed in the feedforward process of the training process, and the similarities are not repeated here.

[0239] 1102. Perform feature extraction on the to-be-processed text through the feature extraction network to obtain a feature representation of the to-be-processed data, wherein the feature representation is plural; and

[0240] Among them, step 1102 performed in the model inference process can refer to the steps performed in the feedforward process of the training process, and the similarities are not repeated here.

[0241] 1103. Perform an orthogonal transformation on the feature representation after the length unitization processing through the prediction network to obtain an orthogonal transformation result, and determine a text prediction result based on the orthogonal transformation result.

[0242] Among them, step 1103 performed in the model inference process can refer to the steps performed in the feedforward process of the training process, and the similarities are not repeated here.

[0243] In the embodiment of the present application, compared with the W*a+b operation adopted by the existing prediction network, the orthogonal transformation can be adapted to the quantum computing of the quantum circuit. The quantum state obtained by the orthogonal transformation can pass through the quantum measurement layer, that is, measure the probability of collapse to each quantum ground state, and then perform the prediction network operation on the quantum circuit. Moreover, by constructing a pre-trained language model represented by complex values, the representation ability of the model is improved, and the performance of the network is improved.

[0244] In one possible implementation, the pre-trained language model is used to perform a target task, and the text processing result is the processing result of the target task; the target task is one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language reasoning, text automatic question answering, text intent recognition, text classification, text simplification, and text story generation.

[0245] In one possible implementation, the fully connected layers in the prediction network can be connected through activation layers, and the module length of the output close to the fully connected layer is normalized by softmax to obtain the probability distribution of each category.

[0246] For example, two fully connected layers can be added to the output of the last complex transformer layer for the prediction network of the mask language model (MLM) and next sentence prediction (NSP) tasks, connected by a Tanh nonlinear activation function for the real and imaginary parts. The final output of both networks is the probability distribution of their respective categories, obtained by normalizing the modulus of the output of the last complex fully connected layer through softmax.

[0247] In one possible implementation, in order to adapt the prediction network to the quantum adaptation structure of the quantum circuit, the input quantum state (unit complex vector) can first be orthogonally transformed (the parameters of the orthogonal transformation can be trained). The parameterization method of the orthogonal transformation layer is as follows:

[0248]

[0249] Here, A is a complex positive definite matrix, which is transformed into an orthogonal matrix U through matrix exponential operations. The network's errors can be backpropagated into the weights W, allowing the entire orthogonal transformation to be trained through backpropagation. The quantum state that has undergone the orthogonal transformation passes through a quantum measurement layer, which measures the probability of collapse to each quantum ground state. The square of the modulus of the complex vector corresponding to the quantum state is the measured probability vector. This probability vector is then passed through a linear projection layer to obtain the sentence's category label.

[0250] Specifically, in a possible implementation, performing an orthogonal transformation on the feature representation after the unitization process includes: performing an orthogonal transformation on the feature representation after the unitization process through an orthogonal matrix: the orthogonal matrix is:

[0251] U=e iA ;

[0252] in, W is the trainable weight and H is the conjugate transpose of the complex matrix.

[0253] It should be understood that the above-mentioned orthogonalization calculation process can be used during model pre-training or model fine-tuning.

[0254] In one possible implementation, a quantum-adapted pre-trained language model can be built upon the aforementioned model and adapted for quantum computation. To ensure the mathematical validity of this quantum computation mapping, the intermediate representation of the complex network's [CLS] character can be constrained to be unitary, allowing it to be considered a quantum state throughout the network, facilitating adaptation to quantum circuits.

[0255] In one possible implementation, the normalization layer may change the operation on CLS characters to a length unitization operation.

[0256] Specifically, in one possible implementation, the feature extraction network includes a transformer layer, which includes a normalization layer. The normalization layer is used to perform length unitization processing on a target character, where the target character is a CLS character inserted at the beginning of the text to be processed. The multi-layer normalization operation can refer to the following formula:

[0257]

[0258] in and σ z are the mean and standard deviation of the complex sequence, respectively.

[0259] In one possible implementation, the feature extraction network includes a transformer layer, the transformer layer includes a feed-forward layer FFN, the FFN includes an activation layer, and the activation layer is used to perform nonlinear activation on part of the data input into the activation layer, the part of the data does not include data corresponding to the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

[0260] In a possible implementation, the portion of data input into the activation layer is a complex number, and the activation layer is specifically configured to perform nonlinear activation on the real part and the imaginary part of the portion of data, respectively.

[0261] In one possible implementation, the prediction network includes a target fully connected layer, which is a fully connected layer close to the output layer in the prediction network, and the target fully connected layer includes a trainable first unit vector and a second unit vector; the result after the orthogonal transformation can be respectively operated with the first unit vector and the second unit vector to obtain the first probability and the second probability, the first unit vector corresponds to the first probability, and the second unit vector corresponds to the second probability, the first probability represents the probability that the text prediction result belongs to the target label, and the second probability represents the probability that the text prediction result does not belong to the target label; the text prediction result is determined based on the first probability and the second probability.

[0262] It should be understood that since the Q matrix, K matrix and V matrix in the embodiments of the present application are complex matrices, that is, each element therein is a complex number including a real part and an imaginary part, it is necessary to adopt an attention mechanism suitable for complex numbers.

[0263] In one possible implementation, since the softmax operation needs to be performed in the real number domain, and the result of the operation between the Q matrix and the K matrix (that is, the object of the softmax operation) is a complex number, the result (complex number) of the operation between the Q matrix and the K matrix can be mapped to the real number domain in an embodiment of the present application.

[0264] In one possible implementation, taking the result of the operation between the Q matrix and the K matrix as the first data as an example, the real part and the imaginary part can be mapped to the second data (real number) according to the values ​​of the real part and the imaginary part of the first data. For example, a preset operation can be performed, that is, a numerical operation can be performed on the values ​​of the real part and the imaginary part to obtain a real number value as the second data.

[0265] In one possible implementation, the modulus of the first data can be determined as the second data based on the values ​​of the real part and the imaginary part of the first data. Since in the transformer model based on complex numbers, the modulus of the complex number and the probability of the final output are related, that is, the modulus of the complex number itself has a definite physical meaning. In the process of mapping the real part and the imaginary part to the second data, the mapping method of the complex modulus can increase the interpretability of the network and improve the accuracy of the network.

[0266] Specifically, in a possible implementation, a head (an attention head in a multi-head attention) can be used to obtain the K matrix and the Q matrix of the text to be processed; the K matrix and the Q matrix are calculated to obtain first data, which is a complex number; the numerical values ​​of the real part and the imaginary part of the first data are mapped to second data, which is a real number; and a softmax operation is performed on the second data.

[0267] Specifically, in a possible implementation, mapping the values ​​of the real part and the imaginary part of the first data to the second data includes: determining the modulus of the first data as the second data according to the values ​​of the real part and the imaginary part of the first data.

[0268] exist Figures 1 to 11 On the basis of the corresponding embodiment, in order to better implement the above solution of the embodiment of the present application, the following also provides related equipment for implementing the above solution. Figure 12 , Figure 12 This is a schematic diagram of the structure of a data processing device 1200 provided in an embodiment of the present application. The data processing device 1200 may be a terminal device or a server. The data processing device 1200 includes:

[0269] An acquisition module 1201 is configured to acquire a text to be processed and a pre-trained language model, wherein the pre-trained language model includes a feature extraction network and a prediction network;

[0270] For a detailed description of the acquisition module 1201 , reference may be made to the description of step 601 and step 1101 in the above embodiment, which will not be repeated here.

[0271] A feature extraction module 1202 is configured to extract features of the text to be processed through the feature extraction network to obtain a feature representation of the data to be processed, wherein the feature representation is plural; and

[0272] For a detailed description of the feature extraction module 1202 , reference may be made to the description of step 602 and step 1102 in the above embodiment, which will not be repeated here.

[0273] The prediction module 1203 is configured to perform an orthogonal transformation on the feature representation after the length unitization processing through the prediction network to obtain an orthogonal transformed result, and determine a text prediction result based on the orthogonal transformed result.

[0274] For a detailed description of the prediction module 1203 , reference may be made to the description of step 603 and step 1103 in the above embodiment, which will not be repeated here.

[0275] In one possible implementation, the pre-trained language model is used to perform a target task, and the text processing result is a processing result of the target task; the target task is one of the following:

[0276] Reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language reasoning, text automatic question answering, text intent recognition, text classification, text simplification, and text story generation.

[0277] In a possible implementation, the prediction module is specifically configured to:

[0278] The orthogonal transformation is performed on the normalized feature representation through an orthogonal matrix: the orthogonal matrix is:

[0279] U=e iA ;

[0280] in, W is the trainable weight and H is the conjugate transpose of the complex matrix.

[0281] In one possible implementation, the feature extraction network includes a transformer layer, the transformer layer includes a normalization layer, and the normalization layer is used to perform length unit processing on the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

[0282] In one possible implementation, the feature extraction network includes a transformer layer, the transformer layer includes a feed-forward layer FFN, the FFN includes an activation layer, and the activation layer is used to perform nonlinear activation on part of the data input into the activation layer, the part of the data does not include data corresponding to the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

[0283] In a possible implementation, the portion of data input into the activation layer is a complex number, and the activation layer is specifically used to perform nonlinear activation on the real part and the imaginary part of the portion of data respectively.

[0284] In a possible implementation, the prediction network includes a target fully connected layer, where the target fully connected layer is a fully connected layer close to an output layer in the prediction network, and the target fully connected layer includes a trainable first unit vector and a second unit vector;

[0285] The prediction module is specifically used to:

[0286] The result after the orthogonal transformation is respectively operated with the first unit vector and the second unit vector to obtain the first probability and the second probability, where the first unit vector corresponds to the first probability and the second unit vector corresponds to the second probability, the first probability represents the probability that the text prediction result belongs to the target label, and the second probability represents the probability that the text prediction result does not belong to the target label;

[0287] A text prediction result is determined according to the first probability and the second probability.

[0288] In one possible implementation, the feature extraction network includes a transformer layer, and the transformer layer includes an attention head;

[0289] The head is used to obtain the K matrix and Q matrix of the text to be processed;

[0290] Calculating the K matrix and the Q matrix to obtain first data, where the first data is a complex number;

[0291] Mapping the values ​​of the real part and the imaginary part of the first data into second data, where the second data is a real number;

[0292] Perform a softmax operation on the second data.

[0293] In one possible implementation, mapping the values ​​of the real part and the imaginary part of the first data into second data includes:

[0294] The modulus of the first data is determined as the second data according to values ​​of the real part and the imaginary part of the first data.

[0295] In a possible implementation, the apparatus further includes:

[0296] A model updating module 1204 is configured to determine a target loss based on the text prediction result;

[0297] Backpropagation of the pre-trained language model is performed according to the target loss, wherein the gradient and momentum used in the backpropagation are complex numbers.

[0298] For a detailed description of the model updating module 1204 , reference may be made to the description of step 604 and step 605 in the above embodiment, which will not be repeated here.

[0299] Next, we will introduce an execution device provided by the embodiment of the present application. Figure 13 , Figure 13This is a structural diagram of an execution device provided in an embodiment of the present application. The execution device 1300 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device or a server, etc., which is not limited here. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 can be one or more, Figure 13 (taking one processor as an example), the processor 1303 may include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 may be connected via a bus or other means.

[0300] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0301] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0302] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1303. Processor 1303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1303. The above processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1303 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1304, and processor 1303 reads the information in memory 1304 and, in conjunction with its hardware, completes the steps of the above method.

[0303] Receiver 1301 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1302 can be used to output digital or character information through the first interface. Transmitter 1302 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1302 can also include a display device such as a display screen.

[0304] In one embodiment of the present application, the processor 1303 is configured to execute Figure 11 The data processing method executed by the device in the corresponding embodiment.

[0305] The present application also provides a training device. Figure 14 , Figure 14 This is a structural diagram of a training device provided in an embodiment of the present application. The training device 1400 may be deployed with Figure 12Corresponding to the data processing apparatus described in the embodiments, specifically, training device 1400 is implemented by one or more servers. Training device 1400 may vary significantly due to configuration or performance differences, and may include one or more central processing units (CPUs) 1414 (e.g., one or more processors), memory 1432, and one or more storage media 1430 (e.g., one or more mass storage devices) storing application programs 1442 or data 1444. Memory 1432 and storage media 1430 may be either transient or persistent storage. The program stored in storage media 1430 may include one or more modules (not shown), each module comprising a series of instruction operations on the training device. Furthermore, CPU 1414 may be configured to communicate with storage media 1430, executing the series of instruction operations stored in storage media 1430 on training device 1400.

[0306] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or, one or more operating systems 1441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0307] In the embodiment of the present application, the central processing unit 1414 is used to execute Figure 6 The data processing method is executed by the data processing device in the corresponding embodiment.

[0308] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0309] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0310] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0311] For details, please refer to Figure 15 , Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor (NPU) 1500. NPU 1500 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1503, which is controlled by controller 1504 to extract matrix data from memory and perform multiplication operations.

[0312] In some implementations, arithmetic circuit 1503 includes multiple processing units (PEs). In some implementations, arithmetic circuit 1503 is a two-dimensional systolic array. Arithmetic circuit 1503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1503 is a general-purpose matrix processor.

[0313] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 1502 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 1501 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 1508.

[0314] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 through the Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 through the DMAC.

[0315] BIU stands for Bus Interface Unit 1510 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1509 .

[0316] The bus interface unit 1510 (BIU) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0317] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or transfer weight data to the weight memory 1502 or transfer input data to the input memory 1501.

[0318] The vector calculation unit 1507 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0319] In some implementations, the vector calculation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in subsequent layers in a neural network.

[0320] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;

[0321] Unified memory 1506, input memory 1501, weight memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0322] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0323] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0324] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0325] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0326] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a text to be processed and a pre-trained language model, wherein the pre-trained language model includes a feature extraction network and a prediction network, wherein the prediction network includes a target fully connected layer, and wherein the target fully connected layer includes a trainable first unit vector and a second unit vector; Performing feature extraction on the text to be processed through the feature extraction network to obtain a feature representation of the text to be processed, wherein the feature representation is plural; and Through the prediction network, an orthogonal transformation is performed on the feature representation after length unitization to obtain an orthogonal transformation result, and the orthogonal transformation result is respectively operated with the first unit vector and the second unit vector to obtain a first probability and a second probability, and a text prediction result is determined based on the first probability and the second probability.

2. The method according to claim 1, characterized in that The pre-trained language model is used to perform a target task, and the text prediction result is a processing result of the target task; the target task is one of the following: Reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language reasoning, text automatic question answering, text intent recognition, text classification, text simplification, and text story generation.

3. The method according to claim 1 or 2, characterized in that The performing an orthogonal transformation on the feature representation after the length unitization process includes: The orthogonal transformation is performed on the normalized feature representation through an orthogonal matrix: the orthogonal matrix is: U=e iA ; in, W is the trainable weight and H is the conjugate transpose of the complex matrix.

4. The method according to claim 1 or 2, characterized in that The feature extraction network includes a transformer layer, the transformer layer includes a normalization layer, and the normalization layer is used to perform length unit processing on the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

5. The method according to claim 2, characterized in that The feature extraction network includes a transformer layer, the transformer layer includes a feed-forward layer FFN, the FFN includes an activation layer, the activation layer is used to perform nonlinear activation on part of the data input into the activation layer, the part of the data does not include data corresponding to a target character, and the target character is a CLS character inserted at the starting position of the text to be processed; or, The prediction network includes an activation layer, which is used to perform nonlinear activation on part of the data input into the activation layer, wherein the part of the data does not include data corresponding to the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

6. The method according to claim 5, characterized in that The part of the data input into the activation layer is a complex number, and the activation layer is specifically used to perform nonlinear activation on the real part and the imaginary part of the part of the data respectively.

7. The method according to claim 1 or 2, characterized in that The target fully connected layer is a fully connected layer close to the output layer in the prediction network; the first unit vector corresponds to a first probability, and the second unit vector corresponds to a second probability, the first probability represents the probability that the text prediction result belongs to the target label, and the second probability represents the probability that the text prediction result does not belong to the target label.

8. The method according to claim 6, characterized in that The feature extraction network includes a transformer layer, and the transformer layer includes an attention head; The head is used to obtain the K matrix and Q matrix of the text to be processed; Calculating the K matrix and the Q matrix to obtain first data, where the first data is a complex number; Mapping the values ​​of the real part and the imaginary part of the first data into second data, where the second data is a real number; Perform a softmax operation on the second data.

9. The method according to claim 8, characterized in that Mapping the values ​​of the real part and the imaginary part of the first data into second data includes: The modulus of the first data is determined as the second data according to values ​​of the real part and the imaginary part of the first data.

10. The method according to claim 1 or 2, characterized in that The method further comprises: Determining a target loss based on the text prediction result; Backpropagation of the pre-trained language model is performed according to the target loss, wherein the gradient and momentum used in the backpropagation are complex numbers.

11. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a text to be processed and a pre-trained language model, wherein the pre-trained language model includes a feature extraction network and a prediction network, wherein the prediction network includes a target fully connected layer, and wherein the target fully connected layer includes a trainable first unit vector and a second unit vector; a feature extraction module, configured to extract features of the text to be processed through the feature extraction network to obtain a feature representation of the text to be processed, wherein the feature representation is plural; and The prediction module is used to perform an orthogonal transformation on the feature representation after length unitization through the prediction network to obtain an orthogonal transformed result, and respectively operate the orthogonal transformed result with the first unit vector and the second unit vector to obtain a first probability and a second probability, and determine a text prediction result based on the first probability and the second probability.

12. The device according to claim 11, characterized in that The pre-trained language model is used to perform a target task, and the text prediction result is a processing result of the target task; the target task is one of the following: Reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language reasoning, text automatic question answering, text intent recognition, text classification, text simplification, and text story generation.

13. The device according to claim 11 or 12, characterized in that The prediction module is specifically used to: The orthogonal transformation is performed on the normalized feature representation through an orthogonal matrix: the orthogonal matrix is: U=e iA ; in, W is the trainable weight and H is the conjugate transpose of the complex matrix.

14. The device according to claim 11 or 12, characterized in that The feature extraction network includes a transformer layer, the transformer layer includes a normalization layer, and the normalization layer is used to perform length unit processing on the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

15. The device according to claim 12, characterized in that The feature extraction network includes a transformer layer, the transformer layer includes a feed-forward layer FFN, the FFN includes an activation layer, the activation layer is used to perform nonlinear activation on part of the data input into the activation layer, the part of the data does not include data corresponding to a target character, and the target character is a CLS character inserted at the starting position of the text to be processed; or, The prediction network includes an activation layer, which is used to perform nonlinear activation on part of the data input into the activation layer, wherein the part of the data does not include data corresponding to the target character, and the target character is a CLS character inserted at the starting position of the text to be processed.

16. The device according to claim 15, characterized in that The part of the data input into the activation layer is a complex number, and the activation layer is specifically used to perform nonlinear activation on the real part and the imaginary part of the part of the data respectively.

17. The device according to claim 11 or 12, characterized in that The target fully connected layer is a fully connected layer close to the output layer in the prediction network; the first unit vector corresponds to a first probability, and the second unit vector corresponds to a second probability, the first probability represents the probability that the text prediction result belongs to the target label, and the second probability represents the probability that the text prediction result does not belong to the target label.

18. The device according to claim 16, characterized in that The feature extraction network includes a transformer layer, and the transformer layer includes an attention head; The head is used to obtain the K matrix and Q matrix of the text to be processed; Calculating the K matrix and the Q matrix to obtain first data, where the first data is a complex number; Mapping the values ​​of the real part and the imaginary part of the first data into second data, where the second data is a real number; Perform a softmax operation on the second data.

19. The device according to claim 18, characterized in that Mapping the values ​​of the real part and the imaginary part of the first data into second data includes: The modulus of the first data is determined as the second data according to values ​​of the real part and the imaginary part of the first data.

20. The device according to claim 11 or 12, characterized in that The device further comprises: A model updating module, configured to determine a target loss based on the text prediction result; Backpropagation of the pre-trained language model is performed according to the target loss, wherein the gradient and momentum used in the backpropagation are complex numbers.

21. A data processing device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to obtain the codes and execute the method according to any one of claims 1 to 10.

22. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on a computer device, cause the computer device to execute the method according to any one of claims 1 to 10.

Images